In the present work, we investigate the effect of "fatigue" on the fatigue behavior and atomic structure of Zr-based BMGs. Fatigue experiments on the failed-by-fatigue samples indicate that the remnants generally have similar or longer fatigue life than the as-cast samples. Meanwhile, the pair-distribution-function (PDF) analysis of the as-cast and post-fatigue samples showed very small changes of local atomic structures. These observations suggest that the fatigue life of the 6-mm in-diameter Zr-based BMG is dominated by the number of pre-existing crack-initiation sites in the sample. Once the crack initiates in the specimen, the fatigue-induced damage is accumulated locally on these initiated sites, while the rest of the region deforms elastically. The results suggest that the fatigue failure of BMGs under compression-compression fatigue experiments is a defect-controlled process. The present work indicates the significance of the improved fatigue resistance with decreasing the sample size.