A passive data-driven method to localize a defect in a structure using the ambient noise is derived. The approach requires only acoustic measurements in a spatially random noise field and no knowledge of the structure. Measurements are taken before and after the perturbation has occurred and Green's functions are retrieved by cross-correlating acoustic measurements. The difference between measured data reveals the perturbation. A frequency domain method based on matched field processing is then performed to localize the perturbation. The Bartlett, minimum variance and white noise gain constraint processors are implemented and their performances are illustrated on a numerical experiment.