By performing quantum-noise-limited optical heterodyne detection, we observe polarization noise in light after propagation through a hollow-core photonic crystal fiber (PCF). We compare the noise spectrum to the one of a standard fiber and find an increase of noise even though the light is mainly transmitted in air in a hollow-core PCF. Combined with our simulation of the acoustic vibrational modes in the hollow-core PCF, we are offering an explanation for the polarization noise with a variation of guided acoustic wave Brillouin scattering (GAWBS). Here, instead of modulating the strain in the fiber core as in a solid core fiber, the acoustic vibrations in hollow-core PCF influence the effective refractive index by modulating the geometry of the photonic crystal structure. This induces polarization noise in the light guided by the photonic crystal structure.