Vibration measurement is a frequent measurement requirement in a number of areas. Optical vibration sensors have many advantages over electrical counterparts. A common approach is to optically detect the vibration induced mechanical movement of a cantilever. Nevertheless, their practical applications are hindered by the cross-sensitivity of temperature and dynamic instability of the mechanical structure, which lead to unreliable vibration measurements. Here, we demonstrate a temperature insensitive vibration sensor that involves an enclosed suspended cantilever integrated with a readout fiber, providing in-line measurement of vibration. The cantilever is fabricated from a highly birefringent photonic crystal fiber by chemical etching and fused to a single-polarization fiber. Mechanical vibration induced periodic bending of the cantilever can significantly modify the state of polarization of the light that propagates along the photonic crystal fiber. The single-polarization fiber finally converts the state of polarization fluctuation into the change of output optical power. Therefore, the vibration could be demodulated by monitoring the output power of the proposed structure. Due to the special design of the structure, the polarization fluctuation induced by a variation of the ambient temperature can be significantly suppressed. The sensor has a linear response over the frequency range of 5 Hz to 5 kHz with a maximum signal-to-noise ratio of 60 dB and is nearly temperature independent.