Displacement measuring interferometry is a crucial component in metrology applications. In this paper, we propose a fiber-based two-wavelength heterodyne interferometer as a compact and highly sensitive displacement sensor that can be used in inertial sensing applications. In the proposed design, two individual heterodyne interferometers are constructed using two different wavelengths, 1064 nm and 1055 nm; one of which measures the target displacement and the other monitors the common-mode noise in the fiber system. A narrow-bandwidth spectral filter separates the beam paths of the two interferometers, which are highly common and provide a high rejection ratio to the environmental noise. The preliminary test shows a sensitivity floor of 7.5pm/Hz at 1 Hz when tested in an enclosed chamber. We also investigated the effects of periodic errors due to imperfect spectral separation on the displacement measurement and propose algorithms to mitigate these effects.