We describe the first investigations of the complete engineering model of the optical metrology system (OMS), a key subsystem of the LISA Pathfinder science mission to space. The latter itself is a technological precursor mission to LISA, a spaceborne gravitational wave detector. At its core, the OMS consists of four heterodyne Mach-Zehnder interferometers, a highly stable laser with an external modulator, and a phase meter. It is designed to monitor and track the longitudinal motion and attitude of two floating test masses in the optical reference frame with (relative) precision in the picometer and nanorad range, respectively. We analyze sensor signal correlations and determine a physical sensor noise limit. The coupling parameters between motional degrees of freedom and interferometer signals are analytically derived and compared to measurements. We also measure adverse cross-coupling effects originating from system imperfections and limitations and describe algorithmic mitigation techniques to overcome some of them. Their impact on system performance is analyzed within the context of the Pathfinder mission.