Electronic speckle pattern interferometry (ESPI), a well-established technique for micro-deformation measurement, can be used to determine both in-plane and out-of-plane displacement components. Although many works in ESPI have been reported for three-dimensional (3D) displacement measurement, few works have focused on the simultaneous measurement of 3D deformation fields. Here we present an ESPI system that consists of three sub-interferometers for simultaneous measurement of all three displacement components and in-plane strain fields. A 3CCD color camera, a specially designed shifting stage, and three lasers with optimal wavelengths are used in this system. The lasers and 3CCD camera provide independent interferograms with different color signals, while the shifting stage allows the sub-interferometers to achieve simultaneous phase shifting. The results of color separation and experimental measurement demonstrate the utility of the system.