Mueller matrix polarimetry (MMP) has been demonstrated and recognized as an effective approach to attaining imaging enhancement as well as revealing polarization properties of an imaged sample. Generally, a minimum of 16 combinations of intensity-only measurements involving both linear and circular polarizations are required to completely and accurately determine the 4 × 4 Mueller matrix (MM) and comprehensively describe the polarization properties of the sample. However, broadband circular polarizations (CP) are rather difficult to obtain for design and fabrication limitations in the terahertz region, which poses a challenge to the acquisition of the 4 × 4 MM. In this circumstance, the 3 × 3 MM degradation using only linear polarizations (LP) is preferred and sufficient for characterization of non-depolarizing samples. In this paper, a multi-spectral 3 × 3 MMP system based on the THz time-domain spectroscopy (THz-TDS) is established from 0.1 to 1 THz. The system demonstrated is capable of fulfilling the accurate determination of the 3 × 3 MM. The Mueller matrix polar decomposition (MMPD), modified to be compatible with the MM degradation, is employed to explore the fine details and properties of the sample. By signal post-processing techniques, the MM elements in the time domain are retrieved, and the time dimension reflecting the depth information facilitates the 3D reconstruction of the sample. This work provides a prototype for 3D imaging of biological samples at higher frequencies in the future.