We report here the realization and commissioning of an experiment dedicated to the study of the optical properties of light-matter hybrids constituted of crystalline samples embedded in an optical cavity. The experimental assembly developed offers the unique opportunity to study the weak and strong coupling regimes between a tunable optical cavity in cryogenic environment and low energy degrees of freedom, such as phonons, magnons, or charge fluctuations. We describe here the setup developed that allows for the positioning of crystalline samples in an optical cavity of different quality factors, the tuning of the cavity length at cryogenic temperatures, and its optical characterization with a broadband time domain THz spectrometer (0.2-6 THz). We demonstrate the versatility of the setup by studying the vibrational strong coupling in CuGeO3 single crystal at cryogenic temperatures.