We present the modifications, performance, and test of a diamond anvil cell for radio frequency dielectric spectroscopy studies of single crystals that can be used from room temperature down to 4 K and up to pressures of 5-6 GPa. Continuous frequency-dependent measurements between 5 Hz and 1 MHz can be performed with this modified pressure cell. The cell has an excellent performance with temperature-, frequency-, and pressure-independent stray capacitance of around 2 pF, enabling us to use relatively small samples with a weak dielectric response.