Dielectric measurements on insulating materials at cryogenic temperatures can be challenging, depending on the frequency and temperature ranges of interest. We present a technique to study the dielectric properties of bulk dielectrics at GHz frequencies. A superconducting coplanar Nb resonator is deposited directly on the material of interest, and this resonator is then probed in distant-flip-chip geometry with a microwave feedline on a separate chip. Evaluating several harmonics of the resonator gives access to various probing frequencies in the present studies up to 20 GHz. We demonstrate the technique on three different materials (MgO, LaAlO3, and TiO2), at temperatures between 1.4 K and 7 K.