The role of metals in urinary stone formation has already been studied in several publications. Moreover, urinary calculi can also be used for assessing exposure of humans to minor and trace elements in addition to other biological matrices, for example, blood, urine, or hair. However, using urinary calculi for biomonitoring of trace elements is limited by the association of elements with certain types of minerals. In this work, 614 samples of urinary calculi were subjected to mineralogical and elemental analysis. Inductively coupled plasma mass spectrometry and thermo-oxidation cold vapor atomic absorption spectrometry were used for the determination of major, minor, and trace elements. Infrared spectroscopy was used for mineralogical analysis, and additionally, it was also employed in the calculation of mineralogical composition, based on quantification of major elements and stoichiometry. Results demonstrate the applicability of such an approach in investigating associations of minor and trace elements with mineralogical constituents of stones, especially in low concentrations, where traditional methods of mineralogical analysis are not capable of quantifying mineral content reliably. The main result of this study is the confirmation of association of several elements with struvite (K, Rb) and with calcium phosphate minerals, here calculated as hydroxylapatite (Na, Zn, Sr, Ba, Pb). Phosphates were proved as the most important metal-bearing minerals in urinary calculi. Moreover, a significantly different content was also observed for Fe, Zr, Mo, Cu, Cd, Se, Sn, and Hg in investigated groups of minerals. Examination of such associations is essential, and critical analysis of mineral constituents should precede any comparison of element content among various groups of samples.