The temperature dependence of the processes which fundamentally limit optical quality factor of ideal crystalline whispering gallery mode resonators is investigated. The example of CaF(2) is used to show that spontaneous Raman scattering is the main limitation of the quality factor at low temperatures. Stimulated Raman scattering is also shown to be important at any temperature. We experimentally demonstrate nonlinear absorption due to stimulated Raman scattering in a real cavity at room temperature and theoretically derive Raman gain of CaF(2). We conclude that optical storage times in excess of one second could be achieved in millimeter sized cavities.