White and black deposits have been frequently observed on the surface of Korean stone cultural heritages, and they are considered as damage factors in both conservation and esthetic points of view. In order to set up the appropriate conservation remedy, it is important to know their origins, characteristics, and compositions. In this study, optical and scanning electron microscopes (SEMs) equipped with an energy-dispersive spectrometer (EDS) and X-ray diffractometer were employed to determine the white and black deposits. It was found that both deposits consisted mainly of calcium carbonate (calcite) and calcium sulfate (gypsum). The calcite and gypsum were characterized by bladed, rhombohedral, tabular, and amorphous morphologies under a SEM. The black deposit was not only composed of calcite or gypsum, but also accompanied amorphous and irregular matrix. SEM-EDS analysis revealed an abundance of silicon, aluminum, iron, phosphorus, and carbon on the matrix, which were major elements of soil, atmospheric deposits, and organisms. The white deposit, on the other hand, barely contained those coloring substances. These salts and deposited substances were caused by chemical reaction and physical adhesion between rock-forming minerals, lime mortar, sulfur in polluted air environment, soil dust, and microorganisms.