The silicon-vacancy center in diamond offers attractive opportunities in quantum photonics due to its favorable optical properties and optically addressable electronic spin. Here, we combine both to achieve all-optical coherent control of its spin states. We utilize this method to explore spin dephasing effects in an impurity-rich sample beyond the limit of phonon-induced decoherence: Employing Ramsey and Hahn-echo techniques at temperatures down to 40 mK we identify resonant coupling to a substitutional nitrogen spin bath as limiting decoherence source for the electron spin.