Coherent generation of indistinguishable single photons is crucial for many quantum communication and processing protocols. Solid-state realizations of two-level atomic transitions or three-level spin-Λ systems offer significant advantages over their atomic counterparts for this purpose, albeit decoherence can arise due to environmental couplings. One popular approach to mitigate dephasing is to operate in the weak-excitation limit, where the excited-state population is minimal and coherently scattered photons dominate over incoherent emission. Here we probe the coherence of photons produced using two-level and spin-Λ solid-state systems. We observe that the coupling of the atomiclike transitions to the vibronic transitions of the crystal lattice is independent of the driving strength, even for detuned excitation using the spin-Λ configuration. We apply a polaron master equation to capture the non-Markovian dynamics of the vibrational manifolds. These results provide insight into the fundamental limitations to photon coherence from solid-state quantum emitters.