The temporal coherence of an ideal Bose gas increases as the system approaches the Bose-Einstein condensation threshold from below, with coherence time diverging at the critical point. However, counterexamples have been observed for condensates of photons formed in an externally pumped, dye-filled microcavity, wherein the coherence time decreases rapidly for increasing particle number above threshold. This Letter establishes intermode correlations as the central explanation for the experimentally observed dramatic decrease in the coherence time beyond critical pump power.