Photophysical and photochemical processes are often dominated by molecular vibrations in various electronic states. Dissecting the corresponding, often overlapping, spectroscopic signals from different electronic states is a challenge hampering their interpretation. Here we address impulsive stimulated Raman spectroscopy (ISRS), a powerful technique able to coherently stimulate and record Raman-active modes using broadband pulses. Using a quantum-mechanical treatment of the ISRS process, we show the mode-specific way the various spectral components of the broadband probe contribute to the signal generated at a given wavelength. We experimentally demonstrate how to manipulate the signal by varying the probe chirp and the phase-matching across the sample, thereby affecting the relative phase between the various contributions to the signal. These novel control knobs allow us to selectively enhance desired vibrational features and distinguish spectral components arising from different excited states.