We propose and analyze theoretically a cavity optomechanical analog of a heat pump that uses a polariton fluid to cool mechanical modes coupled to a single precooled phonon mode via external modulation of the substrate of the mechanical resonator. This approach permits us to cool phonon modes of arbitrary frequencies not limited by the cavity-optical field detuning deep into the quantum regime from room temperature.