We study quantum state transfer between two qubits coupled to a common quantum bus that is constituted by an ultrastrong coupled light-matter system. By tuning both qubit frequencies on resonance with a forbidden transition in the mediating system, we demonstrate a high-fidelity swap operation even though the quantum bus is thermally populated. We discuss a possible physical implementation in a realistic circuit QED scheme that leads to the multimode Dicke model. This proposal may have applications on hot quantum information processing within the context of ultrastrong coupling regime of light-matter interaction.