Volumetric wave-based simulation methods for room and virtual acoustics, such as the finite difference time domain method, are computationally intensive; for large volumes, operation over a regular grid is desirable for the sake of efficiency. In coping with realistic irregular geometries (such as enclosures or scattering objects), form-fitting meshing can disturb grid regularity and introduce new difficulties in terms of maintaining numerical stability. An alternative is the immersed boundary method, allowing for the representation of an irregular boundary over a regular grid through additional forcing terms in the dynamical system. While heavily used in fluid-structure interaction problems, such methods have seen less application in virtual acoustics. In this article, a simplified form of the immersed boundary method tailored to virtual acoustics is presented. Under appropriate passivity-preserving discretisation techniques, simple numerical stability conditions can be proved, and in particular, impedance boundaries may be incorporated easily without any risk of numerical instability. In addition, the method retains a largely explicit character with a small linear system solution required over the immersed boundary surface. Numerical results in two and three dimensions, illustrating various interior and exterior problem scenarios, are presented.