A three-dimensional coupled vibroacoustic finite element model for physics-based simulations of the sound generation by mallet percussion instruments in the time domain is discussed in the present paper. The mechanical model takes the orthotropic material properties of the wooden sound bars and the nonlinear nature of the interaction force between the mallet head and the sound bar into account while the acoustical model considers radiation into an unbounded domain. A direct coupling of the sound bars, acoustical cavity resonators, and the excitation by a mallet is considered with exploiting the modal basis to reduce the number of degrees of freedom of the system. Both the mechanical and acoustical models are validated by comparing them to measurements performed on an Orff xylophone. A case study shows the capabilities of the coupled model, including the analysis of the energy balance, the effect of tuning the resonator, and the excitation of the torsional modes of the sound bar.