Flow patterns arising in a vertical differentially heated layer of nonconducting ferromagnetic fluid placed in an external uniform transverse magnetic field are studied experimentally and discussed from the point of view of the perturbation energy balance. A quantitative criterion for detecting the parametric point where the dominant role in generating a flow instability is transferred between the thermogravitational and thermomagnetic mechanisms is suggested, based on the disturbance energy balance analysis. A comprehensive experimental study of various flow patterns is undertaken, and the existence is demonstrated of oblique thermomagnetic waves theoretically predicted by Suslov [Phys. Fluids 20, 084101 (2008)] and superposed onto the stationary magnetoconvective pattern known previously. It is found that the wave number of the detected convection patterns depends sensitively on the temperature difference across the layer and on the applied magnetic field. In unsteady regimes its value varies periodically by a factor of almost 2, indicating the appearance of two different competing wave modes. The wave numbers and spatial orientation of the observed dominant flow patterns are found to be in good agreement with theoretical predictions.