This paper reports a study of the backscattered ultrasonic signal from a solid layer containing spherical cavities, to determine the conditions in which an effective medium model is a valid description of the response. The work is motivated by the need to model the response of porous composite materials for ultrasonic non-destructive evaluation (NDE) techniques. The numerical simulation predicts the response of a layer containing cavities at a single set of random locations, and compares it to the predicted response from a homogeneous layer with ensemble-averaged material properties (effective medium model). The study investigates the conditions in which the coherent (ensemble-averaged) response is obtained even from a single configuration of scatterers. Simulations are carried out for a range of cavity sizes and volume fractions. The deviation of the response from effective medium behavior is modeled, along with the trends as a function of cavity radius, volume fraction, and frequency, in order to establish an acceptability criterion for application of an effective medium model.