The relevance of neglecting three- and four-body interactions in the coarse-grained version of the Asakura-Oosawa model is examined. A mapping between the first few virial coefficients of the binary nonadditive hard-sphere mixture representative of this model and those arising from the coarse-grained (pairwise) depletion potential approximation allows for a quantitative evaluation of the effect of such interactions. This turns out to be especially important for large size ratios and large reservoir polymer packing fractions.