REML estimation of variance parameters in nonlinear mixed effects models using the SAEM algorithm

Abstract

Nonlinear mixed effects models are now widely used in biometrical studies, especially in pharmacokinetic research or for the analysis of growth traits for agricultural and laboratory species. Most of these studies, however, are often based on ML estimation procedures, which are known to be biased downwards. A few REML extensions have been proposed, but only for approximated methods. The aim of this paper is to present a REML implementation for nonlinear mixed effects models within an exact estimation scheme, based on an integration of the fixed effects and a stochastic estimation procedure. This method was implemented via a stochastic EM, namely the SAEM algorithm. The simulation study showed that the proposed REML estimation procedure considerably reduced the bias observed with the ML estimation, as well as the residual mean squared error of the variance parameter estimations, especially in the unbalanced cases. ML and REML based estimators of fixed effects were also compared via simulation. Although the two kinds of estimates were very close in terms of bias and mean square error, predictions of individual profiles were clearly improved when using REML vs. ML. An application of this estimation procedure is presented for the modelling of growth in lines of chicken.