A consequence of using a parametric frailty model with nonparametric baseline hazard for analyzing clustered time-to-event data is that its regression coefficient estimates could be sensitive to the underlying frailty distribution. Recently, there has been a proposal for specifying both the baseline hazard and the frailty distribution nonparametrically, and estimating the unknown parameters by the maximum penalized likelihood method. Instead, in this paper, we propose the nonparametric maximum likelihood method for a general class of nonparametric frailty models, i.e. models where the frailty distribution is completely unspecified but the baseline hazard can be either parametric or nonparametric. The implementation of the estimation procedure can be based on a combination of either the Broyden-Fletcher-Goldfarb-Shanno or expectation-maximization algorithm and the constrained Newton algorithm with multiple support point inclusion. Simulation studies to investigate the performance of estimation of a regression coefficient by several different model-fitting methods were conducted. The simulation results show that our proposed regression coefficient estimator generally gives a reasonable bias reduction when the number of clusters is increased under various frailty distributions. Our proposed method is also illustrated with two data examples.
Keywords: Clustered survival data; hierarchical likelihood; nonparametric frailty model; nonparametric maximum likelihood estimator; random effect.