In many epidemiological studies time to event data are clustered and the physiological relationship between (time-dependent) covariates and the log hazard is often not linear as assumed in the Cox model. Introducing frailties in the Cox model can account for the clustering of the data and smoothing splines can be used to describe nonlinear relations. These two extensions of the Cox model are introduced jointly and it is shown how penalized partial likelihood techniques can be used to fit the extended model. We demonstrate the need for such a model to study the relation between the physiological covariates milk ureum and protein concentration and the log hazard of first insemination in dairy cows, with the farms as clusters.