A fully automated knee magnetic resonance imaging (MRI) segmentation method to study osteoarthritis (OA) was developed using a novel hierarchical set of random forests (RF) classifiers to learn the appearance of cartilage regions and their boundaries. A neighborhood approximation forest is used first to provide contextual feature to the second-level RF classifier that also considers local features and produces location-specific costs for the layered optimal graph image segmentation of multiple objects and surfaces (LOGISMOS) framework. Double-echo steady state MRIs used in this paper originated from the OA Initiative study. Trained on 34 MRIs with varying degrees of OA, the performance of the learning-based method tested on 108 MRIs showed significant reduction in segmentation errors ( ) compared with the conventional gradient-based and single-stage RF-learned costs. The 3-D LOGISMOS was extended to longitudinal-3-D (4-D) to simultaneously segment multiple follow-up visits of the same patient. As such, data from all time-points of the temporal sequence contribute information to a single optimal solution that utilizes both spatial 3-D and temporal contexts. 4-D LOGISMOS validation on 108 MRIs from baseline, and 12 month follow-up scans of 54 patients showed significant reduction in segmentation errors ( ) compared with 3-D. Finally, the potential of 4-D LOGISMOS was further explored on the same 54 patients using five annual follow-up scans demonstrating a significant improvement of measuring cartilage thickness ( ) compared with the sequential 3-D approach.