The detection of Gad-enhancing lesions in brain MRI of Multiple Sclerosis (MS) patients is of great clinical interest since they are important markers of disease activity. However, many of the enhancing voxels are associated with normal structures (i.e. blood vessels) or noise in the MRI, making the detection of Gad-enhancing lesions a challenging task. Furthermore, these lesions are typically small and in close proximity to vessels. In this paper, we present a probabilistic Adaptive Multi-level Conditional Random Field (AMCRF) framework, capable of leveraging spatial and temporal information, for detection of MS Gad-enhancing lesions. In the first level, a voxel based CRF with cliques of up to size three, is used to identify candidate lesions. In the second level, higher order potentials are incorporated leveraging robust textural features which are invariant to rotation and local intensity distortions. Furthermore, we show how to exploit temporal and longitudinal images, should they be available, into the AMCRF model. The proposed algorithm is tested on 120 multimodal clinical datasets acquired from Relapsing-Remitting MS patients during multi-center clinical trials. Results show a sensitivity of 93%, a positive predictive value of 70% and average False Positive (FP) counts of 0.77. Moreover, the temporal AMCRF results show the same sensitivity as the AMCRF model while decreasing the FP counts by 22%.