Automatic segmentation of the human brain ventricular system from MR images is useful in studies of brain anatomy and its diseases. Existing intensity-based segmentation methods are adaptive to large shape and size variations of the ventricular system, but may leak to the non-ventricular regions due to the non-homogeneity, noise and partial volume effect in the images. Deformable model-based methods are more robust to noise and alleviate the leakage problem, but may generate wrong results when the shape or size of the ventricle to be segmented in the images has a large difference in comparison to its model. In this paper, we propose a knowledge-based region growing and trimming approach where: (1) a model of a ventricular system is used to define regions of interest (ROI) for the four ventricles (i.e., left, right, third and fourth); (2) to segment a ventricle in its ROI, a region growing procedure is first applied to obtain a connected region that contains the ventricle, and (3) a region trimming procedure is then employed to trim the non-ventricle regions. A hysteretic thresholding is developed for the region growing procedure to cope with the partial volume effect and minimize non-ventricular regions. The domain knowledge on the shape and intensity features of the ventricular system is used for the region trimming procedure. Due to the joint use of the model-based and intensity-based approaches, our method is robust to noise and large shape and size variations. Experiments on 18 simulated and 58 clinical MR images show that the proposed approach is able to segment the ventricular system accurately with the dice similarity coefficient ranging from 91% to 99%.