Cobb angle is the most common quantification of the spine deformity called scoliosis. Recently, automatic Cobb angle estimation has become popular with either semantic segmentation networks or landmark detectors. However, such methods can not perform robustly when some vertebrae have ambiguous appearances in X-ray images. To alleviate the above problem, we propose a multi-task model that simultaneously outputs semantic masks and keypoints of vertebrae. When training this model, we propose a heterogeneous consistency loss function to enhance the consistency between keypoints and semantic masks. Extensive experiments on anterior-posterior (AP) X-ray images from AASCE MICCAI 2019 Challenge demonstrate that our method significantly reduces Cobb angle estimation errors and achieves state-of-the-art performances.Clinical relevance- This work shows that a multi-task model has some potential to measure Cobb angles in more challenging situations, and we can directly integrate it into an auxiliary clinical diagnosis system to assist doctors more effectively for subsequent treatments.