In order to achieve housing automatic detection from high-resolution aerial imagery, the present paper utilized the color information and spectral characteristics of the roofing material, with the image segmentation theory, to study the housing automatic detection method. Firstly, This method proposed in this paper converts the RGB color space to HIS color space, uses the characteristics of each component of the HIS color space and the spectral characteristics of the roofing material for image segmentation to isolate red tiled roofs and gray cement roof areas, and gets the initial segmentation housing areas by using the marked watershed algorithm. Then, region growing is conducted in the hue component with the seed segment sample by calculating the average hue in the marked region. Finally through the elimination of small spots and rectangular fitting process to obtain a clear outline of the housing area. Compared with the traditional pixel-based region segmentation algorithm, the improved method proposed in this paper based on segment growing is in a one-dimensional color space to reduce the computation without human intervention, and can cater to the geometry information of the neighborhood pixels so that the speed and accuracy of the algorithm has been significantly improved. A case study was conducted to apply the method proposed in this paper to high resolution aerial images, and the experimental results demonstrate that this method has a high precision and rational robustness.