Detection of defects including cracks and spalls on wall surface in high-rise buildings is a crucial task of buildings' maintenance. If left undetected and untreated, these defects can significantly affect the structural integrity and the aesthetic aspect of buildings. Timely and cost-effective methods of building condition survey are of practicing need for the building owners and maintenance agencies to replace the time- and labor-consuming approach of manual survey. This study constructs an image processing approach for periodically evaluating the condition of wall structures. Image processing algorithms of steerable filters and projection integrals are employed to extract useful features from digital images. The newly developed model relies on the Support vector machine and least squares support vector machine to generalize the classification boundaries that categorize conditions of wall into five labels: longitudinal crack, transverse crack, diagonal crack, spall damage, and intact wall. A data set consisting of 500 image samples has been collected to train and test the machine learning based classifiers. Experimental results point out that the proposed model that combines the image processing and machine learning algorithms can achieve a good classification performance with a classification accuracy rate = 85.33%. Therefore, the newly developed method can be a promising alternative to assist maintenance agencies in periodic building surveys.