The difference analysis of spectrum among tree species and the improvement of classification algorithm are the difficult points of extracting tree species information using remote sensing images, and are also the keys to improving the accuracy in the tree species information extraction in farmland returned to forests area. TM images were selected in this study, and the spectral indexes that could distinguish tree species information were filtered by analyzing tree species spectrum. Afterwards, the information of tree species was extracted using improved support vector machine algorithm. Although errors and confusion exist, this method shows satisfying results with an overall accuracy of 81.7%. The corresponding result of the traditional method is 72.5%. The method in this paper can achieve a more precise information extraction of tree species and the results can meet the demand of accurate monitoring and decision-making. This method is significant to the rapid assessment of project quality.