This paper proposes a new computer-aided method for the skin lesion classification applicable to both melanocytic skin lesions (MSLs) and nonmelanocytic skin lesions (NoMSLs). The computer-aided skin lesion classification has drawn attention as an aid for detection of skin cancers. Several researchers have developed methods to distinguish between melanoma and nevus, which are both categorized as MSL. However, most of these studies did not focus on NoMSLs such as basal cell carcinoma (BCC), the most common skin cancer and seborrheic keratosis (SK) despite their high incidence rates. It is preferable to deal with these NoMSLs as well as MSLs especially for the potential users who are not enough capable of diagnosing pigmented skin lesions on their own such as dermatologists in training and physicians with different expertise. We developed a new method to distinguish among melanomas, nevi, BCCs, and SKs. Our method calculates 828 candidate features grouped into three categories: color, subregion, and texture. We introduced two types of classification models: a layered model that uses a task decomposition strategy and flat models to serve as performance baselines. We tested our methods on 964 dermoscopy images: 105 melanomas, 692 nevi, 69 BCCs, and 98 SKs. The layered model outperformed the flat models, achieving detection rates of 90.48%, 82.51%, 82.61%, and 80.61% for melanomas, nevi, BCCs, and SKs, respectively. We also identified specific features effective for the classification task including irregularity of color distribution. The results show promise for enhancing the capability of the computer-aided skin lesion classification.