Extraction of phenolics from Idesia polycarpa defatted fruit residue was optimized by the maximization of the yield in total phenolics, using the response surface methodology. The optimized conditions were 50% ethanol, 5 h extraction time, 1 : 40 liquid to solid ratio, and 80°C extraction temperature. The experimental average total phenolics yield was 54.49 ± 4.26 mg/g. These antioxidant properties of phenolics were comprehensively analyzed for the first time. All the extracts not only demonstrated the significant free radical scavenging activities and metal chelating activity but also inhibited lipid, lipoprotein peroxidation and revealed reducing power activity. Ethyl acetate extraction (EAE) also inhibited mushroom tyrosinase activity and significantly increased the average skin-whitening index (L value) of the skin of C57BL/6 mice, indicating its potential use for skin hyperpigmentation in humans. The results of cell experiments showed EAE could strongly inhibit cellular tyrosinase activity, which had led to the decrease of melanogenesis in B16 mouse melanoma cells. Overall, EAE is an excellent natural antioxidant and depigmenting agent, which can be developed as a new food additive, medicine, and cosmetic.