Chinese olive contains plenty of polyphenols, which possess a wide range of biological actions. In this study, we aimed to investigate the role of the ethyl acetate fraction of Chinese olive fruit extract (CO-EtOAc) in the modulation of lipid accumulation in vitro and in vivo. In cellular studies, CO-EtOAc attenuated oleic acid-induced lipid accumulation; we then elucidated the molecular mechanisms of CO-EtOAc in FL83B mouse hepatocytes. CO-EtOAc suppressed the mRNA levels of fatty acid transporter genes (CD36 and FABP) and lipogenesis genes (SREBP-1c, FAS, and ACC1), but upregulated genes that govern lipolysis (HSL) and lipid oxidation (PPARα, CPT-1, and ACOX). Moreover, CO-EtOAc increased the protein expression of phosphorylated AMPK, ACC1, CPT-1, and PPARα, but downregulated the expression of mature SREBP-1c and FAS. AMPK plays an essential role in CO-EtOAc-mediated amelioration of lipid accumulation. Furthermore, we confirmed that CO-EtOAc significantly inhibited body weight gain, epididymal adipose tissue weight, and hepatic lipid accumulation via regulation of the expression of fatty acid transporter, lipogenesis, and fatty acid oxidation genes and proteins in C57BL/6 mice fed a 60% high-fat diet. Therefore, Chinese olive fruits may have the potential to improve the metabolic abnormalities associated with fatty liver under high fat challenge.