As being radiation-free, portable, and capable of repetitive use, ultrasonography is playing an important role in diagnosing and evaluating the COVID-19 Pneumonia (PN) in this epidemic. By virtue of lung ultrasound scores (LUSS), lung ultrasound (LUS) was used to estimate the excessive lung fluid that is an important clinical manifestation of COVID-19 PN, with high sensitivity and specificity. However, as a qualitative method, LUSS suffered from large interobserver variations and requirement for experienced clinicians. Considering this limitation, we developed a quantitative and automatic lung ultrasound scoring system for evaluating the COVID-19 PN. A total of 1527 ultrasound images prospectively collected from 31 COVID-19 PN patients with different clinical conditions were evaluated and scored with LUSS by experienced clinicians. All images were processed via a series of computer-aided analysis, including curve-to-linear conversion, pleural line detection, region-of-interest (ROI) selection, and feature extraction. A collection of 28 features extracted from the ROI was specifically defined for mimicking the LUSS. Multilayer fully connected neural networks, support vector machines, and decision trees were developed for scoring LUS images using the fivefold cross validation. The model with 128×256 two fully connected layers gave the best accuracy of 87%. It is concluded that the proposed method could assess the ultrasound images by assigning LUSS automatically with high accuracy, potentially applicable to the clinics.