We aimed to develop a computer-aided diagnostic system (CAD) for predicting colorectal polyp histology using deep-learning technology and to validate its performance. Near-focus narrow-band imaging (NBI) pictures of colorectal polyps were retrieved from the database of our institution. Of these, 12480 image patches of 624 polyps were used as a training set to develop the CAD. The CAD performance was validated with two test datasets of 545 polyps. Polyps were classified into three histological groups: serrated polyp (SP), benign adenoma (BA)/mucosal or superficial submucosal cancer (MSMC), and deep submucosal cancer (DSMC). The overall kappa value measuring the agreement between the true polyp histology and the expected histology by the CAD was 0.614-0.642, which was higher than that of trainees (n = 6, endoscopists with experience of 100 NBI colonoscopies in <6 months; 0.368-0.401) and almost comparable with that of the experts (n = 3, endoscopists with experience of 2,500 NBI colonoscopies in ≥5 years) (0.649-0.735). The areas under the receiver operating curves for CAD were 0.93-0.95, 0.86-0.89, and 0.89-0.91 for SP, BA/MSMC, and DSMC, respectively. The overall diagnostic accuracy of the CAD was 81.3-82.4%, which was significantly higher than that of the trainees (63.8-71.8%, P < 0.01) and comparable with that of experts (82.4-87.3%). The kappa value and diagnostic accuracies of the trainees improved with CAD assistance: that is, the kappa value increased from 0.368 to 0.655, and the overall diagnostic accuracy increased from 63.8-71.8% to 82.7-84.2%. CAD using a deep-learning model can accurately assess polyp histology and may facilitate the diagnosis of colorectal polyps by endoscopists.