The metal artifact reduction (MAR) algorithm is used in most CBCT unit to reduce artifact from various dental materials. The performance of MAR program of a CBCT unit according to the dental material type under different imaging mode was evaluated as introducing automatic quantification of the amount of artifact reduced. Four customized phantoms with different dental prostheses (amalgam, gold, porcelain-fused-metal, zirconia) underwent CBCT scanning with and without the MAR option. The imaging was performed under varied scanning conditions; 0.2 and 0.3 mm3 voxel sizes; 70 and 100 kVp. The amount of artifacts reduced by each prosthesis and scanning mode automatically counted using canny edge detection in MATLAB, and statistical analysis was performed. The overall artifact reduction ratio was ranged from 17.3% to 55.4%. The artifact caused by the gold crown was most effectively reduced compared to the other prostheses (p < 0.05, Welch's ANOVA analysis). MAR showed higher performance in smaller voxel size mode for all prostheses (p < 0.05, independent t-test). Automatic quantification efficiently evaluated MAR performance in CBCT image. The impact of MAR was different according to the prostheses type and imaging mode, suggesting that thoughtful consideration is required when selecting the imaging mode of CBCT.