The grain number on the rice panicle, which directly determines the rice yield, is a very important agronomic trait in rice breeding and yield-related research. However, manual counting of grain number per rice panicle is time-consuming, error-prone, and laborious. In this study, a novel prototype, dubbed the "GN-System," was developed for the automatic calculation of grain number per rice panicle based on a deep convolutional neural network. First, a whole panicle grain detection (WPGD) model was established using the Cascade R-CNN method embedded with the feature pyramid network for grain recognition and location. Then, a GN-System integrated with the WPGD model was developed to automatically calculate grain number per rice panicle. The performance of the GN-System was evaluated through estimated stability and accuracy. One hundred twenty-four panicle samples were tested to evaluate the estimated stability of the GN-System. The results showed that the coefficient of determination (R2) was 0.810, the mean absolute percentage error was 8.44%, and the root mean square error was 16.73. Also, another 12 panicle samples were tested to further evaluate the estimated accuracy of the GN-System. The results revealed that the mean accuracy of the GN-System reached 90.6%. The GN-System, which can quickly and accurately predict the grain number per rice panicle, can provide an effective, convenient, and low-cost tool for yield evaluation, crop breeding, and genetic research. It also has great potential in assisting phenotypic research.