Current distillation methods only distill between corresponding layers, and do not consider the knowledge contained in preceding layers. To solve this problem, we analyzed the guiding effect of the inferior features of a teacher model on the coordinate feature of a student model, and proposed inferior and coordinate distillation for object detectors. The proposed method utilizes the rich information contained in different layers of the teacher model; such that the student model can review the old information and learn the new information, in addition to the dark knowledge in the teacher model. Moreover, the refine module is used to align the features of different layers, distinguish the spatial and channel to extract attention, strengthen the correlation between the features of different stages, and prevent the disorder caused by merging. Exclusive experiments were conducted on different object detectors. The results for the mean average precision (mAP) obtained using Faster R-CNN, RetinaNet, and fully convolutional one-stage object detector (FCOS) with ResNet-50 as its backbone were 40.5%, 39.8%, and 42.8% with regard to the COCO dataset, respectively; which are 2.1%, 2.4%, and 4.3% higher than the benchmark, respectively.
Keywords: inferior distillation; knowledge distillation; object detection; refine module.