Human-machine interfaces (HMIs) based on Electro-oculogram (EOG) signals have been widely explored. However, due to the individual variability, it is still challenging for an EOG-based eye movement recognition model to achieve favorable results among cross-subjects. The classical transfer learning methods such as CORrelation Alignment (CORAL), Transfer Component Analysis (TCA), and Joint Distribution Adaptation (JDA) are mainly based on feature transformation and distribution alignment, which do not consider similarities/dissimilarities between target subject and source subjects. In this paper, the Kullback-Leibler (KL) divergence of the log-Power Spectral Density (log-PSD) features of horizontal EOG (HEOG) between the target subject and each source subject is calculated for adaptively selecting partial subjects that suppose to have similar distribution with target subject for further training. It not only consider the similarity but also reduce computational consumption. The results show that the proposed approach is superior to the baseline and classical transfer learning methods, and significantly improves the performance of target subjects who have poor performance with the primary classifiers. The best improvement of Support Vector Machines (SVM) classifier has improved by 13.1% for subject 31 compared with baseline result. The preliminary results of this study demonstrate the effectiveness of the proposed transfer framework and provide a promising tool for implementing cross-subject eye movement recognition models in real-life scenarios.