In view of the existing method having a great subjectivity for the lunar edge selection, we propose an automatic knife-edge detection method based on the Hough transform to evaluate the on-orbit modulation transfer function (MTF) of the satellite remote sensor. This novel method avoids the dependence of the on-orbit MTF measurement on the edge selection location, overcoming the limitation of the traditional method needing to identify the lunar shape and fix its edge area. On basis of two different edge detection algorithms (Sobel operator and Prewitt operator), the binary edge images of the moon are acquired, thus obtaining a series of edges satisfying the determination requirement by the Hough transform, and the MTFs corresponding to each knife-edge are calculated to obtain the optimal MTF. The automatic knife-edge detection method greatly improves the accuracy of the lunar edge selection, and the MTF obtained by the novel method is obviously better than that of the traditional method. In order to verify the effectiveness of the novel method, the long time series of the on-orbit MTFs for the FY-2G and FY-2E satellite measurements are given, indicating that the FY-2E observation has higher stability and better performance compared with that of the FY-2G satellite. This study has an important practical significance for evaluating the on-orbit stability of the satellite its optical imaging quality.