Infrared imaging has been widely used in the field of sea surface monitoring. Horizon detection is a key step before a target's detection, locating, and tracking in the sea-sky infrared scene. Reducing processing time while ensuring accuracy is the research focus of infrared horizon detection. This paper proposes a novel method of a line segment detector (LSD) algorithm with gradient direction filtering. First, the rough extraction of the sea-sky region is used to limit the size of the detected image, and then the potential horizon line segment is extracted, applying the improved LSD algorithm in the sea-sky region, which probably contains many false extraction results. Then, gradient direction filtering is designed to pick the horizon line segments in this step. Finally, the horizon line segments are stitched to obtain the whole horizon line based on random sample consensus. The results of the comparative experiments show that this novel method has high detection accuracy, and the processing time is significantly shortened; what is more, we can also conclude that this method has a good performance on the detection stability.