Secure key distribution (SKD) schemes based on the interaction between a broadband chaotic source and the reciprocity of a fiber channel exhibit reliable security and a high key generation rate (KGR). However, under the intensity modulation and direct detection (IM/DD) architecture, these SKD schemes cannot achieve a long distribution distance due to the limitations on the signal-to-noise ratio (SNR) and the receiver's sensitivity. Here, based on the advantage of the high sensitivity of coherent reception, we design a coherent-SKD structure where orthogonal polarization states are locally modulated by a broadband chaotic signal and the single-frequency local oscillator (LO) light is transmitted bidirectionally in the optical fiber. The proposed structure not only utilizes the polarization reciprocity of optical fiber but also largely eliminates the non-reciprocity factor, which can effectively extend the distribution distance. The experiment realized an error-free SKD with a transmission distance of 50 km and a KGR of 1.85 Gbit/s.