Secure key distribution (SKD) schemes based on fiber channel reciprocity provide information-theoretic security as well as a simple symmetric structure. However, the nonlinear effects and backscattering effects introduced during the bidirectional transmission process degrade the channel reciprocity. Recent unidirectional SKD schemes avoid non-reciprocal factors but require additional negotiation mechanisms to aggregate the transmitter and receiver data. Here, we propose a unidirectional SKD scheme based on channel physical intrinsic property and polarization reciprocity. The designed loopback structure constructs asymmetry between legitimate and illegitimate parties while aggregating data. The deployment of a broadband chaotic entropy source significantly improves the key generation rate (KGR). In the experiment, the KGR reaches 17.5 Gb/s, and the distribution distance reaches 100 km.