In view of the complexity of port ship supervision and the influence of external factors such as electromagnetic interference in harbor-border inspection, an efficient system combining an unmanned aerial vehicle (UAV) and visible light positioning (VLP) is proposed for locating maritime targets. In this system, a rotatable receiver with five photodetectors (PDs) installed obliquely on UAV is designed for expanding the positioning range and allowing a lower flight altitude. On this basis, we propose the Chan-Taylor (CT) method based on time difference of arrival (TDOA) for target positioning. First, the localization problem is reformulated as a weighted least squares (WLS) problem and provides a good initial estimate via the two-step WLS (TWLS) method. Then, based on Taylor expansion of TDOA equations, estimated error is calculated using the initial estimate, which can correct the estimated position of the target iteratively. To offset the error, weighted centroid CT (WCCT) is proposed by endowing different weights based on error difference to estimated results. For further improving accuracy, a restricted-region fingerprinting positioning based on CT (CT-RFP) is proposed. In restricted area determined by CT, a certain number of fingerprints is generated based on received signal strength (RSS) for matching. Simulation results show that CT is significantly improved over the previous methods. Compared with TWLS, the accuracy of CT is improved by 49.71%. For WCCT, the maximum error is reduced from 8.65 to 6.91 cm, which effectively reduces the influence of error. Moreover, CT-RFP can achieve an accuracy within millimeter level via the appropriate number of fingerprints and ensemble runs of CT, even at high noise levels.