A novel frequency domain nonlinear compensation method, FD-NC, is proposed for orthogonal frequency division multiplexing (OFDM) based visible light communication (VLC) system. By tackling the memory nonlinear impairments from light emitting diodes (LEDs) in the frequency domain rather than in the time domain, the proposed method has much lower computational complexity than the conventional time domain Volterra nonlinear compensation method (TD-NC). Both theoretical derivation and experimental investigation of the proposed method in OFDM based VLC systems with four types of commercial LEDs are presented. The results of experiments show that the proposed low-complexity FD-NC method with a moderate truncation factor achieves a performance comparable to that of the TD-NC. The application of FD-NC method in the bit-power loading OFDM VLC system is also experimentally demonstrated. Compared with the linear equalization case, at a bit error rate (BER) of 3.8 × 10-3 (a), the transmission distance of a 960 Mbps VLC system can be extended from 0.7 m to 1.8 m by the FD-NC, and (b) the achievable system capacity can be enhanced by 18.7%~36.5% for transmission distance in the range of 0.5 m~2 m with the FD-NC. The complexity analysis shows that the required number of real-valued multiplications (RNRM) of the FD-NC is independent of linear or nonlinear memory length. The reduction of RNRM achieved by the FD-NC over the TD-NC becomes more profound for a larger nonlinear memory length or a smaller truncation factor.