This study proposes an encryption scheme combining cellular automata (CA) and DNA encoding to improve security of a coherent optical orthogonal frequency division multiplexing (CO-OFDM) system, wherein key sequences are generated with good randomness and unpredictability by a 4-dimensional hyper-chaotic system. A base scrambling pseudo random binary sequence (PRBS) generated by the CA is introduced, which results in better scrambling effect and randomness in the conventional complex DNA encoding. The randomness, complexity and security of the system is enhanced due to 6 variable keys (key space of ∼10138). An experiment conducted in a 40 GHz 16QAM CO-OFDM system over an 80 km standard single mode fiber (SSMF) shows that the authorized user can successfully decrypt the received signal, while the eavesdroppers cannot derive useful information with bit error rate (BER) at approximately 0.5. An allowable optical signal to noise ratio (OSNR) penalty of 0.5 dB will be introduced to achieve same BER before and after encryption due to the error propagation of cellular automata.