A cyphertext-only attack (COA) on a joint transform correlator encryption system is proposed. Under the proposed COA scheme, the energy spectral density of plaintext can be calculated with cyphertext, and then plaintext information can be recovered by using a phase retrieval algorithm such as the hybrid input-output algorithm. We also numerically and experimentally demonstrate the COA to verify feasibility of the proposed technique.