The image encryption system based on joint transform correlation has attracted much attention because its ciphertext does not contain complex value and can avoid strict pixel alignment of ciphertext when decryption occurs. This paper proves that the joint transform correlation architecture is vulnerable to the attack of the deep learning method-convolutional neural network. By giving the convolutional neural network a large amount of ciphertext and its corresponding plaintext, it can simulate the key of the encryption system. Unlike the traditional method which uses the phase recovery algorithm to retrieve or estimate optical encryption key, the key model trained in this paper can directly convert the ciphertext to the corresponding plaintext. Compared with the existing neural network systems, this paper uses the sigmoid activation function and adds dropout layers to make the calculation of the neural network more rapid and accurate, and the equivalent key trained by the neural network has certain robustness. Computer simulations prove the feasibility and effectiveness of this method.