Based on an optical gyrator transform system, an image encryption algorithm is designed and studied. An original secret image is regarded as the output intensity of the second gyrator transform. A coherent nonuniform optical beam is converted into the input of the first gyrator transform. A Gerchberg-Saxton phase retrieval algorithm is employed for obtaining the compensation phases in the first gyrator transform pair. The compensation phases are regarded as the encrypted image and key in this algorithm. The parameters of the laser beam and gyrator transform can serve as the additional key of encryption method. The decryption process of this encryption algorithm can be achieved with an optical system. Numerical simulations are performed to test the validity and capability of the encryption algorithm.