Image encryption has emerged as a method of disguising an image with a noisy or meaningless appearance to prevent its content from being accessed by unauthorized users. We propose an architecture named flexible image encryption and decryption ResNet (FEDResNet) for diffusing an image in end-to-end mode. The architecture consists of an encryption network for diffusing the image and a decryption network for restoring the plaintext image from the diffused image. To enhance the security of the encrypted image, the diffused image is further processed with two optional operations: parallel scrambling and serial diffusion. Two key planes are constructed based on a user-defined key with a chaotic map to control the authority to access images. The structure and parameters of FEDResNet can be shared publicly by different users; hence, it is more flexible and convenient than previous deep-learning-based image encryption methods. A classification network is trained to classify medical images in ciphertext environments. The proposed FEDResNet is trained and tested on the ImageNet data set. Extensive experiments have been performed, and the experimental results suggest that the proposed model can achieve a high level of security with satisfactory efficiency. The experimental results also show that FEDResNet-encrypted images can be classified directly in the ciphertext domain by authorized users as accurately as plaintext images, which is a superior property that is not possessed by traditional image encryption methods.