Optical cryptosystems combined with compressed sensing can achieve compression and encryption simultaneously. But they usually use the same measurement matrix to sample all blocks of an image, which makes it easy to estimate the measurement matrix in the chosen plaintext attack. In this paper, we propose a robust scheme adopting multiple measurement matrices to overcome this shortcoming. The matrices can be efficiently derived by applying random row exchanging to a basic one, which is also encoded into the fractional Fourier transform (FrFT) domain to improve the visual effect of wrongly decrypted images. Chaos-based pixel scrambling is added into our double FrFT cryptosystem to guarantee its nonlinearity. Simulation results have shown the security and effectiveness of our scheme.