In this paper, an optical color single-channel asymmetric cryptosystem based on the non-negative matrix factorization (NMF) and a face biometric in cyan-magenta-yellow-black (CMYK) space is proposed. To the best of our knowledge, this is the first time that NMF has been introduced into optical color image encryption. In the proposed cryptosystem, the color image in CMYK space is first decomposed into four color channels: C, M, Y, and K. By performing NMF operations on the four color channels, the four basic and sparse matrices can be obtained, respectively, which achieves asymmetry and saves computational resources. The four basis matrices can be used as private keys, and the four coefficient matrices are synthesized by the inverse discrete wavelet transform for subsequent encryption. Finally, the synthesized image is encoded with double random phase encoding based on phase truncation (PT). Compared with the existing PT-based cryptosystems, our cryptosystem can improve security against a special attack. In addition, the chaotic random phase mask is generated by a face biometric, which is noncontact and unique. Numerical simulation results are shown to verify the feasibility and robustness of our cryptosystem. Further, the proposed cryptosystem can be extended to encrypt multiple images conveniently.