We study the use of random nanocomposite material as a photomask absorber layer for the next generation of extreme ultraviolet (EUV) lithography. By introducing nickel nanoparticles (NPs) randomly into a TaN host, the nanocomposite absorber layer can greatly reduce the reflectivity as compared with the standard TaN layer of the same thickness. Finite integral simulations show that the reduction in the reflectivity is mainly due to the enhanced absorption by the Ni NPs. The fluctuation in reflectivity induced by scattering and random position of the NPs is found to be on the order of 0.1%. Based on these observations, we build an effective medium model for the nanocomposite absorber layer and use the transfer matrix method to identify optimal absorber designs that utilize cavity effects to reduce the required volume fraction of Ni NPs. We further perform a process simulation and show that our approach can greatly reduce the HV bias in the lithography process.