We present a modified coupled wave theory to describe the properties of nonslanted reflection volume diffraction gratings. The method is based on the beta value coupled wave theory, which will be corrected by using appropriate boundary conditions. The use of this correction allows predicting the efficiency of the reflected order for nonslanted reflection gratings embedded in two media with different refractive indices. The results obtained by using this method will be compared to those obtained using a matrix method, which gives exact solutions in terms of Mathieu functions, and also to Kogelnik's coupled wave theory. As will be demonstrated, the technique presented in this paper means a significant improvement over Kogelnik's coupled wave theory.