An out-of-plane angular detection scheme with random phase encoding is proposed. A ground glass is attached on a rod, which is rotated around a center point, so that the rotation of the rod induces the displacement of the ground glass in a circular path. To enhance the rotational sensitivity we adjust the Bragg selectivity of the volume holographic optical element encoded by random phase. Therefore, the rotational sensitivity can be tuned over a large range from several degrees to ten thousandth degrees by changing the radius of rotation. The theoretical calculation, as well as experiment, is demonstrated.