The constraints on directions of incoming and outgoing rays of Risley prisms, caused by total internal reflection and surface tilt, are investigated for the four typical configurations. After applying a nonparaxial ray-tracing method based on the vector form Snell's law, the incident angles at prisms' surfaces are calculated and compared with the critical angle. On this basis, the direction limitations of incoming and outgoing rays are investigated. The permissible incoming directions as well as the achievable outgoing directions depend on prisms' orientation. Their ranges over which the rays can pass through the system at any prism orientation are defined as the angular aperture and angular field of view (FOV). The effects of the prisms' refractive index, opening angle, and arrangement on angular aperture/FOV are discussed. It is shown that there exists a direct trade-off between the deviation power and the angular aperture/FOV for Risley prisms, that is, high refractive index and large opening angle yield a smaller angle aperture/FOV. Large angular aperture can be achieved by employing the 2121 configuration, while it is desirable to adopt a 1212 configuration to obtain large angular FOV. The research can afford guidance for prism material, geometry choices, and configuration setting in the design of a Risley prism system for wide-angle beam steering or imaging adjustment.