Light as an external stimulus can be precisely manipulated in terms of irradiation time, site, wavelength, and density. As such, photoresponsive drug/gene delivery systems have been increasingly pursued and utilized for the spatiotemporal control of drug/gene delivery to enhance their therapeutic efficacy and safety. In this review, we summarized the recent research progress on photoresponsive drug/gene delivery, and two major categories of delivery systems were discussed. The first category is the direct responsive systems that experience photoreactions on the vehicle or drug themselves, and different materials as well as chemical structures responsive to UV, visible, and NIR light are summarized. The second category is the indirect responsive systems that require a light-generated mediator signal, such as heat, ROS, hypoxia, and gas molecules, to cascadingly trigger the structural transformation. The future outlook and challenges are also discussed at the end.