Manipulating magnetism by electric current is of great interest for both fundamental and technological reasons. Much effort has been dedicated to spin-orbit torques (SOTs) in metallic structures, while quantitative investigation of analogous phenomena in magnetic insulators remains challenging due to their low electrical conductivity. Here we address this challenge by exploiting the interaction of light with magnetic order, to directly measure SOTs in both metallic and insulating structures. The equivalency of optical and transport measurements is established by investigating a heavy-metal/ferromagnetic-metal device (Ta/CoFeB/MgO). Subsequently, SOTs are measured optically in the contrasting case of a magnetic-insulator/heavy-metal (YIG/Pt) heterostructure, where analogous transport measurements are not viable. We observe a large anti-damping torque in the YIG/Pt system, revealing its promise for spintronic device applications. Moreover, our results demonstrate that SOT physics is directly accessible by optical means in a range of materials, where transport measurements may not be possible.