We present a method to measure the optical torque applied to particles of arbitrary shape such as micrometer-sized micro-organisms or cells held in an optical trap, inferred from the change of angular momentum of light induced by the particle. All torque components can be determined from a single interference pattern recorded by a camera in the back focal plane of a high-NA condenser lens provided that most of the scattered light is collected. We derive explicit expressions mapping the measured complex field in this plane to the torque components. The required phase is retrieved by an iterative algorithm, using the known position of the optical traps as constraints. The torque pertaining to individual particles is accessible, as well as separate spin or orbital parts of the total torque.