Semimetals are being explored for their unique advantages in low-energy high-speed photodetection, although they suffer from serious drawbacks such as an intrinsically high dark current. In this Perspective, we envision the exploitation of topological effects in the photoresponse of these materials as a promising route to circumvent these problems. We overview recent studies on photodetection based on graphene and other semimetals, and further discuss the opportunities created by topological effects, along with the additional challenges that they impose on photodetector designs.