Understanding stray light (SL) is a crucial aspect in the development of high-end optical instruments, for instance space telescopes. As it drives image quality, SL must be controlled by design and characterized experimentally. However, conventional SL characterization methods are limited as they do not provide information on its origins. The problem is complex due to the diversity of light interaction processes with surfaces, creating various SL contributors. Therefore, when SL level is higher than expected, it can be difficult to determine how to improve the system. We demonstrate a new approach, ultrafast time-of-flight SL characterization, where a pulsed laser source and a streak camera are used to record individually SL contributors which travel with a specific optical path length. Furthermore, the optical path length offers a means of identification to determine its origin. We demonstrate this method in an imaging system, measuring and identifying individual ghosts and scattering components. We then show how it can be used to reverse-engineer the instrument SL origins.