Quantum ghost imaging offers many advantages over classical imaging, including the ability to probe an object with one wavelength and record the image with another (non-degenerate ghost imaging), but suffers from slow image reconstruction due to sparsity and probabilistic arrival positions of photons. Here, we propose a two-step deep learning approach to establish an optimal early stopping point based on object recognition, even for sparsely filled images. In step one we enhance the reconstructed image after every measurement by a deep convolutional auto-encoder, followed by step two in which a classifier is used to recognise the image. We test this approach on a non-degenerate ghost imaging setup while varying physical parameters such as the mask type and resolution. We achieved a fivefold decrease in image acquisition time at a recognition confidence of [Formula: see text]. The significant reduction in experimental running time is an important step towards real-time ghost imaging, as well as object recognition with few photons, e.g., in the detection of light sensitive structures.