Free-space all-optical diffractive systems have shown promise for neuromorphic classification of objects without converting light to the electronic domain. While the factors that govern these systems have been studied for coherent light, the fundamental properties for incoherent light have not been addressed, despite the importance for many applications. Here we use a co-design approach to show that optimized systems for spatially incoherent light can achieve performance on par with the best linear electronic classifiers even with a single layer containing few diffractive features. This performance is limited by the inherent linear nature of incoherent optical detection. We circumvent this limit by using a differential detection scheme that achieves greater than 94% classification accuracy on the MNIST dataset and greater than 85% classification accuracy for Fashion-MNIST, using a single layer metamaterial.