We present a hybrid image classifier by feature-sensitive image upconversion, single pixel photodetection, and deep learning, aiming at fast processing of high-resolution images. It uses partial Fourier transform to extract the images' signature features in both the original and Fourier domains, thereby significantly increasing the classification accuracy and robustness. Tested on the Modified National Institute of Standards and Technology handwritten digit images and verified by simulation, it boosts accuracy from 81.25% (by Fourier-domain processing) to 99.23%, and achieves 83% accuracy for highly contaminated images whose signal-to-noise ratio is only -17dB. Our approach could prove useful for fast lidar data processing, high-resolution image recognition, occluded target identification, and atmosphere monitoring.