Recently developed image-free sensing techniques have achieved remarkable performance in various vision tasks. However, existing image-free methods still cannot simultaneously obtain the category, location, and size information of all objects. In this Letter, we report a novel image-free single-pixel object detection (SPOD) technique. SPOD enables efficient and robust multi-object detection directly from a small number of measurements, eliminating the requirement for complicated image reconstruction. Different from the conventional full-size pattern sampling method, the reported small-size optimized pattern sampling method achieves higher image-free sensing accuracy with fewer pattern parameters (∼1 order of magnitude). Moreover, instead of simply stacking CNN layers, we design the SPOD network based on the transformer architecture. It can better model global features and reinforce the network's attention to the targets in the scene, thus improving the object detection performance. We demonstrate the effectiveness of SPOD on the Voc dataset, which achieves a detection accuracy of 82.41% mAP at a sampling rate of 5% with a refresh rate of 63 f.p.s.