Memristor-enabled in-memory computing provides an unconventional computing paradigm to surpass the energy efficiency of von Neumann computers. Owing to the limitation of the computing mechanism, while the crossbar structure is desirable for dense computation, the system's energy and area efficiency degrade substantially in performing sparse computation tasks, such as scientific computing. In this work, we report a high-efficiency in-memory sparse computing system based on a self-rectifying memristor array. This system originates from an analog computing mechanism that is motivated by the device's self-rectifying nature, which can achieve an overall performance of ~97 to ~11 TOPS/W for 2- to 8-bit sparse computation when processing practical scientific computing tasks. Compared to previous in-memory computing system, this work provides over 85 times improvement in energy efficiency with an approximately 340 times reduction in hardware overhead. This work can pave the road toward a highly efficient in-memory computing platform for high-performance computing.