Small vesicles (sEVs) are closely related to many diseases as they carry various bio-markers. Efficient separation of sEVs from complex biological samples is essential and prerequisite for the following treatment and further disease diagnosis. Here we propose a label-free and biocompatible on-chip magnetic separation system for efficient extraction of sEVs from cell culture supernatant. Through an on-chip ultra-high gradient magnetic field module, a magnetic field gradient close to 100 000 T m-1 is generated inside the separation microchannel. By using fluorescent particles of 200 nm and 1000 nm to simulate sEVs and other bioparticles in a complex sample, the system design and the experimental parameters are optimized. Flow cytometry and a proposed fluorescence intensity analysis method both verify that the recovery rate and purity of 200 nm particles can reach 84.91% and 98.02%, respectively. Then, a biocompatible ferrofluid is utilized in the separation system to separate sEVs from the cell culture supernatant. The results tested by nanoparticle tracking analysis show that the recovery rate and purity of sEVs are 85.80% and 80.45%, superiorly exceeding the performance that the ultracentrifugation method can provide.