Coal gangue (CG) as mineral waste was properly treated and applied as the filter media in permeable pavement systems due to its good sorption ability and mechanical strength. Batch experiments show the maximum adsorption capacity of calcined CG to phosphorus could reach 2.63 mg/g. To evaluate the removal effect of typical runoff pollutants including chemical oxygen demand (COD), total nitrogen (TN), total phosphorus (TP) and ammonia (NH4+-N), the gravel and sand in a traditional pavement system were replaced by CG respectively. The leaching behavior of the four pollutants in CG modified systems is limited and comparable with traditional system, indicating pretreated CG as filler would not cause environmental risk. CG-based pavement systems improved the removal efficiency of the four pollutants especially for TP. The removal mechanisms including interception, adsorption and microorganism degradation. The removal rates of COD, NH4+-N and TN by CG modified and traditional systems decreased with rainfall duration, while it is not obviously changed with rainfall recurrence period and drying period. Overall, the permeable pavement with CG layers that replaced both sand and gravel layers show best removal efficiency of all pollutants investigated especially for TP (>95%). This study provides a new way for CG utilization and gives the reference for the process design of permeable pavement.