We propose and demonstrate four-channel all-optical switches based on silicon photonic crystal (PhC) nanobeam cavities, with an area of only 31 μm2 per channel. For these switches, signal extraction and rejection functions have been successfully demonstrated while the signal speed is limited to 2.5 Gb/s due to the slow switching recovery process in silicon. In order to improve the signal speed, first, a blue-detuned filtering method is employed to suppress the slow switching recovery tails of the output signal light based on passive resonant devices. The suppressing mechanism relies on the extraction of the large blue-chirped component in the fast rising edge, while suppressing the red-chirped component in the slow switching recovery tail. A detailed theoretical model is established to analyze the improvement mechanism of the switching dynamic characteristics in the silicon PhC nanobeam cavity. Moreover, the simulated and experimental results have both shown that the switching recovery time of the output signal light is greatly reduced from 500 ps to 50 ps. Thus, the processing of 10-Gb/s optical signals has been experimentally demonstrated with the help of the blue-detuned filtering method.