A polarization beam-splitting multimode filter using pixelated waveguides has been presented and experimentally demonstrated in this paper. Finite difference time domain method and direct binary search optimization algorithm are employed to optimize pixelated waveguides to realize compact size, broad bandwidth, large extinction ratio, low insertion loss, and good polarization extinction ratio. Measurement results show that, in a wavelength range from 1520 to 1560 nm, for the fabricated device working at transverse-electric polarization, the measured insertion loss is less than 1.23 dB and extinction ratio is larger than 15.14 dB, while for transverse-magnetic polarization, the corresponding insertion loss lower than 0.74 dB and extinction ratio greater than 15.50 dB are realized. The measured polarization extinction ratio larger than 15.02 dB is achieved. The device's length is only 15.4 µm.