On-chip optical group-velocity dispersion (GVD) is highly desired for a wide range of signal processing applications, including low-latency and low-power-consumption dispersion compensation of telecommunication data signals. However, present technologies, such as linearly chirped waveguide Bragg gratings (LCWBGs), employ spectral phase accumulation along the frequency spectrum. To achieve the needed specifications in most applications, this strategy requires device lengths that are not compatible with on-chip integration while incurring in relatively long processing latencies. Here, we demonstrate a novel design strategy that utilizes a discretized and bounded spectral phase filtering process to emulate the continuous spectral phase variation of a target GVD line. This leads to a significant reduction of the resulting device length, enabling on-chip integration and ultra-low latencies. In experiments, we show GVD compensation of both NRZ and PAM4 data signals with baud rates up to 24 GBd over a 31.12-km fibre-optic link using a 4.1-mm WBG-based on-chip phase filter in a silicon-on-insulator (SOI) platform, at least 5× shorter compared to an equivalent LCWBG, reducing the processing latency down to ∼ 100 ps. The bandwidth of the mm-long device can be further extended to the THz range by employing a simple and highly efficient phase-only sampling of the grating profile. The proposed solution provides a promising route toward a true on-chip realization of a host of GVD-based all-optical analog signal processing functionalities.