We design and realize an arrival time diagnostic for ultrashort X-ray pulses achieving unprecedented high sensitivity in the soft X-ray regime via cross-correlation with a ≈1550 nm optical laser. An interferometric detection scheme is combined with a multi-layer sample design to greatly improve the sensitivity of the measurement. We achieve up to 275% of relative signal change when exposed to 1.6 mJ/cm2 of soft X-rays at 530 eV, more than a hundred-fold improvement in sensitivity as compared to previously reported techniques. The resolution of the arrival time measurement is estimated to around 2.8 fs (rms). The demonstrated X-ray arrival time monitor paves the way for sub-10 fs-level timing jitter at high repetition rate X-ray facilities.