To improve the receiver sensitivity of the digitized radio-over-fiber (DRoF) transmission system, a vector quantization scheme based on probabilistic quantization codeword shaping (PQCS) is proposed. The PQCS performs quantization bits (QBs) rematching on the original codebook to optimize the proportion distribution of level '0' and level '2' in 4-Pulse Amplitude Modulation (PAM-4) for improving system sensitivity. A 16-Quadrature Amplitude Modulation (16-QAM) DRoF transmission system using intensity-modulation/direct-detection is employed to experimentally verify the proposed scheme. The experimental results indicate that, compared to the traditional vector quantization scheme, the PQCS method offers 1.45 dB shaping gain for system sensitivity at a bit error rate (BER) of 5 × 10-4. Nevertheless, the error vector magnitude (EVM) can be obtained below 2% when 6 and 7 QBs are adopted for 64-QAM and 256-QAM, respectively.