We propose and experimentally demonstrate a radio-frequency digital resolution enhancer (RF-DRE) to mitigate the quantization noise of the RF signal induced by the low-resolution digital-to-analog converter (DAC) in the virtual-carrier-assisted millimeter-wave (mm-wave) signal transmission system. By introducing a bandpass filter (BPF) as the reference for the RF-DRE algorithm, we can design the quantization noise and shape its spectrum inversely to the bandpass filter. By these means, the quantization noise in the target RF frequency range can be effectively mitigated. In the simulation, the bit error rate (BER) of a 4-bit DAC-quantized 16 Gb/s 256QAM signal at 30 GHz is improved from 3.36e-2 to 7.43e-3 by using the RF-DRE. In our experiment, 30 GHz virtual-carrier-assisted mm-wave transmission of 12 Gb/s 64QAM signals over 25 km of standard signal mode fiber (SSMF) is realized. By using the RF-DRE, the BER of a 4-bit DAC-quantized signal can be improved from 6.88e-3 to 1.49e-3, and a 5-bit DAC exhibits a similar performance to an 8-bit DAC without the RF-DRE. Therefore, low-resolution and low-cost DACs can be used for mm-wave signal generation with the help of the proposed RF-DRE.