In this work, we investigate the performance of an ambiguity function-shaped waveform (AFSW) using a millimeter-wave photonics-based radar system at 100 GHz. An AFSW is a radar waveform whose ambiguity function can be shaped to increase the peak-to-sidelobe ratio (PSR) for better detectability of targets in a desired range/velocity region. To the best of the authors' knowledge, this paper is the first in the literature that investigates the performance of such a waveform in a photonics-based radar system. We experimentally compare the AFSW performance to the conventional frequency-modulated continuous wave (FMCW). The experimental results show the ability of the AFSW to achieve a PSR of 38.35 dB compared to the PSR of 14.5 dB obtained using the conventional FMCW. Moreover, we investigate the effects of some optical system impairments on the AFSW, such as: (i) optical modulator nonlinearity, (ii) optical modulator bias drift, and (iii) sampling offset error between the transmitter and receiver.