A broadband millimeter-wave (MMW) white noise signal generated by optical heterodyning of two Fabry-Perot laser diodes (FP-LDs) subject to optical feedback is demonstrated and employed for fast physical random bit generation with a simple least significant bits (LSBs) retaining method. Firstly, under suitable feedback conditions, two external-cavity feedback FP-LDs can be easily driven into chaotic states. In this process, the optical spectra of multi-longitudinal modes are significantly broadened. Then, two spectral broadening multi-longitudinal chaotic signals are mixed and converted into an MMW white noise signal through the heterodyne beating technique combined with a fast photodetector. With such an approach, a high dimensional broadband chaos with perfect characteristics of MMW white noise (3-dB bandwidth beyond 50 GHz without any time-delay signature) is experimentally achieved. Finally, taking the generated MMW white noise as the entropy source, 640 Gb/s physical random bit generation is realized by directly selecting 4-LSBs at 160 GS/s sampling rate after an 8-bit analog-digital-convertor.