This work is to develop an intelligent driver-assistance system which can perceive the physiological state of a driver to avoid fatigue driving. The proposed system includes a camera, a wireless ElectroCardioGram (ECG) sensor patch, and a computation platform. The camera in front of a driver is to catch a face image which is processed to obtain features of a mouth for identifying a yawn. The sensor patch records ECG signals which are computed to yield six Heart Rate Variability (HRV) parameters. Seven healthy subjects of 6 males and 1 female had individually driven a car, which was embedded with our system, for 3 hours at a well-known route, mostly in a freeway road. Based on the captured video and measured ECG signals, the correlations between the yawning frequency and six HRV parameters are investigated by using the regression method to discover that the ratio (LF/HF) of Low-Frequency (LF) spectrum power over High-Frequency (HF) spectrum power yields the relatively highest correlation. In order to effectively identity driver's fatigue, the variations of differential LF/HF are further characterized to attain two thresholds which are accompanied with yawning frequencies to build a fair detection mechanism. The practical road tests demonstrate that the proposed system is very feasible and easily adapted to different drivers.