Remote photoplethysmography (rPPG) is an unobtrusive solution to heart rate monitoring in drivers. However, disturbances that occur during driving such as driver behavior, motion artifacts, and illuminance variation complicate the monitoring of heart rate. Faced with disturbance, one commonly used assumption is heart rate periodicity (or spectrum sparsity). Several methods improve stability at the expense of tracking sensitivity for heart rate variation. Based on statistical signal processing (SSP) and Monte Carlo simulations, the outlier probability is derived and ADaptive spectral filter banks (AD) is proposed as a new algorithm which provides an explicable tuning option for spectral filter banks to strike a balance between robustness and sensitivity in remote monitoring for driving scenarios. Moreover, we construct a driving database containing over 23 hours of data to verify the proposed algorithm. The influence on rPPG from driver habits (both amateurs and professionals), vehicle types (compact cars and buses), and routes are also evaluated. In comparison to state-of-the-art rPPG for driving scenarios, the mean absolute error in the Passengers, Compact Cars, and Buses scenarios is 3.43, 7.85, and 5.02 beats per minute, respectively. Moreover, AD also won the top third place in the first challenge on remote physiological signal sensing (RePSS) with relative low computational complexity.