We propose an absorption intensity heartbeat modulation-averaged shifted histogram (AIHM-ASH) method for estimating human heart rate (HR) using color videos of lip image sequences. When heartbeat occurs, AIHM is generated. Based on the AIHM, HR signals can be demodulated by computing the instantaneous HR modulation depth that presents the relative red blood cell (RBC) concentration from the green channel image of the RGB color video. In addition, the ASH algorithm further suppresses the background tissue and vein signals, and increases the signal-to-noise ratio (SNR). The experimental results for flow phantoms, chicken embryos, and human lips validated the proposed method's optimal estimation conditions and effectiveness, where the accuracy and root mean square error (RMSE) were 99.23% and 0.8 bpm, respectively. The proposed HR estimation method has significant potential to advance health monitoring and disease prevention via conventional color video cameras installed in public places.