Infrared thermography (IRT) has been used to screen febrile passengers in international airports for over a decade. However, fever-based infection screening using IRT suffered from low sensitivity because measurements can be affected by ambient temperature, humidity, etc. In our previous study, we proposed an RGB-thermal image fusion system to measure vital signs i.e., the RGB camera detects tiny changes in color from facial skin, associated with blood flow, to estimate heart rate, and IRT senses temperature changes around the nasal area, caused by respiration, to measure respiratory rate). The inclusion of heart and respiratory rates lead to increased screening accuracy. In the present study, to promote the widespread use of our system in real-world settings, a face detection and tracking method was developed and implemented into the system, thereby enabling the accurate and stable measurement of vital signs. We assessed heart and respiratory rate estimation via an RGB-thermal image fusion system using Bland-Altman plots and statistical analysis.