In this work, we propose a non-contact video-based approach that detects when an individual's skin temperature is elevated beyond the normal range. The detection of elevated skin temperature is critical as a diagnostic tool to infer the presence of an infection or an abnormal health condition. Detection of elevated skin temperature is typically achieved using contact thermometers or non-contact infrared-based sensors. The ubiquity of video data acquisition devices such as mobile phones and computers motivates the development of a binary classification approach, the Video-based TEMPerature (V-TEMP) to classify subjects with non-elevated/elevated skin temperature. We leverage the correlation between the skin temperature and the angular reflectance distribution of light, to empirically differentiate between skin at non-elevated temperature and skin at elevated temperature. We demonstrate the uniqueness of this correlation by 1) revealing the existence of a difference in the angular reflectance distribution of light from skin-like and non-skin like material and 2) exploring the consistency of the angular reflectance distribution of light in materials exhibiting optical properties similar to human skin. Finally, we demonstrate the robustness of V-TEMP by evaluating the efficacy of elevated skin temperature detection on subject videos recorded in 1) laboratory controlled environments and 2) outside-the-lab environments. V-TEMP is beneficial in two ways; 1) it is non-contact-based, reducing the possibility of infection due to contact and 2) it is scalable, given the ubiquity of video-recording devices.