Introduction: Restoration of spontaneous smiling is a key goal in facial reanimation. A major obstacle to quantifying recovery of spontaneous smiling is the current lack of a uniform and objective means of smile quantification. Objective: To compare the facial movements during smiling in patients with facial paralysis as measured by an automated application and human observers. Methods: Video recordings of 25 patients with unilateral facial palsy (FP) watching humorous videos were utilized. Application-derived smile timestamping was compared with manual observer interpretation. Internal reliability of measurements was evaluated through a test-retest approach. Results: Application-derived smile identification demonstrated almost perfect agreement with manual interpretation (kappa 0.861, p < 0.001). There was no statistically significant difference in mean number of smiles between detection method (p = 0.354). Automated smile identification demonstrated a high degree of specificity (95.4%), accuracy (93.1%), positive-predictive value (94.7%), and negative-predictive value (91.8%). This method demonstrated a high degree of reliability (kappa 0.864, p < 0.01). Conclusion: The novel "SmileCheck" mobile phone application performed accurate and reliable smile quantification in FP patients in comparison with manual observation.