Newborns admitted to the neonatal intensive care unit (NICU) require a high level of care due to their precarious condition. Nurses typically monitor their vital signs continuously using wearable sensors such as electrocardiogram (ECG) electrodes placed on their chest and a pulse oximeter on a limb. When the patient moves, this can cause motion artifacts on one or more physiologic signals, potentially resulting in a false alarm on the patient monitor. We therefore propose a motion detection algorithm to mitigate these alarms by providing context. Using a camera positioned above the crib or overhead warming bed, we recorded videos from six patients and annotated all patient movements. These data were used to train and evaluate two different approaches for non-contact motion detection. Results were stronger for the optical flow technique than for the long short-term memory network approach. This represents a challenging problem due to variable lighting, patient clothing and bed coverings, and the complex clinical environment in the NICU.