People Living with Dementia (PLwD) often exhibit behavioral and psychological symptoms of dementia; with agitation being one of the most prevalent symptoms. Agitated behaviour in PLwD indicates distress and confusion and increases the risk to injury to both the patients and the caregivers. In this paper, we present the use of wearable devices to detect agitation in PLwD. We hypothesize that combining multi-modal sensor data can help in building better classifiers to identify agitation in PLwD in comparison to a single sensor. We present a unique study to collect motion and physiological data from PLwD. This multi-modal sensor data is subsequently used to build predictive models to detect agitation in PLwD. The results on Random Forest for 28 days of data from PLwD show a strong evidence to support our hypothesis and highlight the importance of using multi-modal sensor data for detecting agitation events amongst them.