Agitation in persons with dementia (PWD) poses major health risks both for themselves and for their caregivers. Passive sensing based continuous behavior tracking can prevent the escalation of such episodes. But, predicting such behavior from sensor streams, especially in real-world residential settings, is still an active area of research. Major challenges include the sparsity, unpredictability, and variations in such behavior, as well as the "weak" annotations from real-world participants. This paper proposes a novel approach to overcome these issues in predicting agitation episodes from the PWD's wrist motion data. In a transdisciplinary study on dementia dyads residing in their homes, the PWD motion is continuously sensed from their smart watch inertial sensors, while agitation episodes are actively marked by the caregivers. The data from 10 residential deployments, each with 30 days duration, are analyzed in this paper, and multiple-instance learning (MIL) based models are implemented to learn from such sparse and weakly annotated data. These models are compared with single-instance models in predicting the agitated behavior. The results show the potential of MIL models in sparsely labeled behavior inference from wearables in-the-wild.