Stoke is a leading cause of long-term disability, including upper-limb hemiparesis. Frequent, unobtrusive assessment of naturalistic motor performance could enable clinicians to better assess rehabilitation effectiveness and monitor patients' recovery trajectories. We therefore propose and validate a two-phase data analytic pipeline to estimate upper-limb impairment based on the naturalistic performance of activities of daily living (ADLs). Eighteen stroke survivors were equipped with an inertial sensor on the stroke-affected wrist and performed up to four ADLs in a naturalistic manner. Continuous inertial time series were segmented into sliding windows, and a machine-learned model identified windows containing instances of point-to-point (P2P) movements. Using kinematic features extracted from the detected windows, a subsequent model was used to estimate upper-limb motor impairment, as measured by the Fugl-Meyer Assessment (FMA). Both models were evaluated using leave-one-subject-out cross-validation. The P2P movement detection model had an area under the precision-recall curve of 0.72. FMA estimates had a normalized root mean square error of 18.8% with R2=0.72. These promising results support the potential to develop seamless, ecologically valid measures of real-world motor performance.