Previous research showed that frailty can influence autonomic nervous system and consequently heart rate response to physical activities, which can ultimately influence the homeostatic state among older adults. While most studies have focused on resting state heart rate characteristics or heart rate monitoring without controlling for physical activities, the objective of the current study was to classify pre-frail/frail vs non-frail older adults using heart rate response to physical activity (heart rate dynamics). Eighty-eight older adults (≥65 years) were recruited and stratified into frailty groups based on the five-component Fried frailty phenotype. Groups consisted of 27 non-frail (age = 78.80±7.23) and 61 pre-frail/frail (age = 80.63±8.07) individuals. Participants performed a normal speed walking as the physical task, while heart rate was measured using a wearable electrocardiogram recorder. After creating heart rate time series, a long short-term memory model was used to classify participants into frailty groups. In 5-fold cross validation evaluation, the long short-term memory model could classify the two above-mentioned frailty classes with a sensitivity, specificity, F1-score, and accuracy of 83.0%, 80.0%, 87.0%, and 82.0%, respectively. These findings showed that heart rate dynamics classification using long short-term memory without any feature engineering may provide an accurate and objective marker for frailty screening.