The development of sophisticated machine learning algorithms has made it possible to detect critical health conditions like cardiac arrhythmia, directly from electrocardiogram (ECG) recordings. Large-scale machine learning models, like deep neural networks, are well known to underperform when subjected to small perturbations which would not pose a challenge to physicians. This is a hurdle that needs to be removed to facilitate wide-scale adoption. We find this to be true even for models trained using data-augmentation schemes.In this paper, we show that using memory classifiers it is possible to attain a boost in robustness using expert-informed features. Memory classifiers combine standard deep neural network training with a domain knowledge-guided similarity metric to boost the robustness of classifiers. We evaluate the performance of the models against naturally occurring physiological perturbations, specifically electrode movement, muscle artifact, and baseline wander noise. Our approach demonstrates improved robustness across all evaluated noises for an average improvement in F1 score of 3.13% compared to models using data augmentation techniques.Clinical relevance- This approach improves the robustness of deep learning methods in safety-critical medical applications.