The cardiovascular system can be analyzed using spectral, nonlinear, and complexity metrics. Nevertheless, dynamical noise may significantly impact these quantifiers. To our knowledge, there has been no attempt to quantify the intrinsic cardiovascular system noise driving heartbeat dynamics. To this end, this study presents a novel, model-free framework to define and quantify physiological noise using nonlinear Approximate Entropy profile. The framework was tested using analytical noisy series and then applied to real Heart Rate Variability (HRV) series gathered from a publicly-available dataset of recordings from 19 young and 19 elderly subjects watching the movie "Fantasia". Results suggest that physiological noise may account for over 15% of cardiovascular dynamics and is influenced by aging, with decreased cardiac noise in the elderly compared to the young subjects. Our findings indicate that physiological noise is a crucial factor in characterizing cardiovascular dynamics, and current spectral, nonlinear, and complexity assessments should take into account underlying dynamical noise estimates.