Computer-aided diagnosis (CAD) with cine MRI is a foremost research topic to enable improved, faster, and more accurate diagnosis of cardiovascular diseases (CVD). However, current approaches that use manual visualization or conventional clinical indices can lack accuracy for borderline cases. Also, manual visualization of 3D/4D MR data is time-consuming and expert-dependent. We try to simplify this process by creating an end-to-end automated CAD system that segments the critical substructures of the heart. The new domain-related physiological features are then calculated from the segmented regions. These features are fed to a random forest classifier that identifies the anomaly. We have obtained a very high accuracy when testing this end-to-end approach on the Automated Cardiac Diagnosis challenge (ACDC) dataset (4 pathologies, 1 normal). To prove the generalizability of the method we have blind-tested this approach on M&Ms-2 dataset which is a multi-center, multi-vendor, and multi-disease dataset with better than 90% accuracy.