Global single-valued biomarkers, such as ejection fraction, are widely used in clinical practice to assess cardiac function. However, they only approximate the heart's true 3D deformation process, thus limiting diagnostic accuracy and the understanding of cardiac mechanics. Metrics based on 3D shape have been proposed to alleviate these shortcomings. In this work, we present the Point Cloud Deformation Network (PCD-Net) as a novel geometric deep learning approach for direct modeling of 3D cardiac mechanics of the biventricular anatomy between the extreme ends of the cardiac cycle. Its encoder-decoder architecture combines a low-dimensional latent space with recent advances in point cloud deep learning for effective multi-scale feature learning directly on flexible and memory-efficient point cloud representations of the cardiac anatomy. We first evaluate the PCD-Net's predictive ability for both cardiac contraction and relaxation on a large U.K. Biobank dataset of over 10,000 subjects and find average Chamfer distances between the predicted and ground truth anatomies below the pixel resolution of the underlying image acquisition. We then show the PCD-Net's suitability to act as a normality model of 3D cardiac mechanics and capture subpopulation-specific differences between normal subjects and myocardial infarction (MI) patients. Next, we highlight the PCD-Net's interpretability by visualizing abnormal phenotypes between predicted normal 3D shapes and corresponding observed ones. Finally, we demonstrate that the PCD-Net's learned 3D deformation encodings outperform multiple clinical and machine learning benchmarks by 11% in terms of area under the receiver operating characteristic curve for the tasks of prevalent MI detection and incident MI prediction and by 7% in terms of Harrell's concordance index for MI survival analysis.