In this paper, we propose an end-to-end deep learning architecture, referred as MCG-Net, integrating convolutional neural network (CNN) with transformer-based global context block for fine-grained delineation and diagnostic classification of four cardiac events from magnetocardiogram (MCG) data, namely Q-, R-, S- and T-waves. MCG-Net takes advantage of a multi-resolution CNN backbone as well as the state-of-the-art (SOTA) transformer encoders that facilitate global temporal feature aggregation. Besides the novel network architecture, we introduce a multi-task learning scheme to achieve simultaneous delineation and classification. Specifically, the problem of MCG delineation is formulated as multi-class heatmap regression. Meanwhile, a binary diagnostic classification label as well as a duration are jointly estimated for each cardiac event using features that are temporally aligned by event heatmaps. The framework is evaluated on a clinical MCG dataset, containing data collected from 270 subjects with cardiac anomalies and 108 control subjects. We designed and conducted a two-fold cross-validation study to validate the proposed method and to compare its performance with the SOTA methods. Experimental results demonstrated that our method outperformed counterparts on both event delineation and diagnostic classification tasks, achieving respectively an average ECG-F1 of 0.987 and an average Event-F1 of 0.975 for MCG delineation, and an average accuracy of 0.870, an average sensitivity of 0.732, an average specificity of 0.914 and an average AUC of 0.903 for diagnostic classification. Comprehensive ablation experiments are additionally performed to investigate effectiveness of different network components.