The grade groups (GGs) of Gleason scores (Gs) is the most critical indicator in the clinical diagnosis and treatment system of prostate cancer. End-to-end method for stratifying the patient-level pathological appearance of prostate cancer (PCa) in magnetic resonance (MRI) are of high demand for clinical decision. Existing methods typically employ a statistical method for integrating slice-level results to a patient-level result, which ignores the asymmetric use of ground truth (GT) and overall optimization. Therefore, more domain knowledge (e.g., diagnostic logic of radiologists) needs to be incorporated into the design of the framework. The patient-level GT is necessary to be logically assigned to each slice of a MRI to achieve joint optimization between slice-level analysis and patient-level decision-making. In this paper, we propose a framework (PCa-GGNet-v2) that learns from radiologists to capture signs in a separate two-dimensional (2-D) space of MRI and further associate them for the overall decision, where all steps are optimized jointly in an end-to-end trainable way. In the training phase, patient-level prediction is transferred from weak supervision to supervision with GT. An association route records the attentional slice for reweighting loss of MRI slices and interpretability. We evaluate our method in an in-house multi-center dataset (N = 570) and PROSTATEx (N = 204), which yields five-classification accuracy over 80% and AUC of 0.804 at patient-level respectively. Our method reveals the state-of-the-art performance for patient-level multi-classification task to personalized medicine.