Patients with metastasized malignant melanomas (MM) are regularly treated with immune checkpoint inhibitors (CPI). Within our study, we evaluated the predictive value of tumor infiltrating lymphocyte (TIL) clusters in primary MM and its association to molecular subtypes to predict response to CPI treatment. A cohort of 90 MM patients who received CPI treatment were collected from a single center, as well as a validation cohort of 351 patients from the TCGA database (SKCM) who received standard of care. A deep-convolutional-neural network (U-Net) was trained to detect viable tumor areas on H&E whole-slide-images, following a quantitative detection of TILs with help of a separate additional neural network. The number of TIL clusters was associated with response to CPI in 90 MM patients (AUC = 0.6), even more pronounced within the sub-cohort of BRAF V600E/K-mutated MM patients (AUC = 0.7, n = 32). Interestingly, the TIL clusters in NRAS-mutated as well as wildtype MM (BRAF-wt, NRAS-wt) tumors, did not demonstrate a predictive value of CPI response (AUC = 0.5, n = 25). Moreover, PD-L1 expression had a limited predictive value within our cohort. In parallel, within an independent cohort of MM patients (TCGA, n = 351), the number of TIL clusters was associated with improved survival in BRAF V600E/K mutated MM (p < 0.0001, n = 164) but neither in NRAS-mutated (55.7 months vs. 63.0 months, respectively, p = 0.590, n = 85) nor BRAF/NRAS-wildtype MM patients (52.4 months vs. 47.4 months, respectively, p = 0.581, n = 104). While TILs in MM have been associated with improved survival, we show-for the first time-that TIL clusters are associated with response to immunotherapy in BRAF V600E/K mutated MM.