Human leukocyte antigen (HLA) complex molecules play an essential role in immune interactions by presenting peptides on the cell surface to T cells. With significant deep learning progress, a series of neural network-based models have been proposed and demonstrated with their excellent performances for peptide-HLA class I binding prediction. However, there is still a lack of effective binding prediction models for HLA class II protein binding with peptides due to its inherent challenges. We present a novel sequence-based pan-specific neural network structure, DeepSeaPanII, for peptide-HLA class II binding prediction in this work. Our model is an end-to-end neural network model without the need for pre-or post-processing on input samples compared with existing pan-specific models. Besides state-of-the-art performance in binding affinity prediction, DeepSeqPanII can also extract biological insight on the binding mechanism over the peptide by its attention mechanism-based binding core prediction capability. The leave-one-allele-out cross-validation and benchmark evaluation results show that our proposed network model achieved state-of-the-art performance in HLA-II peptide binding. The source code and trained models are freely available at https://github.com/pcpLiu/DeepSeqPanII.