The development of label-free non-destructive techniques to be used as diagnostic tools in cancer research is of great importance for improving the quality of life for millions of patients. Previous studies have demonstrated that Third Harmonic Generation (THG) imaging could differentiate malignant from benign unlabeled human breast biopsies and distinguish the different grades of cancer. Towards the application of such technologies to clinic, in the present report, a deep learning technique was applied to THG images recorded from breast cancer tissues of grades 0, I, II, and III. By the implementation of a convolutional neural network (CNN) model, the differentiation of malignant from benign breast tissue samples and the discrimination of the different grades of cancer in a fast and accurate way were achieved. The obtained results provide a step ahead towards the use of optical diagnostic tools in conjunction with the CNN image classifier for the reliable and rapid malignancy diagnosis in clinic.