Major depressive disorder (MDD) and bipolar disorder (BD) are two major mood disorders with partly overlapped symptoms but different treatments. However, their misdiagnosis and mistreatment are common based on the DSM-V criteria, lacking objective and quantitative indicators. This study aimed to develop a novel approach that accurately classifies MDD and BD based on their resting-state magnetoencephalography (MEG) signals during euthymic phases. A revisited 3D CNN model, Semi-CNN, that could automatically detect brainwave patterns in spatial, temporal, and frequency domains was implemented to classify wavelet-transformed MEG signals of normal controls and MDD and BD patients. The model achieved a test accuracy of 96.05% and an average of 95.71% accuracy for 5-fold cross-validation. Furthermore, saliency maps of the model were estimated using Grad-CAM++ to visualize the proposed classification model and highlight disease-specific brain regions and frequencies. Clinical Relevance - Our model provides a stable pipeline that accurately classifies MDD, BD, and healthy individuals based on resting-state MEG signals during the euthymic phases, opening the potential for quantitative and accurate brain-based diagnosis for the highly misdiagnosed MDD/BD patients.