Little is known about disorder-specific biomarkers of bipolar disorder (BD) and major depressive disorder (MDD). Our aim was to determine a neural substrate that could be used to distinguish BD from MDD. Our study included a BD group (10 patients with BD, 10 first-degree relatives (FDRs) of individuals with BD), MDD group (17 patients with MDD, 17 FDRs of individuals with MDD), and 27 healthy individuals. Structural and functional brain abnormalities were evaluated by voxel-based morphometry and a trail making test (TMT), respectively. The BD group showed a significant main effect of diagnosis in the gray matter (GM) volume of the anterior cingulate cortex (ACC; p = 0.01) and left insula (p < 0.01). FDRs of individuals with BD showed significantly smaller left ACC GM volume than healthy subjects (p < 0.01), and patients with BD showed significantly smaller ACC (p < 0.01) and left insular GM volume (p < 0.01) than healthy subjects. The MDD group showed a tendency toward a main effect of diagnosis in the right and left insular GM volume. The BD group showed a significantly inverse correlation between the left insular GM volume and TMT-A scores (p < 0.05). Our results suggest that the ACC volume could be a distinct endophenotype of BD, while the insular volume could be a shared BD and MDD endophenotype. Moreover, the insula could be associated with cognitive decline and poor outcome in BD.