D4-GDI is a key regulator of Rho GTPases that have been implicated in several aspects of breast tumorigenesis. We have previously found that D4-GDI was selectively expressed in breast cancer cell lines over normal mammary epithelial cells [45]. In this study, we investigated the expression level of D4-GDI in breast tumor specimens (n = 165) by immunohistochemistry using a validated antibody that specifically recognizes the full-length D4-GDI protein. D4-GDI was predominantly expressed in the luminal cells of the duct in contrast to the myoepithelial cells of the outer layer. The percentage of D4-GDI positive samples were found to be higher in the early stages of breast cancers followed by a significant decrease in malignant tumors and metastatic lesions when compared to normal breast tissues (p < 0.01). Analysis of matched samples confirmed the lower expression of D4-GDI in malignant tumors than normal adjacent tissues, while there was no further decrease in metastatic lesions. These results suggest that D4-GDI may function as a biphasic regulator of breast cancer progression and metastasis.