Indigenous Chinese goat mtDNA is highly diverse but lacks geographic specificity; however, whether gene flow or gene exchange contributed to this remains unknown. We reanalyzed a consensus fragment of 481 bp in the D-loop region from 339 individuals. The network and neighbor-joining tree revealed three divergent maternal haplogroups (A, B1, and B2) in 17 local breeds. Although high polymorphism resulting in 198 different haplotypes was observed (h = 0.984 ± 0.002; π = 0.0336 ± 0.0008), neither the distribution of haplotypes nor PCA analysis revealed any obvious geographic structure in the local breeds. Extensive gene flow was widely detected among breeds from southwest China. High levels of gene exchange were detected between Qianbei Brown goats and the other breeds, indicating either more contribution or introgression to their gene pools. This study will be helpful in understanding the phylogeography and gene flow among the goat breeds of southwest China.