We examined the genetic structure of populations of an ambrosia beetle, Xylosandrus brevis (Eichhoff) (Curculionidae: Scolytinae), to understand its colonization dynamics. We collected specimens from 20 sites in Japan and studied the genetic structure of these populations using portions of the mitochondrial cytochrome oxidase I gene (COI). A phylogenetic analysis showed three distinct lineages (clades A, B, and C) within X. brevis. Clade A had 65 haplotypes from all the populations, except for populations from Nagano (Chiisagata-gun) and the Ryukyu Islands (Ishigaki), whereas clade B had 14 haplotypes from nine populations (Yamagata, Saitama, Shimo-Minochi-gun, Chiisagata-gun, Toyota, Watarai-gun, Wakayama, Tottori, and Kochi), and clade C had 6 haplotypes from the Ryukyu Islands (Ishigaki) population (and nowhere else). Nested clade phylogeographic analysis showed 65 clade A haplotypes and 14 clade B haplotypes geographically structured in two clades, respectively. A contact zone was identified in the Chubu and Tokai areas. Our results suggest that gene flow and individual movement of X. brevis between the Ryukyu Islands and the other three main islands of Japan has been rare, and X. brevis may be divided into two populations, eastern-central and northern-western, with little gene flow between the four areas of western Japan (Kinki, Chugoku, Shikoku, and Kyushu).