The successful exploitation of germplasm banks, harbouring plant genetic resources indispensable for plant breeding, will depend on our ability to characterize their genetic diversity. The Vegetable Germplasm Bank of Zaragoza (BGHZ) (Spain) holds an important Capsicum annuum collection, where most of the Spanish pepper variability is represented, as well as several accessions of other domesticated and non-domesticated Capsicum spp from all over the five continents. In the present work, a total of 51 C. annuum landraces (mainly from Spain) and 51 accessions from nine Capsicum species maintained at the BGHZ were evaluated using 39 microsatellite (SSR) markers spanning the whole genome. The 39 polymorphic markers allowed the detection of 381 alleles, with an average of 9.8 alleles per locus. A sizeable proportion of alleles (41.2%) were recorded as specific alleles and the majority of these were present at very low frequencies (rare alleles). Multivariate and model-based analyses partitioned the collection in seven clusters comprising the ten different Capsicum spp analysed: C. annuum, C. chinense, C. frutescens, C. pubescens, C. bacatum, C. chacoense and C. eximium. The data clearly showed the close relationships between C. chinense and C. frutescens. C. cardenasii and C. eximium were indistinguishable as a single, morphologically variable species. Moreover, C. chacoense was placed between C. baccatum and C. pubescens complexes. The C. annuum group was structured into three main clusters, mostly according to the pepper fruit shape, size and potential pungency. Results suggest that the diversification of C. annuum in Spain may occur from a rather limited gene pool, still represented by few landraces with ancestral traits. This ancient population would suffer from local selection at the distinct geographical regions of Spain, giving way to pungent and elongated fruited peppers in the South and Center, while sweet blocky and triangular types in Northern Spain.