Chromosomal inversions allow genetic divergence of locally adapted populations by reducing recombination between chromosomes with different arrangements. Divergence between populations (or hybridization between species) is expected to leave signatures in the neutral genetic diversity of the inverted region. Quantitative expectations for these patterns, however, have not been obtained. Here, we develop coalescent models of neutral sites linked to an inversion polymorphism in two locally adapted populations. We consider two scenarios of local adaptation: selection on the inversion breakpoints and selection on alleles inside the inversion. We find that ancient inversion polymorphisms cause genetic diversity to depart dramatically from neutral expectations. Other situations, however, lead to patterns that may be difficult to detect; important determinants are the age of the inversion and the rate of gene flux between arrangements. We also study inversions under genetic drift, finding that they produce patterns similar to locally adapted inversions of intermediate age. Our results are consistent with empirical observations, and provide the foundation for quantitative analyses of the roles that inversions have played in speciation.