The evolution of the flower is commonly thought to have spurred angiosperm diversification. Similarly, particular floral traits might have promoted diversification within specific angiosperm clades. We hypothesize that traits promoting the precise positional transfer of pollen between flowers might promote diversification. In particular, precise pollen transfer might produce partial reproductive isolation that facilitates adaptive divergence between parapatric populations differing in their reproductive-organ positions. We investigate this hypothesis with an individual-based model of pollen transfer dynamics associated with heterostyly, a floral syndrome that depends on precise pollen transfer. Our model shows that precise pollen transfer can cause sexual selection leading to divergence in reproductive-organ positions between populations served by different pollinators, pleiotropically causing an increase in reproductive isolation through a "magic trait" mechanism. Furthermore, this increased reproductive isolation facilitates adaptive divergence between the populations in an unlinked, ecologically selected trait. In a different pollination scenario, however, precise pollen transfer causes a decrease in adaptive divergence by promoting asymmetric gene flow. Our results highlight the idea that magic traits are not "magic" in isolation; in particular, the effect size of magic traits in speciation depends on the external environment, and also on other traits that modify the strength of the magic trait's influence on non-random mating. Overall, we show that the evolutionary consequences of pollen transfer dynamics can depend strongly on the available pollinator fauna and on the morphological fit between flowers and pollinators. Furthermore, our results illustrate the potential importance of even weak reproductive isolating barriers in facilitating adaptive divergence.