Formation of partially reproductively isolated ecotypes in the rough periwinkle, Littorina saxatilis, may be a case of incipient nonallopatric ecological speciation. To better understand the dynamics of ecotype formation, its timescale, driving forces and evolutionary consequences, we developed a spatially explicit, individual-based model incorporating relevant ecological, spatial and mate selection data for Swedish L. saxatilis. We explore the impact of bounded hybrid superiority, ecological scenarios and mate selection systems on ecotype formation, gene flow and the evolution of prezygotic isolation. Our model shows that ecotypes are expected to form rapidly in parapatry under conditions applicable to Swedish L. saxatilis and may proceed to speciation. However, evolution of nonrandom mating had complex behaviour. Ecotype evolution was inhibited by pre-existing mating preferences, but facilitated by the evolution of novel preferences. While in many scenarios positive assortative mating reduced gene flow between ecotypes, in others negative assortative mating arose, preferences were lost after ecotype formation, preferences were confined to one ecotype or the ancestral ecotype became extinct through sexual selection. Bounded hybrid superiority (as observed in nature) enhanced ecotype formation but increased gene flow. Our results highlight that ecotype formation and speciation are distinct processes: factors that contribute to ecotype formation can be detrimental to speciation and vice versa. The complex interactions observed between local adaptation and nonrandom mating imply that generalization from data is unreliable without quantitative theory for speciation.