Crystallization represents the prime example of a disorder-order transition. In realistic situations, however, container walls and impurities are frequently present and hence crystallization is heterogeneously seeded. Rarely the seeds are perfectly compatible with the thermodynamically favoured crystal structure and thus induce elastic distortions, which impede further crystal growth. Here we use a colloidal model system, which not only allows us to quantitatively control the induced distortions but also to visualize and follow heterogeneous crystallization with single-particle resolution. We determine the sequence of intermediate structures by confocal microscopy and computer simulations, and develop a theoretical model that describes our findings. The crystallite first grows on the seed but then, on reaching a critical size, detaches from the seed. The detached and relaxed crystallite continues to grow, except close to the seed, which now prevents crystallization. Hence, crystallization seeds facilitate crystallization only during initial growth and then act as impurities.