A variety of methods have been successfully used to produce crystals of colloidal particles made of polymeric (or silica) microspheres. Achieving highly accurate growth and control of the packing symmetry, packing efficiency and packing quality of these crystals is of paramount importance for many applications, for example in photonics. If colloidal crystals are formed in self-assembly processes, it is usually the most densely packed (111) set of planes that terminates the crystal-air interface. However, often exposure of (given) different packing facets is required at the crystal surface. In addition, there is ( in photonics) need for crystals exhibiting lower than the tightest packing, and possessing also lower degrees of packing symmetry. These requirements demand development of various engineering approaches for controlled particle assembly in regular structures. The synthesis of polymer colloidal particles with different sizes, shapes and surface charge density is first briefly outlined. The various interactions and forces that control growth for a broad range of colloidal crystals are subsequently discussed. In the main section of this review we give an account of various template-assisted, graphoepitaxial assembly approaches to produce colloidal crystals with tailored packing structures and controlled crystal orientation with respect to the topologically patterned substrates used to direct the assembly process. In the outlook we also describe various selected emerging approaches, which have the potential to produce crystals with low degree of packing symmetries, for example using direct one-to-one colloidal particle assembly.