Colloidal self-assembly refers to a solution-processed assembly of nanometer-/micrometer-sized, well-dispersed particles into secondary structures, whose collective properties are controlled by not only nanoparticle property but also the superstructure symmetry, orientation, phase, and dimension. This combination of characteristics makes colloidal superstructures highly susceptible to remote stimuli or local environmental changes, representing a prominent platform for developing stimuli-responsive materials and smart devices. Chemists are achieving even more delicate control over their active responses to various practical stimuli, setting the stage ready for fully exploiting the potential of this unique set of materials. This review addresses the assembly of colloids into stimuli-responsive or smart nanostructured materials. We first delineate the colloidal self-assembly driven by forces of different length scales. A set of concepts and equations are outlined for controlling the colloidal crystal growth, appreciating the importance of particle connectivity in creating responsive superstructures. We then present working mechanisms and practical strategies for engineering smart colloidal assemblies. The concepts underpinning separation and connectivity control are systematically introduced, allowing active tuning and precise prediction of the colloidal crystal properties in response to external stimuli. Various exciting applications of these unique materials are summarized with a specific focus on the structure-property correlation in smart materials and functional devices. We conclude this review with a summary of existing challenges in colloidal self-assembly of smart materials and provide a perspective on their further advances to the next generation.