Matters when converted into nanosize provide some unique surface properties, which are different from those of the bulk materials. Nanomaterials show some extraordinary behavioral patterns because of those properties, such as supermagnetism, quantum confinement, etc. A great deal of implication of nanomaterials in nanomedicine has already been realized. Utility of nanomaterials as drug nanocarrier projects many potential advantages of them in drug delivery. Despite many such advantages, the potential risk of health and environmental hazards related to them cannot be ignored. Here various physicochemical factors, such as chemical nature, degradability, surface properties, surface charge, particle size, and shape, have been shown to play a crucial role in toxicity related to drug nanocarriers. Evidence-based findings of some drug nanocarriers have been incorporated to provide distinct knowledge to the readers in the field. A glimpse of current regulatory controls and measures required to combat the challenges of toxicological aspects of drug nanocarriers have been described.