The development of robust materials for treating diseases through non-invasive photothermal therapy (PTT) has attracted increasing attention in recent years. Among various types of nanomaterials, inorganic nanomaterials with strong absorption in the near-infrared (NIR) window can be employed as high-efficiency photothermal agents to treat cancer and bacterial infections. In addition, inorganic nanomaterials can be easily combined with other drugs or chemical reagents to construct multifunctional nanomaterials to cascade stimulation responses, enhance therapeutic effects, and perform precise medical treatments. In this review, focusing on the latest developments of inorganic nanomaterials in photothermal therapy, we firstly introduced the light-to-heat conversion mechanism of inorganic nanomaterials. Secondly, we summarized the application of common inorganic nanomaterials, such as metallic nanoparticles, transition metal oxide nanoparticles and two dimensional (2D) nanosheets. In addition, the strategy of developing multifunctional nano-platforms with excellent biocompatibility as well as good targeted capability was also expounded. Finally, challenges and new perspectives for designing effective inorganic nanomaterial-based nanosystems for photothermal assisted therapy were also discussed.