Finetuning of GLIDE stable diffusion model for AI-based text-conditional image synthesis of dermoscopic images

Veronika Shavlokhova; Andreas Vollmer; Christos C Zouboulis; Michael Vollmer; Jakob Wollborn; Gernot Lang; Alexander Kübler; Stefan Hartmann; Christian Stoll; Elisabeth Roider; Babak Saravi

doi:10.3389/fmed.2023.1231436

Finetuning of GLIDE stable diffusion model for AI-based text-conditional image synthesis of dermoscopic images

Front Med (Lausanne). 2023 Oct 20:10:1231436. doi: 10.3389/fmed.2023.1231436. eCollection 2023.

Authors

Affiliations

¹ Maxillofacial Surgery University Hospital Ruppin-Fehrbelliner Straße Neuruppin, Neuruppin, Germany.
² Department of Oral and Maxillofacial Plastic Surgery, University Hospital of Würzburg, Würzburg, Germany.
³ Departments of Dermatology, Venereology, Allergology and Immunology, Staedtisches Klinikum Dessau, Medical School Theodor Fontane and Faculty of Health Sciences Brandenburg, Dessau, Germany.
⁴ Department of Oral and Maxillofacial Surgery, Tuebingen University Hospital, Tuebingen, Germany.
⁵ Department of Anesthesiology, Perioperative and Pain Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, United States.
⁶ Department of Orthopedics and Trauma Surgery, Medical Centre-Albert-Ludwigs-University of Freiburg, Faculty of Medicine, Albert-Ludwigs-University of Freiburg, Freiburg, Germany.
⁷ Department of Dermatology, University Hospital of Basel, Basel, Switzerland.

Abstract

Background: The development of artificial intelligence (AI)-based algorithms and advances in medical domains rely on large datasets. A recent advancement in text-to-image generative AI is GLIDE (Guided Language to Image Diffusion for Generation and Editing). There are a number of representations available in the GLIDE model, but it has not been refined for medical applications.

Methods: For text-conditional image synthesis with classifier-free guidance, we have fine-tuned GLIDE using 10,015 dermoscopic images of seven diagnostic entities, including melanoma and melanocytic nevi. Photorealistic synthetic samples of each diagnostic entity were created by the algorithm. Following this, an experienced dermatologist reviewed 140 images (20 of each entity), with 10 samples originating from artificial intelligence and 10 from original images from the dataset. The dermatologist classified the provided images according to the seven diagnostic entities. Additionally, the dermatologist was asked to indicate whether or not a particular image was created by AI. Further, we trained a deep learning model to compare the diagnostic results of dermatologist versus machine for entity classification.

Results: The results indicate that the generated images possess varying degrees of quality and realism, with melanocytic nevi and melanoma having higher similarity to real images than other classes. The integration of synthetic images improved the classification performance of the model, resulting in higher accuracy and precision. The AI assessment showed superior classification performance compared to dermatologist.

Conclusion: Overall, the results highlight the potential of synthetic images for training and improving AI models in dermatology to overcome data scarcity.

Keywords: GLIDE; cancer; dermatology; dermoscopy; stable diffusion; text-to-image.

Grants and funding

The article processing charge was funded by the Baden-Wuerttemberg Ministry of Science, Research and Art, and the University of Freiburg in the funding program Open Access Publishing.