Attention-based generative models for de novo molecular design

Orion Dollar; Nisarg Joshi; David A C Beck; Jim Pfaendtner

doi:10.1039/d1sc01050f

Attention-based generative models for de novo molecular design

Chem Sci. 2021 May 14;12(24):8362-8372. doi: 10.1039/d1sc01050f.

Authors

Orion Dollar¹, Nisarg Joshi¹, David A C Beck^{1

2}, Jim Pfaendtner¹

Affiliations

¹ Department of Chemical Engineering, University of Washington Seattle 98185 WA USA jpfaendt@uw.edu.
² eScience Institute, University of Washington Seattle 98185 WA USA dacb@uw.edu.

Abstract

Attention mechanisms have led to many breakthroughs in sequential data modeling but have yet to be incorporated into any generative algorithms for molecular design. Here we explore the impact of adding self-attention layers to generative β-VAE models and show that those with attention are able to learn a complex "molecular grammar" while improving performance on downstream tasks such as accurately sampling from the latent space ("model memory") or exploring novel chemistries not present in the training data. There is a notable relationship between a model's architecture, the structure of its latent memory and its performance during inference. We demonstrate that there is an unavoidable tradeoff between model exploration and validity that is a function of the complexity of the latent memory. However, novel sampling schemes may be used that optimize this tradeoff. We anticipate that attention will play an important role in future molecular design algorithms that can make efficient use of the detailed molecular substructures learned by the transformer.