Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_5338_Библиотеки_им_академика_М_И_Перельмана
.pdf
7 Language Models in Molecular Discovery 123
https://t.me/med1917
comprising lab scientists. Additionally, it can be anticipated that this integration will
result in a significant increase in the utilization of the developed AI models and
contribute to scientific discovery. Here, we present an overview of the role of LMs
in accelerated molecular discovery. We commence with the conventional scientific
discovery method and then discuss how molecular generative models can be coupled
with molecular property prediction models. Next, we provide readers looking for
practical usability with a curated list of software tools and libraries for scientific
language modeling. We conclude by envisioning the future of molecule design,
where natural language models, custom-built AI models, and cheminformatics tools
are integrated into the discovery process via chatbot user interfaces.
7.2 Accelerated Molecular Discovery
Molecule discovery, intricately linked to optimizing diverse properties in a vast
space, challenges conventional scientific methods. In chemistry’s Design-MakeTest-Analyze (DMTA) cycle, synthesis costs and time constraints create a bottleneck that hampers hypothesis refinement (cf. Fig.
are largely driven by medicinal chemists who design “molecule hypotheses” which
are biased, ad-hoc, and non-exhaustive. This hinders progress in addressing global
issues, creating a necessity for an accelerated process of molecule discovery. Thus, a
key challenge lies in improving the speed and quality of evaluating such “molecule
hypotheses”, which are grounded on laboratory work.
Deep generative models have recently emerged as a promising tool to expedite the hypothesis/design phase in molecular discovery. However, even the most
advanced molecular generative models require an efficient method for large-scale
virtual screening to test their hypotheses. The accelerated molecular discovery cycle
adds a validation loop to DMTA, rapidly evaluating numerous hypotheses inexpensively (cf. Fig.
only promising hypotheses advance to the synthesis and physical experimentation
stages.
7.1b). This loop enhances the design-phase generative model, ensuring
7.1a). Traditional approaches
7.2.1 Molecule Representation
Data representation plays a crucial role in molecular discovery. It determines the
type of information that is available to the model and consequently the properties
that can be predicted. An overview of commonly used molecular representations
for property prediction is illustrated in Fig.
language models (CLMs), this section focuses on text-representations of molecules.
A more focused discussion on CLMs is covered by [
7.2. Due to the popularity of chemical
14].
124 N. Janakarajan et al.
https://t.me/med1917
Fig. 7.2 An illustration of popular ways of representing a chemical molecule as input to an ML
model. The representations may be a) String-based, such as SMILES, SELFIES, or InChI which
use characters to represent different aspects of a molecule, b) Structure-based, such as Graphs
or MolFiles that encode connectivity and atomic position, and c) Feature-based, such as Morgan
Fingerprints, which encode local substructures as bits
7.2.1.1 Simplified Molecular Input Line-Entry System (SMILES)
SMILES [
12] is a string representation that maps specific characters to atoms,
bonds, branches, aromaticity, rings, and stereochemistry in molecular structures. The
character-level representation enables easy tokenization, making SMILES an ideal
input for LMs. SMILES are non-unique, so each molecule can be represented by
multiple SMILES strings. Hence, SMILES are either canonicalized or, alternatively,
their multiplicity is used as a data augmentation strategy [
performance improvement in molecular property prediction [
generation [
18, 19]. In generative modeling, a common issue is the invalidity of
15], which has shown
15–17] and molecular
SMILES strings due to an uneven number of ring opening/closure symbols or bond
valence violations. SMILES strings can undergo further processing, such as kekulization or stereoinformation removal but employing canonicalized SMILES remains the
most prevalent approach.
Tokenization is the process of splitting a string into vectorizable units. These units
are typically a single character, n-gram characters, or words. Instead of splitting at
the character level, SMILES are typically tokenized at the atom level with regular
expressions [
20] or by additionally including positional and connectivity information,
thereby acknowledging that the same atom can have different encodings based on
21
its location in the molecular structure [
substructure level, as demonstrated by SMILES Pair Encoding (SMILES-PE) [
]. SMILES may also be tokenized at the
22].
This method, inspired by byte-pair encoding, iteratively counts and merges frequently
occurring SMILES token pairs until a given condition is met. Tokenization enables
the creation of a vocabulary for SMILES representations.
Vocabularies are dictionaries mapping tokens to vectors thus serving as a gateway
to LMs. For LMs to learn from SMILES, tokens are typically vectorized via one-hot
7 Language Models in Molecular Discovery 125
https://t.me/med1917
encodings (where each row in the binary matrix corresponds to a SMILES position
and each column signifies a token). However, this discrete method results in sparse,
large matrices and thus, an alluring alternative is to learn a continuous embedding
for each token during training. This facilitates the learning of semantic relationships
between tokens and enhances performance. Since learning good embeddings requires
a lot of data, models pretrained on natural language corpora are a strong option to
learn scientific language embeddings through fine-tuning [
7.2.1.2 Self Referencing Embedded Strings (SELFIES)
SELFIES [24] were introduced as an alternative to SMILES to counter the problem
of generating invalid molecules. Unlike SMILES, SELFIES are generated using
derivation rules to enforce valence-bond validity. It also stores branch length and
ring size to avoid open branches and rings. These supplementary attributes ensure
a valid representation during molecule generation. While this strategy guarantees
100% validity, it could produce strings that are too short to be a useful molecule.
7.2.1.3 International Chemical Identifier (InChI)
Introduced by the IUPAC, InChI [25] are strings encoding structural information
23].
and complex for larger molecules. To counter this, a hash called ‘InChiKey’ was
developed to help with search and retrieval. InChIs are less commonly used in
26].
LMs [
7.2.2 Generative Modeling
Generative modeling involves learning the data’s underlying distribution with the
intent of generating new samples, a technique pivotal in accelerating de novo drug
discovery. A generative model may be conditional or unconditional. A conditional
generative model utilizes provided data attributes or labels to generate new samples
with desired properties, whereas an unconditional model solely provides a way
13
to sample molecules similar to the training data [
larly benefits from the conditional generation approach as it facilitates goal-oriented
27
hypothesis design [
models that act on chemical language to generate molecules satisfying user-defined
conditions.
]. This section describes a few influential conditional generation
]. The DMTA cycle particu-
126 N. Janakarajan et al.
https://t.me/med1917
Fig. 7.3 An illustration of conditional molecule generation using LMs. The process begins with the
collection and processing of multi-modal data, followed by its compression into a fixed-size latent
representation. These representations subsequently pass into a molecular generative model. The
generated molecules then undergo in-silico property prediction, which links back to the generative
model through a feedback loop during training. The in-silico models direct the generative model to
produce property- or task-driven molecules using a reward function. In the inference stage, candidate
molecules generated by the optimized model undergo lab synthesis and subsequent experimental
validation to determine their efficacy for the desired task
7.2.2.1 Recurrent Neural Network (RNN)
The sequential nature of RNNs makes them suitable models for processing chemical languages. Proposed in the 90s, RNNs were the first type of chemical language
models (CLMs) to enter the domain [
15, 20, 28]. RNNs continuously update their
hidden states as new tokens are passed to the network, thus enabling it to encode
contextual information. During the generation process, tokens are produced autoregressively. RNNs find use in generating molecule libraries [
28] which are exten-
sively used in drug development processes like screening. External scoring functions
drive the generation of molecules with desired properties. RNNs are also adept at
29
learning complex distributions [
], even though their inability to count occurrences of ring opening/
valid SMILES [
30
closing symbols poses a challenge [
] and generating a higher proportion of unique and
31, 32].
7.2.2.2 Variational Autoencoder (VAE)
VAEs learn latent distribution parameters of molecules, thus enabling the generation
of new molecules by sampling from this distribution. Their unique ability lies in
learning a smooth, latent space that facilitates interpolation of samples, even for
notoriously discrete entities like molecules [
any network compatible with string inputs can function as a VAE’s encoder and
decoder. Initial works primarily focused on single-modality applications, assessing
latent space quality via downstream tasks [
]. To make it suitable for CLMs,
13
13]. This approach remains prevalent
7 Language Models in Molecular Discovery 127
https://t.me/med1917
and can be used to generate, e.g., catalysts with an RNN-based VAE [33]. Here, a
latent space is learned and assessed by predicting the catalyst binding energy. [
takes it a step further by concatenating a condition vector to the input and the latent
embedding generated by the recurrent network-based VAE’s encoder. This approach
enables the generation of molecules specifically tailored to the given conditions. The
scope of VAEs expanded progressively into multi-modal settings for conditional
molecule generation, as visualized in Fig.
works on task-driven molecule generation incorporate contextual information like
gene expression [
learn embeddings of context information and primer drugs, which are merged before
decoding to produce molecules. A reinforcement-learning-based approach directs
the model to produce molecules with desired properties using rewards.
7.2.2.3 Transformer
The self-attention attribute of Transformers [11] have propelled these models to the
forefront of NLP. Transformers have an encoder module that relies on this selfattention to learn embeddings of the input and the context associated with it. The
decoder module predicts tokens using the context learnt by the encoder and previously generated tokens through attention. For generative modeling, decoder-only
transformers like the Generative Pre-Training Transformer (GPT) [
35] or protein targets [36, 37] or even both [38], where the VAEs
7.3 and exemplified by [35–37]. These
39
] have become
34]
One of the first models to use the GPT architecture for conditional molecule generation is MolGPT [
summarizes the desired properties and scaffolds are passed as input to this model,
which is then trained on the next token prediction task to generate molecules. GPTlike models coupled with RL can also be used to optimize molecular properties like
pIC50 [
strings, and the embedding space is then optimized such that the model samples
molecules with the desired properties. Going beyond just using GPT-like architectures for molecule generation, Regression Transformer [
formulates conditional sequence modeling as a regression problem. This gives rise to
a natural multitask model that concurrently performs property prediction and conditional molecular generation. This is achieved by concatenating conventional molecular tokens with property tokens and employing a training scheme that alternates
which parts of the sequence are masked.
models. The superior quality of learned embeddings coupled with its ability to handle
parallel processing and scalability makes it a top choice for the task of conditional
molecule generation, with promising applications in drug discovery and other areas
of molecular design [
41]. In this two-stage approach, embeddings are first learnt from SMILES
All these works are testament to the generative capabilities of Transformer-based
40]. SMILES tokens concatenated with a condition vector that
] is a seminal work that
42
9].
128 N. Janakarajan et al.
https://t.me/med1917
7.2.3 Property Prediction
Whether a discovery is novel or not, property prediction is a key step in validating the molecules for a given use case. The success of a molecule depends on a
myriad of factors, including how it interacts with its environment. The MoleculeNet
datasets [
from public datasets and comprises over 700,000 compounds tested on various properties. The ToxSmi model [
predict toxicity from SMILES. The model has three kernel sizes for the convolutional network and uses a Bahdanau attention mechanism [
superior performance overall on various MoleculeNet tasks compared to all other
SMILES-based models. A recent trend is to use transformer-encoders to learn embeddings for molecules and then apply a multilayer perceptron (MLP) on the embeddings for property prediction. MolBERT [
examples. These transformer-based models use a BERT backbone to learn molecular embeddings from SMILES and predict properties. Similarly, Molformer [
uses a transformer-encoder with linear attention and relative positional encoding
to learn compressed molecular representations which are then fine-tuned on chemical property prediction benchmarks. To equip transformers with better inductive
biases to handle molecules, adaptations of the attention mechanism were proposed.
The molecule attention transformer (MAT) incorporates inter-atomic distances and
graph structure into the attention mechanism [
the relative-MAT which fuses the distance embedding, bond embedding and neighborhood embedding and achieves competitive performances on a range of property prediction tasks [
to address tasks such as drug sensitivity prediction [
affinity prediction [
43] are a commonly used benchmark for property prediction. It is curated
44] uses a multiscale convolutional attention model to
45]. The model shows a
46] and ChemBERTA [47]) are two such
48]
49]. An improvement over this model is
50]. Chemical LMs are often extended to multimodal inputs
51] or protein-ligand binding
52].
7.3 Software Tools for Scientific Language Modeling
The paradigm shift toward open-sourcing software has profoundly influenced chemistry. Commonly listed implications of open-sourcing in the context of drug discovery
include catalyzation of methodological development, fostering collaboration, and
ease of scientific reproducibility [
assets (e.g., Python packages or cloud-based web apps) that are key to enable
molecular discovery.
53]. In this section we present several software
7 Language Models in Molecular Discovery 129
https://t.me/med1917
7.3.1 Natural Language Models
The success story of the Transformer [11
architecture goes hand in hand with the rise of the transformers library [
created in 2019 by
formers have been adopted across disciplines, e.g., in computer vision [
ment learning [
provides the largest public hub of language models and it offers implementations of
all recent models as well as a diverse collection of pretrained models available for
fine-tuning or inference. While most of their models focus on NLP, select models are
designed for life science applications, in particular, molecular property prediction
(e.g., ChemBerta [
ular generation (e.g., MolT5 [
(e.g., ProtBert, ProtAlbert, ProtXLNet and ProtT5 [
models like Multimodal Text and Chemistry T5 [
that extend beyond the above mentioned tasks to include additional functions like
predicting forward/backward reactions.
HuggingFace. Initially intended for NLP applications, Trans-
56], protein folding [57], and, naturally, chemistry [58]. HuggingFace
47]), molecular captioning (e.g., MolT5 [59]), text-based molec-
59]), and also unsupervised protein language modeling.
] as the most widely adopted neural network
54],
55], reinforce-
60]). Furthermore, some available
23] are prompt-based multitaskers
7.3.2 GT4SD—Generative Modeling Toolkits
Python libraries like gt4sd (the
Therapeutics Data Commons [62]) or deepchem [63] were developed
TdC (
primarily for molecular discovery applications. Particularly, gt4sd provides exten-
sive support for LMs. gt4sd is designed to enable researchers and developers to use,
train, fine-tune, and distribute state-of-the-art generative models for sciences with
a focus on the design of organic materials. I t is compatible and inter-operable with
many existing libraries and, beyond transformers, it also gives access to diffusion models (diffusers [
Besides established benchmarks for molecular generation such as Moses [
66
GuacaMol [
genetic algorithms, and many evaluation metrics for molecular design, gt4sd
also supports contemporary models like the Regression Transformer for concur-
rent sequence regression and property-driven molecular design [
highly diverse candidate generation [
generation [68
line tools that access a registry of generative models to run or train any model with
a few lines of code. Trained models can be shared to a cloud-hosted model hub
and the library is built to facilitate consumption by containerization or distributed
computing systems. To date, it includes
molecules, proteins, and crystals, and overall hosts
material design, 20
] which includes VAEs, generative adversarial networks (GANs),
]. gt4sd ships with a harmonized interface and a set of command
free webapps [69] and many Jupyter/Colab notebooks.
Generative Toolkit for Scientific Discovery [61]),
64]) or graph generative models (TorchDrug [65]).
] and
30
42
], GFlowNets for
67] and MoLeR for motif-constrained molecule
∼ 50 property prediction endpoints for small
∼ 30 pretrained algorithms for
130 N. Janakarajan et al.
https://t.me/med1917
7.3.3 RXN for Chemistry: Reaction and Synthesis Language
Models
Once a molecule has been selected for experimental validation, a tangible synthesis
route has to be identified. Since the most important tasks in chemical reaction
modeling can be framed as sequence conversion problems, the methodology devel-
74]or
] and
81
58].
71].
oped for natural language translation can be seamlessly translated to chemistry [
In this analogy, atoms are characters, molecules are words, reactions are sentences,
and precursors are translated into a product or vice versa.
The most advanced and flexible library for reaction modeling with LMs is the
package rxn4chemistry [70]. It wraps the API of the IBM RXN for Chem-
istry platform, a
language models for different tasks in reaction chemistry. The primary architecture
is the Molecular Transformer (MT), an autoregressive encoder-decoder model, orig-
inally applied to predict outcomes of chemical reactions in organic chemistry [
Notably, the MT uses a purely data-driven, template-free approach that, unlike many
graph-based models, can directly represent stereochemistry and thus also exhibits
excellent performance on regio- and stereo-selective reactions [
applied to single-step retrosynthesis [
step retrosynthesis models that used, e.g., a hypergraph exploration strategy [
minimized reaction fingerprint distances [
to enzymatic reactions with a tokenization scheme based on enzyme classes, which
facilitated biocatalyzed synthesis planning and paved the road toward more sustainable and green chemistry [
single-step retrosynthesis [
by Thakkar et al. [
to mark a disconnected side in the reactant. Interestingly, an encoder-only derivative of the MT (that replaced the autoregressive decoder with a classification head
and leveraged BERT-style [
predicting reaction classes [
found to encode reaction types thus allowing to map reaction atlases and to perform
reaction similarity search. This gave rise to the rxnfp package for chemical reac-
tion fingerprinting. Interestingly, masked language modeling later led to the discovery
that the learned attention weights of the Transformer are “secretly” performing atom
mapping between products and reactions [
atom mapping without supervision or human labeling bridged the gap between rulebased and data-driven approaches in reaction modeling, making this once tedious
experimental task more efficient.
To achieve automation in organic chemistry, once the precursors for a molecule’s
synthesis route are identified, the subsequent crucial phase involves seeking an
actionable, stepwise synthesis protocol that is ideally amenable for autonomous
execution on a robotic platform, such as IBM RoboRXN . In two seminal works,
Vaucher et al. demonstrated that encoder-decoder Transformers can extract chemical
synthesis actions, first from experimental procedures described in patents [
freely accessible web application that gives access to a rich set of
72]. The MT was
73] and became a vital component for multi-
75]. This approach was later generalized
76]. Derivatives of the MT helped to enhance diversity in
73
] and a prompt-based disconnection scheme proposed
77] significantly improved controllability by allowing the user
78] self-supervised pretraining on reactions) excelled in
79
]. The hidden representations of such a model were
80]. The epiphany that CLMs accomplish
7 Language Models in Molecular Discovery 131
https://t.me/med1917
later predict them directly from the reaction SMILES [82]. Notably, all the aforementioned models are available via the IBM RXN for Chemistry
additionally allows to control and monitor the robotic platform directly from the
web interface. For the daunting task of multistep retrosynthesis planning, RXN also
includes non-transformer-based models like AiZynthFinder [
Search approach built on top of an RNN. Most of the RXN models can also be
executed via the rxn4chemistry Python package.
83], a Monte Carlo Tree
platform which
7.3.4 Specialized Libraries
7.3.4.1 Molecular Property Prediction
HuggingMolecules is a library solely devoted to aggregating, standardizing, and
distributing molecular property prediction LMs [
CLMs, some of them with geometrical and structure-aware inductive biases (e.g.,
the MAT [
models that were trained on SMILES (e.g,. MolBERT [
7.3.4.2 Data Processing
49] or its successor, the R-MAT [50]), while others are pure BERT-based
84]. It contains many encoder-only
46]or ChemBERTA [47]).
RDKit [85] is a library for manipulating molecules in Python. For narrower applications like ML data preparation, several tools exist. First, rxn-chemutils is a
library with chemistry-related utilities from RXN for Chemistry. It includes func-
tionalities for standardizing SMILES (e.g., canonicalization or sanitization) but also
conversions to other representations (e.g., InChI). It harmonizes reaction SMILES
and prepares them for consumption by CLMs, including SMILES augmentation
(by traversing the molecular graph in a non-canonical order) and tokenization.
35, 36
Another library with a similar focus is pytoda [
tion SMILES but implements richer preprocessing utilities, allowing to chain >10
SMILES transformations (e.g., kekulization [
24
(e.g., SELFIES [
]). Similar functionalities are available for proteins, including different
22
PE [
languages (IUPAC, UniRep, or Blosum62) and protein sequence augmentation strate-
87
gies [
facilitate the integration with LMs by storing vocabularies, performing online transformations. Datasets for predicting molecular properties, drug sensitivity, proteinligand affinity, or for self-supervision on small molecules, proteins, or polymers are
available in the public domain.
]. For small molecules, proteins, and polymers, dedicated language classes
]orBigSMILES [86]) and tokenization schemes (e.g., SMILES-
44
]. It does not support reac-
]). It supports different languages
132 N. Janakarajan et al.
https://t.me/med1917
7.3.5 General Purpose Platforms
Several general-purpose platforms for molecular discovery have been launched
recently, sometimes even preserving privacy through federated learning (i.e., decentralized, distributed training). For example, MELLODDY [
effort aimed at cross-pharma federated learning of 2.6 billion confidential activity
data points. Similarly, VirtualFlow [
scale virtual screening that was shown to identify potent KEAP1 inhibitors. With a
focus on de novo drug design, Chemistry42 [
AI with computational and medicinal chemistry techniques.
89] is an open-source platform facilitating large-
90] is a proprietary platform integrating
88] is a collaborative
7.4 Future of Molecular Discovery
A few years ago, the idea of querying an AI model—like one would a search engine—
to not only extract scientific knowledge but also perform computational analyses
was an overly ambitious feat. Scientific thinking comes from the ability to reason,
and AI models cannot reason like humans, yet. However, these models can learn
from humans. Our propensity to document everything has enabled us to train Large
] and GitHub Copilot [92], to mimic
Language Models (LLMs), like ChatGPT [
human responses. When brought into the context of computational science, this
could equip non-experts to confidently conduct computational analyses through welldesigned prompts. With human-in-the-loop, a synergistic effect could be created
where the scientist provides feedback to the model on its output, thus aiding in better
model optimization (a strategy called reinforcement learning from human feedback
(RLHF) that has been proven critical for ChatGPT [
reduce the barrier for individuals from non-scientific backgrounds to gain a more
hands-on experience in conducting scientific analyses without having to go through
formal training in computational analysis.
This section provides a sneak peak into what’s next for molecular discovery.
Riding the LLM wave, the future holds a place for chatbot-like interfaces that
may take care of all things computational in molecular discovery. This includes,
for example, generating and iteratively improving design ideas, synthesis planning,
material purchasing, performing routine safety checks, and validating experiments.
91
93
]). These applications also
7.4.1 The Rise of Foundation Models in Chemistry
Conventionally, neural networks are trained for a single given task to achieve
maximum performance. This essentially renders the models useless for other tasks,
thus requiring a new model for every new task, even when the training domain is the
Соседние файлы в папке Библиотека им академика М.И. Перельмана