Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_5364_Библиотеки_им_академика_М_И_Перельмана.pdf
11 Ultra-Large-Scale Virtual Screening 305
Our Giga-Scale VS Experience
We started our first venture into billion-scale VS in 2021. On the surface, the
project seemed simple: We knew how to run ligand prepar ation and docking
pipelines for millions of compounds. All we needed to do was to think a little
bigger this time! We quickly realized that generating 3D conformers for over
1.5 billion compounds and docking them to a protein target on an HPC cluster
was not as simple as that. HPCs are shared environments and rely on fair
sharing of computational resources among all users, so we had to ensure that
our study would not disrupt anyone else’s work [33]. Tools were not designed
for handling the huge input, data sizes began to surge, and temporary files
started clogging up the storage space and straining the file system.
The lessons learned in this endeavor went into the preparation of this
section to h elp others avoid similar pitfalls [23, 33].
3.1 Implications of Dataset Size
When working with enumerated chemical libraries, compounds will c ommonly be
obtained in 1D (SMILES) or 2D (sdf) representation. On the million scale, storing
this data is usually straightforward. However, ultra-large enumerated libraries begin
to require a considerable amount of disk space: For example, SMILES strings of
1.56 billion lead-like compounds used in our work had a total size of 89 GB, and
10 billion SMILES consistently were reported to take up around 1 TB of disk space
[4, 23].
To speed up the download process and make the file(s) easier to handle, ultralarge libraries are usually offered in a compressed format (e.g., zipped) and in
multiple parts. If the compression was performed on individual parts, and depending
on the intended use of the library, you may not need to store the fully uncompressed
data on your disk. Nonetheless, keep in mind that uncompressed data will be severalfold larger than the compressed files, depending on the data format and compression
algorithm used. For instance, the compressed version of the 1.75 billion SMILES of
SAVI takes up 440 GB, while the fully uncompressed data would require 4.4 TB of
storage space. The 95 million compounds of CHIPMUNK in compressed sdf format
stand at 109 GB and inflate to 588 GB when uncompressed. Thu s, while one may
simply obtain, decompress, and work with smaller libraries, it is advisable to
carefully consider the amount of required disk space when handling ultra-large
libraries [8]. Some space can often be saved by limiting decompression to parts
that are actively being worked on.
Many VS approaches rely on the availability of 3D conformations for the small
molecules. When obtaining ultra-large libraries in 1D or 2D representation, 3D
conformers first need to be generated. This step, unsurprisingly, is heavily affected
by the library size and requires careful consideration. First, the number of structures
will increase during conformer generation. Therefore, the storage needs and the time
306 I. Pöhner et al.
required to process the library in subsequent steps will grow. How many conformers
per molecule to generate is ultimately method- and case-dependent, but, as a rule of
thumb for ultra-large approaches, one should strive for the lowest acceptable number
of conformers for the chosen method.
For example, the limited conformer generation with a maximum number of eight
tautomers and four stereoisomers per tautomer reported by Sivula et al. [23]inflated
the number of structures in a 1.56 billion lead-like SMILES-based library around 2.4
times. In this study, the resulting 3D conformer databases in the proprietary
Schrödinger Phase database format were 10.3 TB while compressed, and around
22.1 TB when uncompressed. These sizes illustrate the massive increase in disk
space requirement when converting from 1D (89 GB) to 3D (22.1 TB). Similarly,
Michino et al. [21] converted a version of Enamine REAL with 1.12 billion
compounds into binary 3D structure files for their GPU-based shape-screening
approach. In their case, they generated 404 files of 15 GB each, totaling 6 TB.
Thus, although the exact difference is highly format-dependent, even the generation
of a single 3D structure per SMILES string would already significantly increase the
data size, and the problem is aggravated by the compound number inflation during
conformer generation.
Structure inflation therefore needs to be borne in mind when planning 3D ligand
or receptor-based screening campaigns. In addition to the 2.4-fold inflation reported
by Sivula et al. [23], Gentile et al. [34] report conformer generation to inflate their
12.3 billion input SMILES by over three-fold to around 38.7 billion unique structures. Thus, the number of structures to process in subsequent steps of the VS
workflow can easily double or triple and more structures inevitably cause increased
time- and resource requirements.
A final size-related consideration affects the 3D conformer generation and all the
following calculation steps: Many tools will first read all input molecules from one
file into memory. This can become a limiting factor when working with ultra-large
libraries, as the file sizes may exceed the available memory. Thus, it is essential to
assess the memory requirements of different calculation steps, and the dataset needs
to be split into appropriate batches to reflect the available resources.
3.2 Where and Ho w? Guidelines for Planning and Running
Ultra-Large VS Campaigns
In light of the limited throughput and the massive disk space and memory requirements discussed above, it is hardly surprising that most ultra-large-scale screening
campaigns utilize HPC or cloud computing resources. While they have become more
readily available over the past few years, massive computing resources often remain
costly [5, 22, 35]. Consequently, when planning and executing an ultra-large
screening campaign, the computing environment choice is entangled with the
screening timeframe and budget. You may need to benchmark your tool(s), optimize
11 Ultra-Large-Scale Virtual Screening 307
the performance for the chosen computing environment, select appropriate data
splits, and monitor progress and potential errors. Below, we outline some key
questions we and others have found helpful in planning VS on the ultra-large scale
and navigating the specifi c challenges of data-intensive screening campaigns.
Should one use CPU or GPU? This question is often predetermined by your choice
of tools. However, workflows can sometimes run on either CPU or GPU, and
dedicated GPU implementations of otherwise CPU-bound tools exist (e.g., the
docking tool AutoDock4 and its corresponding GPU-reimplementation
AutoDockGPU) [36, 37]. GPUs are best suited when tasks can be split into many
independent subtasks to execute in parallel. With the independence criterion met,
GPUs excel at processing particularly high volumes of data. On the other hand, tasks
that involve sequential steps and require communication between different subtasks
are more suitable for processing on CPUs.
When faced with the choice, throughput may dictate your decision, but cost and
availability should also be considered: GPUs tend to be more expensive and, in an
age of deep learning (DL) popularity, often more sought after. For example, as a rule,
the prediction step in a DL workflow is faster on GPU. However, if CPU resources
are significantly cheaper and awarded more swiftly, it may be more economical to
run predictions on a larger number of CPUs to offset the speed difference [23]. Technically, it would also be possible to run the training step on CPUs. However, the
speed difference is much more pronounced during training, making training on
CPUs impractical. When you choose the processing hardware for your ultra-largescale screening workflows, we recommend factoring in the different costs and
queuing times for different types of resources in planning the various workflow
stages.
Get to know your tool(s)—Profiling and parallelization When planning an ultralarge VS campaign, a prerequisite for optimizing workflows and budgeting time and
resources is a clear picture of the expected resource consumption and behavior of
your tool(s). In most HPC and cloud computing environments, you are required to
specify and reserve computing resour ces, for example, the number of CPUs and
GPUs and the memory required by your job. Resource reservations will often
directly determine your billing but also affect, for example, your queuing time. In
a nutshell, you want to ensure sufficient resources for your workflow to run without
problems but avoid a major resource overhead you’d pay for without being able to
utilize it.
Most computing environments offer special queues or instances for testing to help
you determine how many CPUs/GPUs and how much memory to reserve. Test
environments often have limited resources and a short wall time, but are awarded
swiftly and enable you to profile tools in the relevant production environment. Some
environments will offer graphical tools for monitoring the resources consumed by
your test job(s). Alternatively, HPCs commonly rely on batch job management tools
such as SLURM, which can provide an overview of resource consumption with
dedicated commands such as seff, sacct, or scontrol. It is worth familiarizing yourself
with the relevant profiling commands of the tool used in your environment to review,
308 I. Pöhner et al.
e.g., CPU and GPU utilization and memory consumption of your job. Finally, as
almost all HPC environments are UNIX-based, handy command-line tools such as
time, (h)top, and free can help you in monitoring runtime and resource consumption
in the UNIX command line .
In practice, to decide on the number of computing units to reserve, start by
determining the scaling behavior of your tool(s). Is your tool single-threaded or
internally parallelized, that is, can it utilize more than a single CPU (or GPU)? If so,
ideally, you would find a linear scaling behavior: Whenever you double the number
of computing units, the computing time halves [7]. However, many tools do not
(fully) scale linearly or have a hard limit on the number of computing cores they can
utilize. In the latter case, when exceeding, for example, 8 CPUs, runtime will no
longer improve. Efficient usage of multiple GPUs is, as a rule, more challenging to
implement. Thus, many GPU-based tools rely on using only a single GPU.
Often, you can find hints on the scaling behavior from method publications,
published use cases, repositories, or online documentation. Alternatively or additionally, you can also profile the behavior of your tool(s) yourself in a test environment. Time a small test run with a diverse random set of compounds with
incrementally increasing fixed numbers of computing units (set in your test reservation and the tool—most tools will have settings/command line switches when
internally parallelized). It may also be advisable to use a couple of different dataset
sizes and monitor how the memory footprint is affected by input data size.
Once you established a CPU/GPU count where your tool scales favorably and
assessed its memory footprint in dependence on the input data, you can put it all
together and plan the degree of external parallelization, that is, decide how many
instances of your tool to run in parallel in the production system. Keep license
limitations in mind at this stage, where they apply. Having determined the number of
instances and required CPU, GPU, and memory resources, you should now be able
to estimate the overall runtime depending on your chosen compound library—which
leads to the question of how to distribute an ultra-large dataset between the different
parallel instances.
Splitting your input data As with any rule, there are exceptions, but most of the
time, you want to split your ultra-large input data into at least as many chunks as
processing instances launched. Keep in mind that different compounds have different processing speeds and most similar speeds can be expected for closely related
compounds. Thus, in particular for combinatorial enumerated libraries, you should
randomize the compound order in the workflow input. As a rule, compounds with
high molecular weight and/or many rotatable bonds require higher processing time.
Randomization will help to avoid idling resources due to compound batches with
many compounds having a significantly slower processing speed.
For similar reasons, it may also be beneficial to serially run several smaller
batches rather than one larger batch, as individual runtime differences would
decrease and be less likely to result in idling resources. However, note that many
HPCs use distributed file systems such as Lustre. One caveat is their unfavorable
behavior with heavy file I/O stemming from large numbers of small files. Thus, if
11 Ultra-Large-Scale Virtual Screening 309
you split your data into too many chunks, your runtime may suffer a negative impact
from slow file system response (be also mindful of temporary files in this context). In
addition, HPCs often offer fast local storage solutions based on NVMe/SSD, that can
help to circumvent file I/O problems.
Optimal file sizes should ensure that memory does not become an issue and
individual jobs can finish processing well within the system wall time. If the file
system negatively affects the performance of your workflow, you can often counteract: Many computing environments offer fast local storage resources, and solutions like RAMDisk can move load off the file system and into memory.
An exception applies when you fetch compound data from remote databases
rather than working with files. In this case, network speed will influence your
throughput. You may want to run separate queries for each instance of your tool.
If you massively parallelize and run thousands of instances and queries simultaneously, this can in itself create problems when operating on a single ultra-large
database. For databases, a counteracting measure is a concept called sharding,
reminiscent of batch creation for files, where you dist ribute data into individual
smaller databases.
Finally, if you have different tools in your workflow, you may want to separately
optimize the size of compound batches/shards for each tool, as different tools can
have rather different throughput and requirements [4].
3.3 Ultra-Large VS in a Nutshell: Key Questions
and Trouble-Shooting at a Glance
For all of the pitfalls we discussed above, a lot more could be said, which would be
beyond the scope of this work. We hope, however, to provide a starting point and
relevant keywords for prospective users. To summarize, if you plan on embarking on
your own ultra-large VS campaign and can answer the following questions, you are
off to a good start:
Questions to answer when planning ultra-large VS
How experienced are you in running workflows in an HPC/cloud environment, and is there user
support available for optimization or troubleshooting?
Are your tools single- or multi-threaded?
How do the tools scale with multiple CPUs/GPUs?
What is the memory footprint, and how is it affected by input data sizes?
Are there any license limitations?
How many tools can you run in parallel in available queues/instances of your computing
environment?
What is the available wall time?
Is your input data read from files or a remote database?
Do you require specialized storage solutions to circumvent heavy I/O load on distributed file
systems?
310 I. Pöhner et al.
Table 11.2 Typical VS performance bottlenecks, how to recognize them, and potential
countermeasures
Type Symptoms Solutions
CPU/
GPU
bound
File I/O
bound
Memory
bound
Network
bound
100% utilization on one or more CPU
cores/GPUs, potentially while other
reserved ones idle
Low CPU utilization, high file system
utilization
Low CPU utilization with high swap
memory usage, program crash
Low CPU usage, low file I/O Prestage or preload data prior to
Increase parallelization by running more
instances or increase method internal
parallelization (if possible). Reserve/
spread over more computing resources
Use fast local storage or fast SSD storage. Move (temporary) files to
RAMDisk
Reserve more memory or reduce memory usage by processing smaller chunks
of data
processing
If, despite your best efforts, you find your performance unsatisfactory, Table 11.2
summarizes typical performance bottlenecks and how to recognize them and proposes potential solutions.
Arguably, different methods have their own specific challenges and not all of
them can be covered in this work. Thus, consult other campaigns using the tools of
your choosing. The following sections will provide a vast set of examples to help
you inform your approach. Additionally, for further reading, several published
examples discuss lessons learned from their ultra-large screening or key challenges
faced in method development for large-scale screening (e.g., references [4, 7, 23]),
and published protocols provide step-by-step instructions (e.g., references [24, 38]).
4 Ligands on the Ultra-Large Scale
At the heart of any ultra-large VS are the ultra-large ligand libraries. As we discussed
ultra-large chemical libraries and spaces, we established their massive sizes and that
specialized approaches for liga nd-based VS are required when handling
nonenumerated chemical spaces. This section will discuss examples of ligandbased 2D searches in enumerated and nonenumerated compound collections and
dive deeper into 3D conformer generation strategies and 3D ligand-based VS.
4.1 Ultra-Large 2D Similarity Searches
In an era of ultra-large make-on-demand chemical libraries and vast commercially
available nonenumerated chemical spaces, the search for analogs and substructure
11 Ultra-Large-Scale Virtual Screening 311
Fig. 11.1 Schematic illustration of a classical fingerprint-based search. The query molecule is
transformed into a bit vector describing its structural features, such as “secondary amine” (blue) and
“pyridine heterocycle” (green). In the example, different structural features (bits) are represented by
different colors, where white indicates that the bit is off (i.e., the feature is not present). Once
generated, the query bit vector is compared one by one with similarly constructed bit vectors for
every compound in a large compound database
matches of known ligands represents a straightforward way to study structure–
activity relationships (SARs) and extend the hit pool.
2D similarity searches achieve particularly high throughput and require limited to
no preparation of the ultra-large libraries. They are rarely applied as the sole mode of
VS, but their throughput makes them particularly valuable as prefilters to more
computationally demanding techniques. However, they require prior knowledge of
compounds/scaffolds with the desired properties and have limited potential in
proposing entirely novel scaffolds. Additionally, as also illustrated schematically
in Fig. 11.1 on the example of classical fingerprint-based searches, many approaches
to 2D similarity searches rely on an exhaustive pairwise comparison and become
slow and impractical when proces sing ultra-large libraries [ 9 ].
To enable efficient searches in ultra-large libraries, recent works have proposed
various novel approaches and strategies that address the limited throughput. These
involve, for example, novel types of fingerprints, the use of specialized formats and
database operations, graph-based search methods, or searches in the reagent
space [ 9 ].
In classical fingerprint-based searches (see Fig. 11.1), the bulk of the computation
time is spent on fingerprint generation. Thus, precomputing fingerprint bit population statistics can speed up the process (e.g., in chemf p [39] and MolSoft’s GigaSearch [40]), and some software vendors provide precom puted fingerprints for ultralarge databases to remove the fingerprint computing step altogether. Examples
include Schrödinger’s FP-Sim GPU and OpenEye’s Molecules as a Service
(MaaS) module in their cloud-based Orion software [41, 42]. Schrödinger’s GPU
Similarity search is available for billion-scale library screening via LiveDesign and
has also been released as an open-source incentive version (see Table 11.10 in the
Appendix) [9, 41, 43]. With four GPUs, Schrödinger’s GPU Similarity search was
able to screen 1 billion molecules in around 0.2 s [43]. OpenEye’s MaaS achieved a
2D similarity search of 800 million Enamine REAL molecules in 3 s or less.
However, to achieve this processing speed, data in MaaS is held in memory. MaaS
therefore required an Amazon Web Services (AWS) instance with 768 GB of
312 I. Pöhner et al.
memory, which underlines how memory limitations can become major bottlenecks
even for these relatively fast approac hes [9].
The recent release of ZINC20 strives to support direct similarity searches by
relying on NextMove Software Ltd.’s tools, SmallWorld and Arthor
[20, 44]. SmallWorld utilizes maximum common subgra phs and graph edit distance
in their graph-based similarity search. Importantly, using graph-based instead of
fingerprint-based approaches is more suitable for and adaptable to large databases
[9, 44]. Arthor is focused on high-performance SMARTS pattern and substructure
matching and scales favorably on chemical libraries with hundreds of millions of
compounds while handling relevant tautomers and resonances [9, 45]. ZINC’s
choice of tools underl ines that, while not every similarity search tool fits the bill,
efficient algorithms to sift through the massive state-of-the-art ultra-large libraries
exist.
Another strategy tackles huge make-on-demand combinatorial chemical spaces
by relying on their building blocks. Over 20 years ago, a graph-based strategy,
FTrees-FS, was developed [46]. This method relies on so-called Feature Trees, and
its searches conceptually resemble pharmacophoric-feature-based screens. Only
building blocks and corresponding reactions that match the query get enumerated
into virtual compounds, which allows for the efficient screening of ultra-large
libraries [9, 46].
A more recent topological fingerprint similarity search called SpaceLight can,
likewise, take advantage of combinatorial chemical spaces like the Enamine REAL
Space [9, 47, 48]. It can process more than 20 billion compounds within seconds on a
desktop computer and scales with the number of fragments/building blocks, rather
than the number of virtual products [9]. Finally, another tool, termed SpaceMACS,
was developed to enable a search for specific structural features within ultra-large
chemical spaces [9, 49]. Since SpaceMACS first loads spaces into memory, it
requires significant memory resources, but once loaded, ultra-large libraries and
spaces can be screened for desired structural features within seconds [9].
FTrees-FS has, for example, been one of the various screening strategies
employed by Glaab et al. [50] to discover novel inhibitors of SARS-CoV-2 main
protease (Mpro, or 3CLpro in their work). They performed a ligand-based similarity
search of close to 900 million compounds, largely sourced from ZINC15, which
were compared to previously reported inhibitors of SARS-CoV-2 and SARS-CoV
Mpro. Br üschweiler et al. [51] aimed to target the DNA -binding domain of the
nuclear factor erythroid 2-related factor 2 (NRF2). The binding pocket of NRF2 has
limited druggability and consequently generally displays modest hit rates at best.
Thus, to find additional analogs of their initial fragment-based screening hits, the
authors used the infiniSee software, similar in concept to FTrees-FS, to search 15.5
billion virtual compounds from the Enamine REAL Space [9, 26, 51].
Given their dependency on previous knowledge and limited generalizability, one
should take particular care when selecting the template (sub)structure(s) for 2D
similarity searches. At the same time, the presented application examples demonstrate how 2D similarity searches can act as powerful screening tools in conjunction
with other methods or support the extension of the hit pool in later stages of VS.
11 Ultra-Large-Scale Virtual Screening 313
4.2 Recent Examples of Ultra-Large 3D Ligand-Based
Screening
Recent examples of 3D ligand-based screening cover, in particular, ultra-large 3D
shape similarity searches, as schematically illustrated in Fig. 11.2, which commonly
leverage GPUs for increased throughput.
For example, Grebner et al. [4] utilized the GPU-based FastROCS method in
OpenEye’s cloud-based platform Orion to screen subsets of the Astra Zeneca virtual
chemistry space with up to 12.7 billion compounds. The shape similarity searches
were initiated from known crystallographic ligands of ten diverse target receptors,
including G Protein-Coupled Receptor Kinase 2, estrogen receptor, dipeptidyl
peptidase 4, and MEK1 kinase. In their work, the authors present a comprehensive
benchmark of FastROCS on AWS. They report detailed database generation and
shape search timings and resource requirements for screens of varying sizes: Screening 51 million, 172 million, 1.33 billion, and 12.7 billion molecules, they demonstrate the scalability of 3D shape similarity searches [4].
To address the growing need for novel anti-mycobacterial agents, Michino et al.
[21] targeted the challenging mycobacterial lipoamide dehydrogenase (Lpd). An
extensive experimental high-throughput assay of around 2.5 million compounds had
identified only two lead compounds. To leverage recent advances in ultra-large VS
and extend the hit pool, the authors utilized the GPU Shape screen method by
Schrödinger to screen 1.12 billion compounds from Enamine REAL. Around
800,000 candidates entered a post-filtering process including molecular docking,
and a final set of around 100 candidates was purchased, yielding six novel hit
compounds with IC
narrow chemical space, the new hit compounds displayed some variation compared
to previously known inhibitors. The authors did not report the time invested in
s between 5 and 100 μM. While Lpd requires a rather specific
50
Fig. 11.2 Schematic illustration of a shape-based search. First, multiple 3D conformers are
generated for the query molecule (only one is shown exemplarily in the illustration). For each
conformer, molecular shape and, often, 3D-pharmacophore-type features are determined. Next, the
overlap between the query shape/feature ensemble and predetermined shape and feature descriptors
of all compounds in a large database can be computed
314 I. Pöhner et al.
Table 11.3 Summary of recent ultra-large 2D and 3D ligand-based VS campaigns
Compounds
15.5 billion NRF2 infiniSee NA Brüschweiler
Up to 12.7
billion
1.12 billion Lpd GPU
900 million SARS-CoV-2
Reported throughput covers only the screening process and does not account for database
preparation
NRF2 nuclear factor erythroid 2-related factor 2, Lpd mycobacterial lipoamide dehydrogenase, NA
not available
Target
(s) screened Tool Approx. throughput Reference(s)
et al. [51]
10 diverse
targets
Mpro
FastROCS 480,000 - 1.5 million cpds/
min/GPU
120,000 cpds/min/GPU Michino et al.
shape
FTrees-FS NA Glaab et al. [50]
Grebner et al. [4]
[21]
library preparation, but the shape screen achieved an impressive throughput of
around 120,000 compounds per minute per employed GPU [21].
Both 2D and 3D ligand-based VS examples are summarized in Table 11.3.
4.3 3D Conformer Generation on the Ultra-Large Scale:
Shortcuts and Acceleration Strategies
Whenever approaches like ligand shape similarity searches or docking utilize the
ligand 3D structural infor mation, ligand conformer generation becomes the first step
of the VS workflow. As already discussed in Sect. 3.1, conformer generation will
noticeably increase the amount and size of the ligand data. Additionally, when
working with ultra-large libraries, a significant portion of computing time and
resources will be spent on the generation of the 3D conformers.
For example, to use the GPU-based Rapid Isotere Discovery Engine (RIDE) by
MolSoft [52] for a 3D similarity search of SAVI, the necessary generation of a
conformationally expanded version of the database consumes around eight million
CPU hours [9]. On the other hand, once the 3D ligand database is created, RIDE can
achieve a screening performance of around 500,000 conformations per second per
GPU. Within hours, the full SAVI can thus be screened against a search template
[9]. 3D conformer generation is a single-time effort as the conformationally
expanded database can be used for any number of searches.
Thus, beyond following the general guidelines outlined in the previous section to
optimize ligand conformer generation parameters and tool performance, obtaining
pregenerated 3D conformers can save time and resources. In the spirit of reusable
digital assets, the results of such one-time efforts are increasingly made available to
remove the conformer generation step from others’ workflows altogether.