Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_5884_Библиотеки_им_академика_М_И_Перельмана.pdf
142 P. O. Fernandes and V. G. Maltarollo
provides a comprehensive approach on how the molecular representations evolved
over time.
After the descriptor calculation, the division of the data set into training and test
sets is a critical step to build a robust and generalizable model. This process can be
performed randomly; however, a rational choice can perform better. [83–87] Generally, the aim of rational sampling is to maximize the descriptors space diversity of
the training data and also maintaining the test set representative. There are many
algorithms used in the QSAR context to perform this selection such as sphere
exclusion, Kennard-Stone, k-means clustering, hierarchical clustering analysis
(HCA), minimal test set dissimilarity, Self-Organizing Maps (SOM), and others
[40, 83, 84, 88–91]. One tool available for performing it is the MASSA Algorithm, a
Python software to provide an automated rational sampling using HCA, a clustering
algorithm. [92] For more details on clustering algorithms, please refer to Chap. 5.
To apply the mathematical/statistical methods, a variety of options can be used.
The one with more freedom is through programming, especially using Python, a
popular language in cheminformatics. Several Python librarie s are already available
to deal with chemical structure such as RDKit [93]. Also, the Scikit-learn [94] and
SciPy [95 ] libraries are capable of building and validating QSAR models using
different statistical and machine-learning methods. For those who are not familiar
with programming in Python, the TeachOpenCADD [96] is an amazing tool to learn
about QSAR and other computer-aided drug design techniques. Also, platforms with
a graphical user interface (GUI), such as KNIME [97], Orange [98], and Weka [99],
may be used to build and validate the QSAR model. Both KNIME and Weka had
available workflows for QSAR modeling in the literature [100, 101].
The SYBYL platform was a popular tool for developing different QSAR models.
It is proprietary, nevertheless, there are open tools for performing QSAR modeling.
There is software to perform the whole process of QSAR modeling such as the
webserver 3D-QSAR.com [102]. The server can preprocess the ligand files, perform
the molecular alignment, and apply the CoMFA strategy combined with PLS.
Another similar example is the 3D-QSARpy [103], a Python software for building
3D-QSAR models using MIFs. The fields available are based on the Lennard-Jones
potential, the Coulomb electrostatic interactions, hydrogen bond forming atoms, and
hydrophobic atoms. In this software machine learning algorithms are available to
build the QSAR models.
Another example is the QSARINS [104], a software to perform de build, validate,
and analyze QSAR models. This program applies an MLR approach to buil d the
model and has a series of visualization tools to perform a graphical inspection.
Nevertheless, the descriptors should be calculated using external software such as
the ones described in Table 6.2. Furthermore, the LQTA-QSAR software [105]
allows the construction of 4D-QSAR models from molecular dynamics trajectories,
building MIFs using different types of atoms, ions, and functional groups. After, the
variables are selected using the Ordered Predictor Selection (OPS) algorithm [106]
and then the PLS is applied to build the QSAR models. Other software for
performing QSAR modeling are described in Table 6.3.
6 QSAR and Machine Learning Predictors 143
Table 6.3 Example of available software for QSAR modeling
Name Description Source
VCCLAB [60] A web server containing several methods to calculate
QSAR modeling [107]
CORALQSAR/QSPR
[108]
BILIN The software is suitable for calculating linear regres-
BuildQSAR
[109]
molecular descriptors. It also can apply PLS and neural networks to modeling the data
The software uses PLS-based regression and has
methods for variable selection and model validation
The software builds QSAR models from structures in
the SMILE format using the Monte Carlo technique
for function optimization
sion and performing some nonlinear analyses
The software can perform the mathematical modeling,
hypothesis testing, and graphics/correlation analysis
https://vcclab.org/
lab/
https://lqta.iqm.
unicamp.br/
http://www.insilico.
eu/coral/
https://www.kubinyi.
de/bilin-program.
html
–
4 Validations and Controls
Statistical validation is not the same as experimental validation. For example, QSAR
models are extensively validated by statistical strategies and metrics but its proof-ofconcept for drug design purposes should be the designing of a new chemical entity
followed by the experimental evaluation of the predicted activity/property.
4.1 Internal and External Validation
The early QSAR modeling relies on the division of data sets into two smaller
subsets: training and test sets, usually with 80 and 20% of the total data, respectively.
Of course, this proportion could be different ranging from 90/10 to 50/50.
The training set compounds are employed to teach the algorithm or to extract
information to build the equation that correlates biological activity with the descriptors. However, this very same subset is often used to validate internally the ability of
the QSAR model to make predictions (in this case, often called the robustness of the
model). It is expected that the predictions are highly correlated with experimental
data used to train the model (measured as r
divided into smaller subsets to validate the predictions. This technique is called
cross-validation (or k-fold cross-validation) and consists of a split training set k
times into k sub-subsets, and using one subgroup for validation purposes and the
remaining groups for training the model. In this sense, 5-fold and 10-fold
crossvalidation as well as leave-one-out (LOO) crossvalidation are the most
employed strategies. As the name suggests, LOO leaves one compound out of the
training set, generates a model, predicts the activity for this compound, stores the
predicted value, and then, returns this sample for the original training set, removes
2
). However, the training set should be
144 P. O. Fernandes and V. G. Maltarollo
Fig. 6.6 Validations and controls for training and test/repeated k-Fold/Leave-One-Out (LOO)
another compound, and does this set of tasks until all compounds were used for
prediction. Finally, the stored predicted values are used to calculate errors of
prediction and correlation coefficients (Fig. 6.6).
In contrast, 5-fold crossvalidation does this process only five times instead of
leaving one single compound, this strategy leaves 1/5 (or 20%) of samples out at the
same time, 5 times (Fig. 6.5). Likewise, 10-fold crossvalidation does this sub-subset
division 10 times, leaving 1/10 (10%) of samples out for internal validation 10 times.
Some works validate the final reported models using the strategy leave-many-out,
which consists of varying the sample ratio in sub-subset division in several replicates
(generally, 3 or 5 replicates), systematically to cover small and large sub-subsets of
test compounds. In that sense, this strategy could evaluate the consistency of the
model to keep its robustness throu ghout variation o f sub-training and sub-test sizes
and suggest that the generated model was not obtained as an artifact of a single
replicate of a specific data set composition.
4.1.1 Regression Metrics
For regression models, there are several metrics (Table 6.4) that should be calculated
to assess both robustness and predictability. Robustness could be defined as the
ability to maintain its predictive performance across different data sets and is
6 QSAR and Machine Learning Predictors 145
Table 6.4 Regression metrics commonly used to validate QSAR models
Criteria Metrics
Internal Q
2
Golbraikh and Tropsha
criteria
External Q
2
functions
Concordance correlation
coefficient (CCC)
Roy et al. criteria
Root mean square error
Mean absolute error
n
TR
ðÞ
Q2= 1 -
2
=
R
2
R
= 1 -
0
k =
2
R
and R
0
through the origin, k and k
2
= 1 -
Q
F1
2
Q
= 1 -
F2
2
Q
= 1 -
F3
CCC =
2
= r21 - r2- r
r
m
2
r
=
m
2
Δr
= r
m
2
r
and r
regression function, calculated using the experimental and the
predicted data of the prediction set, forcing respectively the origin
of the axis (r
i =1
n
TR
i = 1
n
EXT
i =1
n
EXT
ðÞ
yi- y
i = 1
n
EXT
yi- y
ðÞ
i =1
n
EXT
i = 1
n
EXT
yiy
i
i =1
≈ 1; k0=
2
n
EXT
y
i
i = 1
02
are calculated forcing the regression line to pass
0
n
EXT
i =1
n
EXT
i = 1
n
EXT
i = 1
n
EXT
i = 1
n
EXT
i = 1
n
EXT
½
i =1
n
EXT
yi- yðÞ2þ
i = 1
2
2
r
þr ′
ðÞ
m
m
2
2
2
- r ′
m
m
2
are respectively the determination coefficients of the
0
2
) or not (r
values on the ordinate axis, using them on the abscissa
n
EXT
RMSE =
MAE =
i =1
n
EXT
yi- y
jj
i = 1
n
EXT
2
2
yi- y
i
≈ 1
2
yi- yðÞ
yi- yðÞyi- y
n
2
EXT
i = 1
2
r
0
i
≈ R2; R
2
yi- y
n
EXT
yiy
i = 1
n
EXT
i = 1
′
are the slope of the regression lines
2
yi- y
ðÞ
l
2
yi- yTRðÞ
2
yi- y
ðÞ
l
= 1 -
2
yi- y
ðÞ
EXT
2
yi- y
=n
ðÞ
yi- y
ðÞ
2
2
0
EXT
n
EXT
i = 1
l
> 0:5
EXT
2
=n
TR
yi- yðÞyi- y
n
EXT
yi- y2þn
i = 1
< 0:2
2
2
). r
0
m
2
yi- y
ðÞ
i
n
EXT
i
2
≈ 1
2
yi- y
i
2
y
i
02
= 1 -
0
≈ 1; y
PRESS
TSS
TSS
= 1 -
EXT y
ðÞ
PRESS
EXT y
EXT
r
0
= kyi; y
i
TR
ðÞ
EXT
PRESS
TSS
EXT y
yi- y
n
EXT
i =1
n
EXT
i = 1
ðÞ
2
r
0
yi- y
ðÞ
i
2
yi- yðÞ
r
0
= k0y
i
=
n
EXT
=
n
EXT
TR
2
≈ R
i
2
is calculated using experimental
TR Training set; EXT External set, yiExperimental data values; yiPredicted data values; y Average
of the experimental data values;
y Average of the predicted values
calculated using different internal validation strategies. The predictability is the
predictive performance of a QSAR model for unseen new molecules (test set). For
example, R
validation strategy, a metric to estimate the robustness of a model. Usually, some
authors report an R
same formula as Q
2
-derived metrics, such as Q2calculated with LOO or other internal
2
(calibration coefficient) metric which is calculated with the
2
but with no internal validation. Together, R2and Q2could be
146 P. O. Fernandes and V. G. Maltarollo
used as a guide to prevent overfitting: ideally, the difference between those two
metrics should be lower than 0.3 (R
2
– Q2> 0.3) [93, 94].
In addition, errors such as RMSE and/or MAE metrics could be calculated in both
internal and external validation procedures. The Root Mean Squared Error (RMSE),
derived from MSE, offers an interpretable measure by maintaining the same units as
the target variable. The Mean Absolute Error (MAE), which measures the average
magnitude of errors in predictions without considering their direction, provides a
straightforward interpretation of prediction accuracy.
Nowadays, it is recommended that a consensus of all metrics should be used in
the evaluation of a given QSAR model’s quality [110]. Those metrics are Q
coefficients discussed by Consonni and colleagues [112], r
2
parameters were intro-
m
duced by Roy and collaborators [111], and CCC from Gramatica work [113] as well
as the error s of predictions.
4.1.2 Classification Metrics
Usually, classification models use the amoun t of true positive and negative (TP and
TN) as well as false-positive and -negative (FP and FN) predictions to calculate
access robustness and predictivity. Those classes of predictions are commonly
organized in the confusion matrix (Fig. 6.7).
Predicted condition
Predicted negativePredicted positive
Actual
condition
False negative (FN)True positive (TP)Positive
True negative (TN)False positive (FP)Negative
2
Fig. 6.7 Elements of a confusion matrix
Table 6.5 Classification metrics applied to validate classification models using the elements of the
confusion matrix
Metric Formula
True Positive Rate (TPR), recall, sensitivity
True Negative Rate (TNR), specificity, selectivity
Accuracy (ACC)
Balanced Accuracy (bACC)
Matthews’ correlation coefficient (MCC)
F1-Score
CK
Enrichment factor at X% of the screened dataset
TPR =
TNR =
ACC =
bACC =
MCC =
F1 Score =
CK =
EF
2TP
2TPþFPþFN
TN
TNþFP
TPþTN
PþN
TPRþTNR
2
p
TPþFPðÞTPþFNðÞTNþFPðÞTNþFNðÞ
actives
dataset
=
actives
X%
dataset
TP × TNðÞ- FP × FNðÞ
TPþFPðÞþTPþFNðÞþTNþFPðÞþTNþFNðÞ
2TP
2TPþFPþFN
2 × TP × TNþFP × FNðÞ
X%
X%
total
total
r
6 QSAR and Machine Learning Predictors 147
Fig. 6.8 Schematic
example of a Receiver
Operating Characteristic
(ROC) Curve and the
interpretation of the Area
Under the ROC Curve
(AUC-ROC)
ROC Curve
True positive rate
False positive rate
Area under the ROC Curve (AUC-ROC)
BetterWorse
0.5 1.0
Random classifier Perfect classifie
Among the metrics to validate classification models (Table 6.5) and quantify their
performance, accuracy (ACC) is a fundamental metric that represents the proportion
of correct predictions out of the total predictions made and may be the most
employed one. However, accuracy alone can be misleading, especially with imbalanced data sets (a set of compounds with more samp les present in one class than
others). To address this, recall or true positive rate (TPR) and sensitivity or true
negative rate (TNR) are also used to avoid models with rates of false positives and
negative predictions. In this sense, those metrics could cover the ability of a given
model to predict in each class. In the same sense, the Matthews’ Correlation
Coefficient (MCC) is a metric used to evaluate the performance of classification
models, particularly in binary classification tasks taking into consideration all four
categories of the confusion matrix: true positives (TP), true negatives (TN), false
positives (FP), and false negatives (FN).
Additionally, the Receiver Operating Characteristic (ROC) Curve and the Area
Under the Curve (AUC) (Fig. 6.8) offer insights into the model’s ability to distinguish between classes across different threshold settings, with the AUC summarizing the overall performance. While the area under the ROC curve (AUC-ROC)
provides an overall assessment of the ability to distinguish between classes (active
and inactive), the enrichment factor metrics and AUC-BEDROC offer information
about this distinguishing ability in the initial fractions (X% of the data set) of the
148 P. O. Fernandes and V. G. Maltarollo
curve. Usually, authors define X% as equal to 0.5, 1.0, 2.0, 5.0, and/or 10.0%. In
other words, these two metrics evaluate the protocol’s ability to distinguish between
molecules with the highest likelihood of being biologically active according to the
probabilities of prediction of a given classification model [ 114–116].
As for regression models, an ensemble of all metrics from the confusion matrix as
well as from ROC curves are considered the state-of-the-art analyses for the model’s
evaluation.
4.2 Applicability Domain
The applicability domain (AD) assessment is another analysis that is not properly a
validation but is extremely important to ensure that external validations were
performed adequately as well as predictions on unseen databases (e.g., virtual
screening) are trustable or not. This importance is also highlighted and highly
recommended by OECD principles for QSAR prediction rule number three. AD is
a modeling step that compares the test set and/or external sets (e.g., libraries for
future applications) with the training set considering their similarity in the space used
to train the model. Structural similarity using fingerprints is also used for this task. In
this sense, compounds similar to training set ones are considered inside the applicability domain and, therefore, predictions done to this sample could be considered
trustworthy rather than just obtained by chance. In contrast, dissimilar compounds
are considered outside the applicability domain and, therefore, the predictions made
for this sample could be considered just a guess or random prediction from the
model. There are a few methods to estimate the applicability domain of a given
model such as based on the descriptor ranges, the knowledge density in the descriptors space around a query compound, or using a convergence of elements to support
the prediction [117].
An interesting example was reported by Serafim and colleagues [118], discussing
the implications of analyzing the applicability domain and illustrating it with an
unsuccessful virtual screening campaign. In this work, hits were submitted to
experimental validation and failed in this step due to the very dissimilar structures
and properties in comparison to the training set of the model employed in the
predictions.
4.3 Randomization Tests
After the internal and external validations, it is important to ensure that the observations in the models are genuine, and not an artifact of random correlation. This
validation is made by shuffling the data, and it is expected that the model using these
scrambled data performs worse than the original model.
6 QSAR and Machine Learning Predictors 149
Y-scrambling is a method to evaluate if the generated model was obtained by
chance. In this method, a model is trained using the same training set of compounds,
and the same set of descriptors but the activity (the Y property) is randomized. As
expected, this new “random” model should perform worse than the original
unscrambled model. In other words, correlation coefficients calculated values should
be lower than acceptable thresholds of quality, and errors of prediction should be
higher than acceptable. Of course, it is recommended that this task should be carried
out several times, usually, reported works use from 20 to 200 runs of Y-scrambling
tests.
Other metrics that should be considered in this validation are the correlation
between q2 calculated with internal validation and the correlation coefficient
between the original Y values and scrambled Y values. In this sense, it is expected
that the degree of Y-randomization is correlated with the degree of “quality loss”
(or decrease in q2, for example) of scrambled models.
There are some variations such as progressive scrambling implemented with the
original CoMFA in the Sybyl package. This method progressively randomizes the Y
values in distinct ranges of biological activity. For example, if a data set compound’s
pIC50 values range from 4 to 10, this method generates several bin windows for
randomization: first, it spli ts the data set into six bins of 1 log unity (randomizes the
pIC50 values of compounds inside that bin); then, in three bins of 2 log unities; and
so on until randomizing the biological activity values in one single bin (the entire
data set such as the traditional Y-scrambling).
Lastly, X-scrambling validation is usually made for classification methods since
the Y value is binary (generally, 0 for inactive and 1 for actives). In this sense,
X-scrambling aims to evaluate the same feature of the model (possibility to be
obtained by chance) but randomizing the X variables.
5 Interpretation
Seeking the improvement of performance and generality, the QSAR models increase
their compl exity, however, this gain in complexity is only reasonable if the simpler
models cannot handle the problem [119]. The prediction understanding promotes
transparency and integrates with expert knowledge promoting useful insights
learned by the observed patterns, and the interpretation can be made in two different
ways, global and individual prediction [48] (Fig. 6.9).
The global model interpretations are related to what were the prioritized variables
and how they are used to perform the predictions. A classic example of global model
interpretations is linear regressions. Analyzi ng the coefficients in the equation, it is
possible to observe how each variable contributes to the model (positively or
negatively) and their magnitude. The same analogy can be applied in linear Support
Vector Machine models, observing the feature weights. Tree-based algorithms are
150 P. O. Fernandes and V. G. Maltarollo
Fig. 6.9 Strategies to interpret a QSAR model. The first one is observing globally how the features
impact the model, and the second one is using an individual prediction to see how the feature in the
molecule impacts the prediction
also globally interpreted in terms of information theory, estimating the magnitude of
a contribution [48]. Besides the algorithm interpretation, the feature permutation is a
viable and method-independent strategy for interpreting the model globally [120].
In contrast to the global interpretation are the individual predictions. This
approach tries to understand what patterns and features are important in the prediction of a specific compound. Not necessarily a globally important feature will be
important in the prediction of a single compound and both can be complementary.
Several agnostic methods have been used to interpret individual predictions such as
Locally Interpretable Model-Agnostic Explanations (LIME) [121] and Shapley
Additive Explanations (SHAP) [122]. LIME generates local explanations by
perturbing input features and observing how these perturbations affect the model’ s
predictions, resulting in a contribution that explains how each feature influenced the
prediction. SHAP also quantifies the contribution of each feature to the model’s
output but is based on the principle of Shapley values derived from cooperative
game theory. The importan ce of each feature is accessed based on its contribution to
the prediction relative to all possible feature combinations. SHAP can provide both
local interpretations and global insights. The local interpretation is obtained when
Shapley values are calculated for each feature to explain the contribution of that
specific prediction and globally when this process is made across all predictions in
the data set.
6 Practical Advice During QSAR Modeling
The process of modeling can be quite challenging in the first look. In summary,
based on the work experience, some questions are described in Table and the
answers may help guide when planning a QSAR study.
6 QSAR and Machine Learning Predictors 151
Useful considerations during the QSAR model development
Questions to answer when planning a QSAR modeling
Data collection and preprocessing
How were the experimental data generated? Is there a unified protocol? How to deal with these
differences?
Is the training/test data representative in terms of protocol and terms of the descriptors used?
How will missing values, outliers, and redundant descriptors be handled?
Will the data need to be standardized or transformed?
Model selection
Which modeling techniques (e.g., linear regression, neural networks, and support vector
machines) will be employed?
How will the model’s complexity be determined and optimized?
Model validation
What validation methods (e.g., cross-validation, external validation) will be used to assess model
performance?
How will overfitting be addressed?
Interpretation and visualization
How will the QSAR model results be interpreted in the context of chemical and biological
knowledge?
What visualization techniques will be utilized to aid in understanding the relationships between
chemical structure and activity?
Application and deployment
How will the QSAR model be applied to predict the activity of new compounds?
How will the models be accessible to nonexperts?
What are the limitations and assumptions of the model, and how will they be communicated?
Iterative improvement
How will the pipeline be iteratively improved based on feedback and new data?
Are there opportunities to incorporate additional data sources or enhance modeling techniques?
7 Application
As examples of QSAR applications in the drug discovery field, five works published
between 2023 and 2024 were described in detail. As selection criteria, the selected
articles used computational techniques to identify active compounds that were
experimentally validated.
Moreira-Filho et al. [123] used classification and regression machine learning
models to predict the schistosomicidal activity of compounds that had not been
experimentally tested. Subsequently, the prioritized compounds underwent testing
on both schistosomula and adult stages of Schistosoma mansoni. Among them, four
compounds demonstrated substantial activity against schistosomula, with 50%
effective concentration values ranging from 9.8 to 32.5 μM, while showing no
toxicity in animal and human cell lines.