Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_5338_Библиотеки_им_академика_М_И_Перельмана
.pdf
92 M. Yamada and M. Sugiyama
https://t.me/med1917
Fig. 5.3 The WL procedure. Given two graphs G and G', after second iteration, new labels of v
'
and v
become different
0
Fig. 5.4 New node labels after first iteration in WL. These subgraphs are enumerated and counted
0
5 Primer on Graph Machine Learning 93
https://t.me/med1917
structure. In ECFPs, such new labels are divided by a vector with the fixed length,
a fingerprint S, and the remainder (
lv mod S ) is stored in the index of S as a bit
that represents the existence of subgraph structures. The algorithm is shown in Algorithm 1. Counts of bits can be used instead of a bit as well. This algorithm can
compute the fingerprint explicitly, but there is a problem of bit collision that indices
of different subgraph structures can be duplicated, which does not occur in the WL
kernel.
5.4 Graph Neural Networks
Graph neural networks (GNNs) are originally introduced in [13, 14] by extending
recursive neural networks into graphs to take topological information of graphs into
account. The recursive definition of node and edge features is adopted to update them
through parametric functions. Although the first GNNs have a drawback about proper
training; the function over graphs must be a “contraction map” and many iterations
are required to reach a stable state [
of GNNs, gated graph sequence neural networks, has been proposed [
To date, a number of variants of GNNs have been proposed, and message passing
neural networks are unified forms of most of them [
is basically equivalent to the WL scheme, that is, it aggregates node features of
neighboring nodes. After aggregating node features, these features are updated as a
(t)
m
new message
e
at the tth iteration round. Hidden features hv are updated through a node update
vw
function
Ut. This procedure is defined as
for a node v through the message function Mt with edge features
v
(t+1)
m
v
15]. To overcome this problem, an improvement
16].
17
]. The message passing scheme
=
w∈N (v)
Mt(h
(t)
(t)
, h
, evw), (5.7)
v
w
where, in the summation, N (v) denotes the set of neighbors of v inagraph G.The
readout phase to extract graph features at the final round T of message passing is
defined as follows:
where R is a readout function that can be a learnable differentiable function. Most of
the existing GNNs can be represented as the message passing scheme. For example,
(t+1)
h
= Ut(h
v
ˆ
y = Rh
(t)
(t+1)
, m
v
(T )
| v ∈ G, (5.9)
v
), (5.8)
v
94 M. Yamada and M. Sugiyama
https://t.me/med1917
Neural Graph Fingerprints [18] are related to the procedure of the circular fingerprints
and WL [
12]. The pseudo-code is shown in Algorithm 2. The main difference between
circular fingerprints and neural graph fingerprints is to embed aggregated node labels
into vectors instead of hashing. After that, the features are updated through the
sigmoid function and allocated the index of vectors. Neural graph fingerprints can
be defined as the message passing framework in the following.
M (hv, hw, ewv) = CONCAT(hw, evw), (5.10)
(t+1)
, (5.11)
m
v
(t)
), (5.12)
t
Ut(h
(t)
, m
v
R = f
(t+1)
v
) = σH
v,t
deg(v)
t
softmax(Wth
where σ is the sigmoid function, Ht is a learned matrix at a timestep t, and R has
(t)
h
skip connections to all previous hidden states
.
v
5 Primer on Graph Machine Learning 95
https://t.me/med1917
Algorithm 1 Circular fingerprints (ECFPs)
96 M. Yamada and M. Sugiyama
https://t.me/med1917
Algorithm 2 Neural graph fingerprints
5 Primer on Graph Machine Learning 97
https://t.me/med1917
Recent graph neural networks are unified with the graph networks (GN) [19]. The
GN framework is based on building blocks of functions: an “update” function
three “aggregation” functions,
ρ.Let u be graph features. Three update functions are
φ and
defined as follows:
'
e
= φe (ek , v
k
'
v
= φv (¯ei, vi, u), (5.14)
k
, v
, u), (5.13)
r
s
k
k
u'= φu (¯e', ¯v', u). (5.15)
Aggregation functions are defined as
'
e→v (E'
¯e
= ρ
i
¯e'= ρ
¯v'= ρ
), (5.16)
i
e→u (E'
), (5.17)
v→u (V'
), (5.18)
where
'
={(e
i
'
, rk , sk )}
k
V'={v
'
={(e
i
rk=i,k =1:|E |
'
}
i=1:|V |
i
'
, rk , sk )}
k
, (5.19)
, (5.20)
, (5.21)
k=1:|E |
E
E'=∪iE
rk is the index of the receiver node, and sk is the index of the sender node. The
mapping
φe acts across all edges, φv across all nodes, and φu across the entire graph
features to compute an update of features obtained from aggregation functions. The
function
ρ aggregates each element and reduces it to a single element as a message,
which must be invariant to permutations; examples of aggregation functions include
the summation, mean, and maximum.
98 M. Yamada and M. Sugiyama
https://t.me/med1917
5.5 Reinforcement Learning
Reinforcement learning is a discipline of machine learning, which aims at making an
agent intelligent by learning the optimal behavior so as to maximize a (often numerical) reward through interaction with the environment, as shown in Fig.
forcement learning has been applied to various domains such as game, robotics, simulation, and chemoinformatics. As for molecular graph generation, model-free and
model-based algorithms can be considered because if molecular graphs are treated
as discrete structures instead of graph embedding, model-based algorithms such as
Monte Carlo tree search (MCTS) can be applied to directly search graph structures
so as to satisfy target properties. In contrast, if graphs are embedded and generated
through graph neural networks and generative models, model-free algorithms, such
as deep Q-network (DQN), deep deterministic policy gradient (DDPG), advantage
actor critic (A2C), soft actor critic (SAC) and proximal policy optimization (PPO),
can be applied to search the latent space to generate graphs. In such cases, other
optimization techniques such as Bayesian optimization and meta-heuristics can be
also applied to search the latent space.
Mathematically, an agent interacts with an environment at t = 0, 1,.... At each
time step t, the agent receives a state
policy computed from the current state, the agent takes the action
environment returns the new state
st and an immediate reward rt. Based on some
at. As a result, the
s
and the numerical reward r
t+1
agent-environment interaction, the agent trains the optimal action t o maximize the
cumulative rewards.
The policy of an agent is denoted as π(at|st), which is the probability of taking
an action
at given the state st. The agent tries to compute actions to maximize the
expected reward, which is defined as
5.5. Rein-
. Through this
t+1
where γ(0 ≤ γ ≤ 1) is a discount rate to control how the future reward is weighted.
γ is equal to zero, agents take an action based on the immediate reward, not
If
considering the future rewards. As
more and more into account.
5.5.1 Value Function Approximation
Most reinforcement learning algorithms are basically based on approximation of a
value function by some machine learning model, such as neural networks. Value
functions represent the values of how good the state or a pair of state and action are
when maximizing the cumulative reward. Given a policy
a at time step t, the state value function can be defined as follows:
Gt = r
t+1
+ γ r
∞
k=0
γ k r
, (5.22)
t+k +1
t+2
+ γ 2 r
+ ··· =
t+3
γ approaches 1, the agents take future rewards
π,astate s, and an action
5 Primer on Graph Machine Learning 99
https://t.me/med1917
Fig. 5.5 The diagram of environment interaction in reinforcement learning
∞
V π (s) = E
k=0
γ k r
| st = s, (5.23)
t+k +1
where E denotes the expected cumulative reward when an agent takes an action based
on policy
π. Similarly, the action state value function when an agent takes an action
based on states is defined as
∞
Qπ (s, a) = E
k=0
γ k r
| st = s, at = a. (5.24)
t+k +1
The above action value function is especially called as a Q-function. The optimal
value function
π∗ is recursively obtained through a Bellman equation as below
∗
π
V
(s) = max
a
r + γ
P(s'|s,π(s))V
'
s
∗
π
(s'), (5.25)
where P is transition probability of the state when the agent takes an action. Based on
the Bellman equation, value functions can be estimated by temporal difference (TD)
learning and Q-learning. In TD learning, the value function is updated in recursive
manner as follows:
V (s) ← V (s) + α(r + γ V (s') − V (s)), (5.26)
where α is a learning rate and r + γ V (s') is called temporal difference target. In
Q-learning, an optimal action is determined by the maximum values from value
functions, that is,
a = maxa Q(s, a). The update process of the value function is
based on TD learning, represented as
Q(s, a) ← Q(s, a) + αr + γ max a Q(s', a) − Q(s, a)
(5.27)
100 M. Yamada and M. Sugiyama
https://t.me/med1917
To build a model for estimating Q-function, any regression models can be used by
minimizing the mean squared Bellman error (MSBE).
L = E
(s,a,s',r)∼D
Q
(s, a) − (r + γ max
target
'
a
Q(s', a')
2
. (5.28)
In the offline settings, when the agent is difficult to interact with the environment
under the situations where the agent cannot take an action easily, such as it is
too dangerous or too costly to conduct an experiment, the Q-function is estimated
using a pre-collected dataset. Fitted Q-iteration [
Randomized Trees, to estimate the Q-function, while Neural Fitted Q-iteration [
20] uses tree-based models, such as
21]
uses neural networks to estimate the Q-function. Recent algorithms such as conservative Q-learning can alleviate the overestimating of Q-values by regularizing MSBE
with a simple Q-value regularizer [
22].
In online settings, the variants of deep Q-networks have been proposed, which have
achieved the state-of-the-art performance on Atari [
23]. The Q-function is estimated
by deep neural networks, and the episodic data buffered based on the Q-function and
trained. In DQN, multiple techniques have been proposed to stabilize the training
of the Q-network and the performance achieved the expert human levels on Atari.
Other applications, such as generating new molecules using DQN, called MolDQN,
24
have been also proposed by Zhou et al. [
].
5.5.2 Policy Gradient and Proximal Policy Optimization
To directly optimize the policy, the policy gradient method is often used. It is based
on the differentiation of the objective:
PG
L
(θ ) = ˆE
where πθ is a stochastic policy function and
ˆ
E
step t. The expectation
indicates the empirical average over a finite batch
t
t
log π
θ (at|st
ˆ
A
of samples. A generalized advantage estimator is computed from exponentially
weighted discounted cumulative average over a reward sequence [
can be used for continuous control.
Proximal policy optimization (PPO) is based on the trust region policy optimiza-
tion (TRPO) method [
]. The key idea of PPO is to use the clipped surrogate objective as
gradient [
27
26] to prevent the high variance while learning in policy
follows:
L
CLIP
(θ ) = ˆE
minr
t
(θ ) ˆAt, clip(rt(θ ), 1 − ϵ, 1 + ϵ) ˆA
t
, (5.29)
) ˆA
t
is an advantage estimator at time
t
25]. This algorithm
, (5.30)
t
5 Primer on Graph Machine Learning 101
https://t.me/med1917
rt(θ ) =
π
π
θ (at|st
(at|st)
θ
old
)
, (5.31)
where ϵ is a clip parameter. The first term inside the min operator is conservative
policy iteration [
28], and the parameter θ of policy is updated through mini-batch
stochastic gradient ascent. The second term modifies the surrogate objective by clipping the probability ratio, which removes the incentive for moving
interval
[1− ϵ, 1+ϵ ] [27]. The PPO also exploits the value function for estimating the
rt outside of the
cumulative rewards, and the entropy term for exploration. In the end, the objective
considering the value function and exploration is defined as follows:
CLIP+VF+S
L
t
(θ ) = ˆE
where, c1 and c2 are coefficients, S denotes the entropy bonus, and L
loss of
Vθ (st) and the target value function. By introducing these techniques, the
t
L
CLIP
t
(θ ) − c1L
VF
(θ ) + c2S[πθ](st), (5.32)
t
VF
is the squared
t
PPO algorithm can outperform other online policy gradient methods on several
benchmarks.
Acknowledgements This work was supported by JST, CREST Grant Number JPMJCR22D3,
Japan, and JSPS KAKENHI Grant Number JP21H03503.
References
1. Hansch C, Fujita T (1964) p-σ − π analysis: a method for the correlation of biological activity
and chemical structure. Journal of the American Chemical Society 86(8):1616–1626
2. Takigawa I, Mamitsuka H (2013) Graph mining: procedure, application to drug discovery and
recent advances. Drug Discovery Today 18(1):50–57
3. Schölkopf B, Smola AJ (2002) Learning with kernels: support vector machines, regularization,
optimization, and beyond. MIT press
4. Haussler D (1999) Convolution kernels on discrete structures. Tech. rep., UCSC-CRL-99-10,
Computer Science Department, UC Santa Cruz.
5. Gärtner T, Flach P, Wrobel S (2003) On graph kernels: Hardness results and efficient alternatives. In: Proceedings of Learning Theory and Kernel Machines: 16th Annual Conference on
Learning Theory and 7th Kernel Workshop (COLT/Kernel 2003), Springer, pp 129–143
6. Borgwardt K, Ghisu E, Llinares-López F, O’Bray L, Rieck B (2020) Graph kernels: State-ofthe-art and future challenges. Foundations and Trends®in Machine Learning 13(5–6):531–712
7. Sugiyama M, Ghisu E, Llinares-López F, Borgwardt KM (2018) graphkernels: R and python
packages for graph comparison. Bioinformatics 34(3):530–532
8. Vishwanathan S, Schraudolph NN, Kondor R, Borgwardt KM (2010) Graph kernels. Journal of
Machine Learning Research 11(40):1201–1242,
html
9. Sugiyama M, Borgwardt KM (2015) Halting in random walk kernels. In: Advances in Neural
Information Processing Systems 28, pp 1630–1638
10. Shervashidze N, Schweitzer P, van Leeuwen EJ, Mehlhorn K, Borgwardt KM (2011) Weisfeilerlehman graph kernels. Journal of Machine Learning Research 12(77):2539–2561,
org/papers/v12/shervashidze11a.html
http://jmlr.org/papers/v11/vishwanathan10a.
http://jmlr.
Соседние файлы в папке Библиотека им академика М.И. Перельмана