Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_5525_Библиотеки_им_академика_М_И_Перельмана.pdf
Artificial Intelligence in Adaptive Radiation Therapy
9.2.2 What is deep learning?
Deep learning can be defined as a subset within machine learning. Whereas machine
learning combines several features directly into an output, ‘deep’ learning is often
associated as having several intermediary or ‘hidden’ layers. A single hidden layer,
as shown in figure 9.1, would most appropriately be referred to as a ‘shallow’
network, with ‘deep’ requiring two or more hidden layers. The benefit of multiple
hidden layers is the ability to represent predictions beyond simple linear regression
when non-linear activation functions are used (activation functions are spoken of
more in the following sections). The downside to the increase in the prediction
capabilities of the model is the decrease in interpretability of the model predictions.
Deep learning is often cited as being a ‘black box’, although this does not mean it is
impossible to crack open the box and take a peek inside now and again.
9.3 Deep learning: the basic components
While deep learning has evolved considerably over the past decades, most models
can be broken down into a combination of several key components. Having a solid
understanding of each part will help one understand better how more complex
models are generated in the future.
9.3.1 Convolutional neural networks: looking at the picture
Convolutional layers form the bedrock of deep learning image classification,
segmentation, and registration models. The basis of these three tasks all relies on
the same thing: the computer needs to understand what is occurring within an image.
Humans, as well as most animals, have the benefit of eyes which enable us to capture
a large amount of information within our field of view. For a computer this can be a
relatively daunting task.
The first thing which needs to be defined is the computer’s receptive field of view.
A convolution is very simply the creation of a new matrix by multiplication and
resultant addition of elements on an image. A visual representation of this concept is
shown in figure 9.2. Here, the input image is convolved with a simple three-by-three
kernel to create a resultant feature image.
If the kernel were a vertical line surrounded by zeros, the output feature map
would show where vertical lines are present in the image.
One of the downsides of convolutions is that they are locally dependent. This means
that since a kernel size is fixed, images of varying zoom or scale might not properly
align with a previously defined kernel. A natural question which arises from this, is
why not push towards large kernels which are able to cover large portions of the input
image? There are two main concerns with this approach: (i) the larger the kernel, the
more computational resources are required to perform the convolution and addition
and (ii) larger kernels have more variables present, where an n by m matrix will have n
× m + 1 variables. A three-by-three kernel has nine variables (plus a bias to make ten
total), while a nine-by-nine kernel has 81 variables (plus a bias to make 82).
9-4
Artificial Intelligence in Adaptive Radiation Therapy
Figure 9.2. General representation of a convolution. The input image (left) convolved with the 3 × 3 kernel
(middle) creates a feature map (right).
Figure 9.3. Example of a two-by-two max-pooling layer.
This modest increase in kernel size creates significantly more variables to train. A
remedy to this local dependency issue comes in the form of pooling layers.
9.3.2 Pooling layers: keeping what matters most
Pooling layers are a way of taking only the most important aspect of a generated
feature map. While they come in a variety of forms, a very commonly used pooling
layer is max-pooling. Pooling layers are like convolutional layers in that they are a
matrix, typically of size two-by-two for two-dimensional images, but rather than
multiplying and adding, a max-pooling layer will simply take the maximum value
within the defined matrix size, figure 9.3.
One of the most immediate implications of this is that the feature map itself has
now decreased in size by a factor of 2 in each direction. This can either be seen as
beneficial or detrimental, depending on the desired output. From a positive
perspective, max-pooling layers have multiple benefits. The first of these benefits
is that the most intense, or important, feature is maintained, while presumably
less important features are removed. Second, subsequent convolutions on the
9-5
Artificial Intelligence in Adaptive Radiation Therapy
post-max-pooled image require significantly less memory—as stated before, the
feature map has been reduced in size. A 2 × 2 max-pooled image will be 1/4 the
original image size, meaning less memory is required for the convolutional
operation. Third, subsequent convolutions now cover a relative area that is larger
than the original image size. A three-by-three kernel on an image with resolution of
1 mm × 1 mm per voxel would have a receptive field size of 3 mm by 3 mm. After
max-pooling, the resolution of the image is 2 mm × 2 mm per voxel, so a three-bythree kernel has a relative receptive field of view of 6 mm by 6 mm. This is
particularly useful in images where the scale/zoom of the image can vary. If the
model is designed to identify that a dog is present within an image, it should not
matter how large or small the dog presented is.
Despite these benefits for image classification, if the user wishes to identify
individual voxels relating to a class (segmentation), the pooling layers can lead to a
significant decrease in resolution. A representation of max-pooling followed by
bilinear up sampling to the original image size is shown in figure 9.4. Note how the
fine resolution text is quickly lost by the fourth pooling, while general features such
sa color can be maintained for multiple layers. Maintaining the higher-resolution
information for later decision making is a main factor in the popularity of the U-Net
style architecture discussed later in this chapter.
One thing we have yet to cover is how these features can lead to a prediction. A
model needs a method of combining these features into a final prediction; this
combination of features is defined as a fully connected layer.
Figure 9.4. Illustration of multiple max-pooling layers and bilinear resampling to illustrate how fine resolution
information (text) is quickly lost, while large information (general colors) remains.
9-6
)
Artificial Intelligence in Adaptive Radiation Therapy
9.3.3 Fully connected (dense) layers: bringing it all together
Fully connected, or dense, layers can be imaged as the opposites of convolutional
layers. While convolutional layers are locally dependent, the fully connected layer
will combine every aspect of the input into an output, figure 9.5. The number of
inputs and outputs in a dense layer can vary and are user defined. The mathematical
value of each output is
out
i
out
=×+
Fh W B
(
i
−
i
1
ii
, where F is an activation function,
W is the weighting matrix, and B is the bias.
Within convolutional neural networks there can be any number of convolutions
and pooling layers. These maps can be represented as several n by m matrices, where
n and m are likely smaller than the original image dimensions. In order to feed these
features into a dense layer they must first be ‘flattened’. This can be imaged as taking
the feature image and laying it into a single vector: converting the n-by-m matrix
into an n × m vector. These features can then be used in the final classification. For
the example in figure 9.6, we might say that if ears, whiskers, and paws are present,
then a dog is present in the image.
The low-level and abstract features of lines, curves, or fuzziness are most likely
to appear early in the architecture. Higher level features, such as ears and whiskers,
can be imagined arising deeper, as a combination of overlapping low-level
features. Finally, all of these features can be combined together after flattening
to identify if a dog is present somewhere in the image. Note that our output does
nothing to identify where the dog is present. Furthermore, if the model is truly
identifying whether ears, whiskers, and paws are present, it very likely confuses a
cat for a dog.
Figure 9.5. A fully connected, or ‘dense’ layer. Output features are a function of weights from each input
feature.
9-7
Artificial Intelligence in Adaptive Radiation Therapy
Figure 9.6. Generalized example of a convolutional neural network for the identification of a dog. The first
convolutional layers are very low-level; lines, edges, fuzziness. Further down, these features combine to
identify ears, paws, or whiskers, before finally feeding into the decision-making process of a fully connected
layer.
9.3.4 Activations
Throughout the previous sections we have discussed features, convolutions, and
dense layers, but have purposefully been neglecting a very important step in the
process. Part of the true power in deep learning comes not only from these critical
parts, but also from the final step after each: the activation function.
9.3.4.1 Linear activations
The activation function is a deceivingly simple thing. A simple statement can
summarize its purpose: using a defined function, map an input value to an output
value. One of the most basic forms of activation is a linear activation: f(x) = mx + b,
which can have a scaling (m) and bias (b) impact on the outputs. While linear
activations can be a useful way of scaling and shifting values, stacking multiple
linearly activated layers does not add any new information to the system. To
demonstrate this, imagine two linear feed-forward layers, where x is the original
values:
=+ymxb
11
1
=+ymyb.
2
2
2
1
The entirety of this network y2can be defined and simplified as
=++→++→+ymmxbbmmxmbbMxB,
()
21 1 2 21 21 2
2
where m2× b1+ b2can be combined as a single bias, B, and m2× m1can be a single
scalar, M. This means that regardless of how many linearly activated layers are
placed together, they are only capable of expressing linear relationships. For
modeling something with non-linear such as gravity, they would inevitably fall
short.
9-8
Artificial Intelligence in Adaptive Radiation Therapy
9.3.4.2 Non-linear activations
The power of multiple layers becomes more apparent as we transition towards nonlinear activation functions. Stacking multiple non-linearly activating layers can add
increasing complexity to the model’s predictive abilities. For example, if the
activation function were f(x) = mx
representational ability of several polynomials: x
2
+ gx + c, two such layers would have the
4
, x3, x2, x, and bias.
A set of four commonly seen non-linear activation functions is shown in
figure 9.7.
Figure 9.7. List of common activation functions, sigmoid, rectified linear unit (ReLU), leaky ReLU, and
exponential linear unit (ELU).
9.3.4.3 Sigmoid activation
The sigmoid activation function, while not the most commonly seen in the
intermediate layers of deep learning architectures, is a common final activation
function for a binary prediction. This function is commonly expressed as being
synonymous with neurons firing within the brain. Given a number of inputs, if the
value exceeds some critical threshold the neuron will fire. A sigmoid activation is an
appropriate activation for a binary classification system such as our dog model
shown in figure 9.6, where ‘0’ would indicate no dog, and ‘1’ would indicate the
presence of a dog.
In the case of multiple classification options, the sigmoid activation is often
exchanged for the soft-max activation. This provides a probability of a class scaled
to the sum of all class’s probability:
z
c
==
ycx
∣)
(
C
∑
=
j
e
,
z
j
e
1
where c is a particular class and C is the total number of classes.
9.3.4.4 Non-linear activations: rectified linear
The rectified linear unit activation (ReLU) was one of the first and most popular
activation functions used in modern deep learning architectures. The function is
relatively simple: if x is less than 0, the returned value is 0, otherwise the returned
value is x. The ‘leaky’ ReLU and exponential linear unit (ELU) are both variations
9-9
Artificial Intelligence in Adaptive Radiation Therapy
on the ReLU, where the differences focus on what occurs when values less than 0 are
presented to the function. One of the major issues with ReLU was ‘dying’ nodes,
meaning if a kernel initialized at a value less than 0, all gradient through the node
‘died’. The leaky and ELU activation allow at least some gradient.
An important question to raise at this point is ‘Why use these non-linear
activations?’ The answer to this lies in the most interesting part of deep learning:
loss and back-propagation.
9.3.5 Loss: driving the model
When making a prediction, we need some method of identifying the ‘correctness’ of
the model prediction. This metric is referred to as the loss. There are several ways of
expressing a model loss, although, for mathematical reasons, it is desired that this
loss is something that the model is attempting to minimize. An in-depth discussion of
back-propagation and gradient descent is beyond the scope of this chapter, please
refer to other texts discussed at the beginning of this chapter [25, 26] for a detailed
explanation.
9.3.6 Auto-encoders: remove the noise
If one were to be told to draw a bicycle, remove while playing a game of
Telestrations, they would likely draw out two simple circles as wheels, a frame
connecting them, and a set of handlebars. Perhaps peddles and spokes on the wheels
would be included for the more artistically inclined, and yet, this drawing of a
bicycle would be relatively far removed from the appearance of an actual bicycle.
While there are hundreds, if not thousands, of different types of bicycles, most
people would be able to review this crude drawing and conclude that the image is
supposed to be that of a bicycle. The human brain is amazing in this regard: a
bicycle, in all its many shapes and forms, can be condensed down to just a few simple
aspects. This ‘condensing of important features or removing of unnecessary parts
(noise) is the basis of auto-encoders [27].
The goal of an auto-encoder is to simplify, or compress, a set of inputs into its
most important aspects, figure 9.8. By reducing the number of features and
accurately reproducing the input, the model is able to identify what are the most
important aspects of the input. This can similarly be used as a form of principal
component analysis (PCA) [28] and is a common building block of both supervised
and unsupervised learning models. The so-called ‘stacked’ auto-encoder is simply a
combination of auto-encoders with significantly noted improvement in model
predictive capability [29]. The training of these models is often pieced together,
with each layer trained separately before being combined and fine-tuned in the final
model.
9.3.7 Supervised versus unsupervised learning
Distinctions between supervised and unsupervised learning can be simplified as a
statement about data labeling. If the user understands what the desired output
should be, e.g. ‘is a stop sign present in this image?’ and uses those labels to train a
9-10
Artificial Intelligence in Adaptive Radiation Therapy
Figure 9.8. Basic example of an auto-encoder.
model, we are operating under ‘supervised’ learning. However, if we simply had
many pictures of stop signs and other signs with no ‘ground truth’ labeling, we
would require an unsupervised learning technique. This is not to say that unsuper-
vised learning does not require an equal amount of work for training as a supervised
learning model.
9.3.8 Pre-trained convolutional neural networks
Much of the previous discussion revolves around feature extraction (convolutions),
refinement (pooling), and combination of those features (dense layers), all with the
understanding that relevant features are being identified by the model. However, this
is not a guarantee. When first creating a convolutional network, such as one for
image classification by the visual geometry group 16 (VGG-16) [30], figure 9.9, every
convolutional kernel is going to be a randomly generated distribution of numbers.
Deep learning researchers all suffer from this issue. Updating these random
kernels based on correct predictions is a labor intensive and sensitive process that
occupies most, if not all, of a researcher’s time and effort when beginning a new
classification process. The VGG-16 model was trained on the ImageNet challenge,
which contains more than 14 million images. This is far above what most medical
researchers have available for training their own models. However, this can still be
used to our advantage with the concept of using this pre-trained model.
9-11
Artificial Intelligence in Adaptive Radiation Therapy
Figure 9.9. Representation of the visual geometry group (VGG-16) classification architecture.
The fundamental basis of using a pre-trained model is that certain low-level
features are ubiquitous. As humans, we are able to identify many different objects all
of which are built on the same fundamental pieces: lines, curves, edges, etc. We are
then able to take this pre-trained model that has been trained on thousands of
different images and utilize everything before the fully connected layers as a feature
extraction network. This saves potentially countless hours of training; by freezing
the convolution layers, we can simply train the end classification. Note: Even after a
new classification model is trained, it is common to un-freeze the earlier layers in a
process of fine tuning.
9.4 Image registration: bringing two images together
Throughout the course of a patient’s care there is a potential for multiple forms of
medical imaging to be acquired for various purposes, e.g. defining tumor boundaries, identifying surrounding critical normal tissue, and characterizing tumor and
normal tissue function. For example, a magnetic resonance image (MRI) can exhibit
exquisite contrast of soft tissues in ways that computed tomography (CT) scans
cannot, and an FDG positron emission tomography (PET) scan is able to identify
regions of metabolic hyperactivity. Currently, standard practice within radiation
oncology includes a CT scan enabling the visualization of a desired treatment site
and enabling radiation dose calculation, although there is significant research into
MRI based generation of electron density.
An ability to refer to multiple imaging modalities to evaluate tumor extent or
normal tissues is vital to the decision-making and treatment planning process.
Unfortunately, these PET, MRI, and CT scans are rarely acquired at the same time
or with exactly consistent positioning of the patient. These changes in time,
positioning, and/or anatomy have led to the necessity of tools to align different
modalities/images about a region of interest in the anatomy. This aligning of
different modalities or images is referred to as image registration.
This is not to say that image registration is only important during the planning
process. Accurate alignment of the planning images with treatment imaging (conebeam CT, fan-beam CT, MRI) is equally vital to a safe and successful treatment.
9-12
Artificial Intelligence in Adaptive Radiation Therapy
With ART, it is often vital to understand the distribution of previously delivered
radiation for guidance in future planning. This understanding relies on accurate
representation and registration of the new imaging with previous plans. It is equally
important to understand how a registration is evaluated and functions, the
‘registration metric’, per AAPM Task Group 132 [31]. After all, when registering
two images, how does one know when the registration is ‘good’?
There are multiple ways in which a registration can be evaluated. Often, the
registration is focused on a particular region of clinical relevance. Identifying the
region of relevance is the first and most important aspect of image registration.
9.4.1 Registration similarity metrics
There are broadly two ways of evaluating/driving a registration: intensity-based and
feature-based.
9.4.1.1 Intensity based registration
Intensity based registration can be defined as a comparison of intensities between the
two images, either as a whole, or about a defined sub-region of interest in the image
[32–34]. When operating between two images of the same modality, simple
comparisons of the intensity values between the two can offer a reasonable
evaluation of the registration, possible as the sum of squared differences (SSD)
[35, 36]:
N
1
=−
()
IISSD
XY
∑
N
=
i
1
2
ii
,
where I is the intensity and subscript X and Y refer to two images across the total
number of voxels being evaluated. While this metric can be very useful, it suffers
when the voxel intensities throughout the two images vary (a multi-phase contrast
enhanced CT). To this end, the normalized cross-correlation (NCC) [36–38]isa
wonderful metric, accounting for the differences in intensity between the two images,
although assuming that relatively high and low intensities correlate with each other:
n
−−
xxyy
()()
i
∑
=
i
=
CC .
1
n
−−
xx yy
()()
i
∑∑
==
i
1
i
n
2
i
1
2
i
An important note from both methods is that they require the intensities between the
images to be correlated: a bright spot in one image corresponds to a bright spot in
the other, etc. For multi-modality images (CT to MR), this is not a guarantee.
A commonly used metric for the registration of images of different modalities is
mutual information [39, 40]. This metric is based on the mutual probabilities
between the two images and has no reliance on the absolute intensity of the images:
9-13