Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_5525_Библиотеки_им_академика_М_И_Перельмана.pdf
Artificial Intelligence in Adaptive Radiation Therapy
Figure 9.10. Visual representation of the DSC.
pI I
,
()
XY
I, log
() ()
=II PII
XY
∑∑
XY
XY
⎛
⎜⎟
2
⎝
pI pI
()()
XY
⎞
,
⎠
where p is the probability distribution function of the intensities of X and Y,
respectively, and p(I
, Iy) is the joint probability distribution function.
x
9.4.1.2 Feature-based registration
Feature-based registrations focus not on intensities between the two images, but
instead on a de fined feature present between the two [41, 42]. These features could be
points, lines, or entire structures, such as a contoured brainstem.
Feature-based registrations can broadly be described as minimizing the distance
between points or surfaces of interest. When evaluating points, the sum of squared
distances between the points appears very similar to the SSD equation shown
previously:
N
oint differe nce
1
=−
()
PP
XY
∑
=
i
1
ii
N
2
.
With a surface, this equation changes to evaluate the sum of differences from each
point to that of the surface of interest.
A common metric of evaluating the overlap between two surfaces (2D or 3D) is
the Dice similarity coefficient (DSC):
DSC 2 .
+
YX
XY
∩
=
When two surfaces/volumes completely overlap, the DSC will be equal to 1, and
when there is no overlap present it will be equal to 0, figure 9.10.
9.4.1.3 AI-based feature registration
Both intensity and feature-based registrations as shown above can be roughly called
pre-made/hand-crafted features, that have been selected over time because of their
9-14
Artificial Intelligence in Adaptive Radiation Therapy
success. There are a variety of other options available as well, such as Gabor filters
[43] and hand-crafted features that a user identifies to best match their respective
images [44–46].
One of the most common uses of convolutional neural networks is in asking the
question ‘Are these the best features to guide our registration?’ and ‘Can the model
figure out what features are most useful?’.
9.4.2 Types of registrations
Image registration algorithms have three main components: similarity index, transformation algorithm, and optimization method. For comprehensive descriptions,
the reader is referred to existing chapters and review articles; here we will summarize
for the context of AI in adaptive radiotherapy. Transformation algorithms are
broadly classified into rigid and deformable.
9.4.2.1 Rigid registration
Rigid registration is characterized by applying a series of translations and rotations
to the image. There are three potential translations relating to the left–right,
superior–inferior, and posterior–anterior directions of the image, and three potential
rotations relating to the pitch, roll, and yaw. These values can be used to directly
relate any point in one image to another via the following transformation matrix,
where M
, M2, and M3are the three translations, and Tx, Ty, and Tzare the three
1
rotations, figure 9.11.
A particular thing to note from this transformation is that there is a single, direct
relationship between every point in reference B to reference A.
9.4.2.2 Deformable registration
Deformable, or non-rigid registrations, can be broadly categorized as anything that
falls outside of the framework of rigid registration. Rather than maintaining a linear
and direct relationship from each point in one image space to another, deformable
registration enables each voxel to have its own individual transformation. The
transformation of a single voxel from one space to the next can be described as the
deformation vector field: a matrix of vectors in the x-, y-, and z-plane which describe
how an individual voxel from image A transforms into image B. Common
deformable registration algorithms used in medical applications include Demons
[47] based algorithms, B-spline [48], and the finite element model (FEM) [49, 50].
One of the concerns in deformable registration is this lack of consistent transformation. There is no requirement that every voxel in image A be present in image
Figure 9.11. Transformation matrix describing how a point (x, y, z) transforms from image B to image A.
9-15
Artificial Intelligence in Adaptive Radiation Therapy
B, and vice versa. Deformable registrations often have very complex optimization
issues, with difficulty interpreting results on a voxel-by-voxel basis. For this reason,
there are many considerations regarding the optimization and evaluation of
deformable registration models that need to be considered before clinical use.
When interacting with non-rigid registrations it is equally important to understand the implications of the registration results. For example, bone is not an object
we would expect to readily deform. Just because the evaluation metric is optimal does
not mean the deformation is realistic or accurate. The user should not evaluate a
registration based solely on the same method used to guide the registration, as there
is an implicit bias towards a positive result. For example, if the registration is driven
by contour matching, the DSC between the contours is expected to be very high,
however, this does not promise accurate or physiologically reasonable registration
within the contoured structure.
9.5 AI-based image registration
Within image registration in AI there are multiple groups: supervised learning,
unsupervised learning, reinforcement learning, and generative adversarial networks
(GANs). For all techniques, the metric and methodologies used to evaluate the final
registration is vital as an indication of model efficacy. For all these groups, feature
extraction via convolutional neural networks (new or pre-trained) is the common
bedrock that connects them all. The features extracted can then be fed into existing
models (Demons [47, 51], histogram matching [46], etc) as methods of registration,
or used as inputs for new models.
9.5.1 Supervised learning
Within supervised learning, a previously determined registration between two
images exists and is defined as the ‘correct’ answer. This is often the case in image
registration, although this now limits the model’s ability to learn based on the
accuracy of the previous registration and will perpetuate any systematic errors
present in the ground truth training data.
9.5.1.1 Convolutional neural network registrations
Convolutional neural networks (CNNs) operate by convolving a number of kernels
(typically 3 × 3 or 5 × 5 matrix in a 2D image) to identify features present in the
images being registered. One of the advantages of CNNs is that these convolutions
are not computationally expensive, the same kernel is applied across the space.
However, a disadvantage is that the kernel itself has a relatively small receptive field.
If we imagine a single convolutional kernel used to identify diagonal lines (an ear),
shown in blue in figure 9.12, we can easily see that while the kernel might be able to
identify the ear, the receptive field size might not be large enough to see enough of
the image to overcome a poor initialization of the two images.
This represents a very important concept in many deep learning image registration programs: both image initialization and the overall receptive field size of the
model can have significant impacts on the final accuracy of the prediction. For this
9-16
Artificial Intelligence in Adaptive Radiation Therapy
Figure 9.12. (Left) Representation of two images with an applied shift. The blue box represents a kernel for
identifying diagonal lines (ear kernel). (Middle) Because of a poor initialization point and a lack of a larger
receptive field, the model could incorrectly register the final two images based on this feature as a local
minimum, rather than the (right) ideal registration.
reason, networks often include several layers of convolution and pooling, extracting
both fine and coarse image information to guide the final registration.
9.5.1.2 Generative adversarial networks
Generative adversarial networks (GANs) exist as a combination of two networks: a
decoder which attempts to produce a realistic registration/deformation based on the
input and target, and an encoder which attempts to differentiate between ‘real’
registrations and generated registrations. GANs have shown success in the creation
of synthetic modality images (synthetic CT from MRI [62–64], T1-weighted to T2weighted and vice versa [65]) and applied towards the medical registration task [61].
9.5.2 Unsupervised learning
Rather than be forced to train on explicitly defined answers, unsupervised learning
for image registration requires that the image, either in whole or in part, be reducible
to some form of feature vector that represents the patch/image. This can take the
form of independent component analysis (ICA) [66] or PCA [28]. The model’s
driving goal then is to reduce a complicated space/patch (2D or 3D MRI of the brain
for example) into the principal components/features that represent that space. The
model can then try to reduce the separation between these found features.
9.5.2.1 Reinforcement learning
These architectures are often built with some aspect of long-short-term memory
(LSTM), the goal of which is to make an informed decision based on previous
historical information. In the context of image evaluations, the convolutional LSTM
[67] is often seen [55]. This imagines a 3D scan as a series of 2D images, each highly
correlated to the image immediately above and below. The largest benefit seen by
reinforcement learning, is the ‘reward’ feedback. Other regression-based methods
rely heavily on the image initialization point. Reinforcement learning provides the
9-17
Artificial Intelligence in Adaptive Radiation Therapy
model an ‘action’, with a predefined ‘reward’ acting as a feedback mechanism. Hu
et al have shown that a simple transformation matrix: rotation, scale, and translation can be used as the driving ‘action’, with a landmark based error acting as the
reward function [55].
9.5.3 Registration in ART
The architectures presented in table 9.1 represent only a small portion of the work
performed in AI-based registrations. These rapid registrations can alleviate temporal
burden of recontouring targets and organs at risk, which would preclude online or
real-time ART. They can further enable plan adaption by calculating a predetermined plan on daily anatomy, representing ‘dose of the day’ [68–70]. With
propagated contours and dose, the users can create a new plan going forward, or
summate the dose on the original plan [71].
9.5.4 Commonalities in architectures
There are several deep learning components that are often seen in both supervised
and unsupervised training. Auto-encoders and stacked auto-encoders are often set as
the bedrock of deep learning models in image registration. The goal of such
components is to remove unnecessary information and identify the principal
components of the image.
We have previously identified both feature- and intensity-based metrics that can
drive the registration process. Several AI models have instead flipped the question to
ask: ‘Why not allow the model to figure out what features are important?’
9.5.4.1 Pre-trained convolutional neural networks
A common requirement from these architectures is the ability to identify useful
features to guide the registration process. These features/intensities can be local or
global and can either be user defined or determined by the model itself.
When using pre-trained models, the user must take extra care to match the new
image intensities to that of the images used for original training. Further explanation
of utilizing pre-trained networks is elaborated on later in this chapter within section
9.6.
9.6 Image segmentation
9.6.1 Introduction: coloring by the numbers
In the creation of a 3D treatment plan, the voxel-wise identification, or segmentation, of targets and normal tissues/organs-at-risk (OARs) is a vital part of the
planning process, with significant downstream repercussions [72–74]. Across a
number of treatment sites, there can be a large number of OARs and targets
required for the treatment planning process, creating barriers to re-planning,
adaptation, and dose accumulation efforts due to the significant time required for
manual segmentation. Providing rapid and accurate segmentations is a vital part of
the overall workflow in ART, particularly in the online and real-time setting.
9-18
extraction, learned features
fed into histogram matching
Artificial Intelligence in Adaptive Radiation Therapy
algorithm
deformation prediction
hierarchical learning
(PEHL) —> facilitates 2D/
3D registration via local
feature extraction
layers
LSTM post convolutional
3D registration vector field
Demons iterative approach
network used for feature
extraction, warping
transform from feature
distances
(Continued)
Brain MRI CNN CNN used for feature
deformable registration of MR brain
images
CNN Pose estimation via
T1/T2 Brain MRI CNN Encoder–decoder for
Fluoroscopic video,
registration
x-ray and
transesophageal
echocardiography
probe
2D/3D registration
learning
Reinforcement
nasopharyngeal
carcinoma
patients
Paired CT/MR
registration via reinforcement learning
SPREAD CT [57] CNN Pair of 3D patches generates
Brain MRI CNN 2D affine —> 3D affine with
scale 3D convolutional neural networks
DR and DRR CNN Xception [59] pre-trained
diffeomorphic image registration with
very large deformations
via common representations learning
and differentiable geometric constraints
Table 9.1. List of various authors and techniques for rigid and deformable image registration. This list is by no means exhaustive but represents a small sampling of the
work done by various groups.
Author (year) Title Dataset Technique Explanation
Wu et al [52] Unsupervised deep feature learning for
Yang et al [53] Fast predictive multimodal image
Miao et al [54] A CNN regression approach for real-time
9-19
Hu et al [55] End-to-end multimodal image
Sokooti et al [56] Non-rigid image registration using multi-
Zhao et al [51] Deep adaptive log-Demons:
Liu et al [58] Multimodal medical image registration
from 3D patches, decoder
Auto-encoders identify features
network provides larger
FOV context
Artificial Intelligence in Adaptive Radiation Therapy
from GAN network,
evaluated based on Dice
of two cubic patches,
evaluated based on Dice
a ‘dose of the day’
registration between daily
CBCT and planning CT
structure propagation in
male pelvis from CT to daily
Algorithm comparison of
CBCT
auto-encoders
Brain MRI CNN with stacked
registration framework by
unsupervised deep feature
GAN Deformation fields generated
Retinal, cardiac
representation learning
CNN CNN estimating dissimilarity
images
using generative adversarial networks
A deep metric for multimodal registration Neonatal brain
MRI
Adult brain MRI
CT-CBCT Not AI, Nifty-Reg Evaluating dose summation as
and neck patients: Feasibility study on
using CT-to-CBCT deformable
CT-CBCT Not AI, normalized
registration for ‘dose of the day’
calculations
Deformable image registration for
gradient field
measure distance
and smoothing
adaptive radiotherapy with guaranteed
local rigidity constraints
regularizer
Table 9.1. (Continued )
Author (year) Title Dataset Technique Explanation
Wu et al [60] Scalable high-performance image
Mahapatra et al [61] Deformable medical image registration
(2016) [112]
Simonovsky et al
Veiga et al [70] Toward adaptive radiotherapy for head
(2016 [113]
Konig et al
9-20
Artificial Intelligence in Adaptive Radiation Therapy
As discussed previously, rigid, and deformable registration solutions offer
opportunities for contour propagation from previous imaging. However, these
methodologies all rely on the accuracy of the guiding registration and the accuracy
of the previously defined structures. Strategies of identifying a subset of OARs based
on disease site proximal to the target has also shown success in online ART [9].
Deep learning, particularly convolutional neural networks, have been shown to
be highly successful in the task of OAR segmentation. For OARs, there are multiple
vendor solutions available (MIM Contour Protégé [75], Raystation Deep Learning
Segmentation [76], Radformation AutoContour [77], and Varian AI-Rad
Companion [78]). Likewise, several groups have successfully created models for
the segmentation of OARs present in the brain [79], head and neck [80–83], lung [84–
86], abdomen [87–90], and pelvis [91–94].
Target delineations offer several new difficulties when creating predictive models:
inter- and interobserver variabilities tend to be larger for targets compared to OARs
[72–74, 95–98]. Particularly for clinical target volumes (CTVs) ‘a volume encompassing visible gross tumor volume and subclinical malignant disease’ per ICRU 50
[99], there is often discussion about how generous to make certain contours, which
leads to significant challenges in automating this process.
Convolutional networks of CTVs and gross tumor volumes (GTVs) have been
successfully implemented in a number of sites for brain tumors [100], rectal cancer
[101], nasopharyngeal cancer [102, 103], breast cancer [104], oropharyngeal cancer
[105], and arteriovenous malformations [106].
It is important to note that that with supervised machine learning, there is an
implicit bias of the created model towards the training data. Models trained on
manual contours from institution X could create suboptimal contours for institution
Y if there is a systematic difference in practice between institutions. Furthermore,
quantitative metrics do not always provide adequate clinical relevance of generated
segmentations [94, 107–109]. Including a qualitative assessment of the model’s
predictions is vital for clinical feasibility.
9.6.2 Segmentation networks
A small sampling of convolutional neural networks tasked with the segmentation of
OARs and targets is presented in table 9.2. While these studies all have unique
qualities speci
fically designed to address the task at hand, there are several elements
present across a majority of the networks. First is the difficulty of both small field
high resolution and large field context. As discussed previously in section 9.3,
convolutions suffer from local dependence. Pooling layers can alleviate this dependence by reducing the overall search space for successive convolutions. However, as
demonstrated in figure 9.4, recovering to the original image resolution after
consecutive pooling layers results in a severe loss of fine resolution information.
For this reason, many segmentation architectures incorporate skip connections. The
most famous example of this is the fully convolutional neural network called U-Net
[110]. Here high-resolution information from each part of the encoding path is
directly transferable to the decoding path (figure 9.13).
9-21
Artificial Intelligence in Adaptive Radiation Therapy
segmentation
OAR
skip
connections
multi-head
attention
3D U-Net with
Mindboggle-101
Brain MRI
segmentation
OAR
ARTIX HN CT 3D organ-specific
residual U-Net
—> nn-U-Net
OAR
HN CT Mirada medical
segmentation
segmentation
CNN
U-Net OAR
breast
Left-sided whole
MR/CT
segmentation
U-Net-GAN OAR
thoracic
challenge CT
2017 AAPM
enhancing
Tumor core,
—> dense
Cascaded CNN
BRATS 2018
Brain MR
tumor, edema
segmentation
GTV, high-risk
connection
CT HN Stacked auto-
CTV
segmentation
encoder
Table 9.2. List of various authors and techniques for the segmentation of OARs and targets. This list is by no means exhaustive, but represents a small sampling of the
work done by various groups.
Author (year) Title Dataset Technique Goal
Segmentation using self-attention modules in MRI
images.
Deep 3D neural network for brain structures.
[79]
Laiton-Bonadiez et al
Cubero et al [80] Deep learning-based segmentation of head and neck
OARs with clinical partially labeled data.
organs at risk by deep learning contouring.
Van Dijk et al [83] Improving automatic delineation for head and neck
for improved cardiac sparing.
Morris et al [84] Cardiac substructure segmentation with deep learning
9-22
images using U-Net-GAN.
Dong et al [88] Automatic multiorgan segmentation in thorax CT
Ranjbarzadeh et al [100] Brain tumor segmentation based on deep learning and an
attention mechanism using MRI multi-modality brain
images.
risk oropharyngeal clinical target volumes with built-
Cardenas et al [105] Deep learning algorithm for auto-delineation of high-
in Dice similarity coefficient parameter optimization
function.
Artificial Intelligence in Adaptive Radiation Therapy
Figure 9.13. Basic representation of the U-Net style architecture, called such because of its ‘U’ shape. Note
that information from the encoding (left) side of the architecture is maintained to the decoding (right) side,
facilitating finer resolution segmentation.
Several alternative strategies exist, although at the fundamental level the goal
stays the same: enabling fine resolution evaluation for voxel-wise segmentation,
while gaining coarse resolution information for guided context.
9.6.2.1 Pre-trained convolutional neural networks
Pre-trained classification networks (VGG-16 [30], Xception [111], InceptionV3) can
equally be applied for the task of semantic segmentation. Just as the features which
are being extracted can be applied to other tasks in classification, the early
convolutional layers can be useful in training a new segmentation model. Users
often apply skip connections to the architecture encoder, freezing the previously
trained layers and specifically training an entirely new decoder. After an initial
learning process, the original encoder layers can be ‘unfrozen’ for fine tuning,
figure 9.14.
9.6.3 Best practices
9.6.3.1 Preprocessing
9.6.3.1.1 Intensity values
A major consideration for any convolutional neural network is data pre-processing.
It is always beneficial to normalize input images about the intensity values of
interest. For example, if trying to segment the lungs, thresholding the HUs about
values present in the lungs can help the model focus on important regions.
This window/leveling and thresholding is especially important when utilizing
pre-trained networks. Recall that many pre-trained networks are trained on
9-23