Добавил:
Sekretar
kiopkiopkiop18@yandex.ru
t.me/Prokururor I Вовсе не секретарь, но почту проверяю
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Ординатура / Хирургия / Библиотека им академика М.И. Перельмана / Книга_1260_Библиотеки_им_академика_М_И_Перельмана
.pdf
Augmented reality for laparoscopic liver surgery 49
the preoperative information from different perspectives.
This can reduce the mental effort required compared with
classic computed tomography (CT) analysis. When used
to augment the surgical view, the display also allows
structures lying beneath the surface or out of the endoscopic field of view to be visualized, improving spatial
awareness and aiding in structure localization. Augmented reality can also be used to overcome other challenges in laparoscopic surgery. For example, a surgeon
might be able to compensate for the loss of depth perception with the laparoscopic image by overlaying computergenerated distance maps of specific structures of interest,
or by sounding an alert when structures of interest are
approached.
Using 3D anatomical reconstructions, surgical procedures can also be defined preoperatively in order to
optimize surgical outcomes through, for instance, the
maximization of preserved functionaltissueor the removal
of a sufficient resection margin. These plans can then be
viewed interactively, relative to tool positions during procedures. Additionally, the display of anatomy or planned
locations on the patient can aid in port placement or the
labeling of structures and interactive task descriptions can
be utilized as a powerful training tool during laparoscopic
procedures.
The following sections present an overview of how
these augmented views for laparoscopic surgery can be
achieved and provide examples of their use during ARguided laparoscopic procedures.
4.3 How is augmented
reality achieved?
An augmented reality view of the surgical scene is composed of two primary components: the original or “real”
view of the patient or surgical scene and an overlaid
“virtual” scene. Optimally, the real view of the patient
used in an augmented view would be the normal surgical
view. Therefore, in laparoscopic surgery, the laparoscopic
image projected by the operating camera is used. Based on
this real scene, and using a virtual dataset, an augmented
view is created. For example, virtual models of real
anatomy, labels or interactive measurements can be created from medical imaging data. The next step is that the
virtual data must be rendered at the same view as the real
scene in order to align the two scenes. The alignment
requires registration of the two datasets, and for automatic image alignment, calibration of the laparoscope in
order to determine its view relative to its tracked position.
Once aligned, the virtual data can be superimposed onto
the real scene to create a single merged view. Finally, any
dynamic components in the virtual scene must be tracked
relative to the patient and updated in the virtual scene
accordingly. The primary processes in the achievement of
an augmented endoscopic view are depicted in Figure 4.2
and are discussed in more detail in the following sections.
4.3.1 Generation of virtual data
The virtual scene can be composed from a wide range of
computer-generated data but is typicallyconstructed from
preoperativeor intraoperative images. Such imagery often
takes the form of virtual 3D reconstructions of anatomical
structures of interest. While any 3D imaging modality
providing sufficient contrast, resolution, and quality can
be used for model construction, preoperative CT or magnetic resonance imaging (MRI) images are typically used.
Three-dimensional anatomical reconstructions can take
one of two forms: a volume-rendered image or 3D surface
models. While the former can be easily and automatically
created by applying a color and opacity transfer function,
3D surface models provide an intuitive visualizationof the
patientanatomy and, through the definitionof anatomical
boundaries, allow for the automatic calculation of
Figure 4.2 Overview of the primary steps required to create an augmented reality view.
50 Chapter 4
Figure 4.3 Three-dimensional anatomical surface models are created from the selection of anatomy in each image slice.
Figure 4.4 Virtual anatomical models of the liver, including organ surface, vascular structures, and tumors with liver segments and
vascular supply (left) and the planned resection margin (right).
geometric parameters such as distances and volumes.
Three-dimensional surface models of structures of interest, such as organ surfaces,vessels, and tumors,are created
by identifying and separating the structures through each
image slice in a process known as segmentation (Figure
4.3). Nevertheless, augmented reality is not restricted to
anatomical models. Functional information such as liver
segments and vascular supply can also be modeled from
image data and included in the virtual scene. Additionally,
surgical plans incorporating information such as safe
resection margins can also be fitted onto generated 3D
models and included in the augmented view for intraoperative guidance (Figure 4.4).
augmented reality view during surgery. As the real view
changes, or as objects of interest within the real scene
change their position, the virtual scene must be updated,
realigned, and displayed faster than the eye can detect.
The most basic method of data alignment consists of
manually transforming images of the virtual scene such
that features within the virtual model align to those in the
laparoscopic image. This technique, as employed by Marescaux et al. [3], requires the preoperatively defined
virtual scene to be manually resized and oriented on a
monitor to the view of the laparoscopic image according
to features visible in both views. Such a technique can be
performed with basic technology and is unaffected by
organ motion. However, the need for an operator dedi-
4.3.2 Data alignment
Accurately aligning the virtual scene with the real scene is
the primary challenge pertaining to the creation of an
cated to image alignment and suitable devices on which
the alignment can be performed decreases the feasibility
of such a technique. Additionally, the accuracy of this
Augmented reality for laparoscopic liver surgery 51
method is highly dependent on the user’s ability to align
the image correctly at a speed corresponding to the
movement of the laparoscope.
Alternatively, data alignment can be performed automaticallyby employing an external position measurement
system to track the position of the laparoscopeand surgical
instruments relative to the patient within a common coordinate system. Determining the position of the patient
anatomy relative to the virtual model can be achieved
via a process performed commonly in image-guided surgery known as “patient-to-image registration.”The process
can be performed at the commencement of the procedure
and if necessaryat intervals throughoutthe procedure (e.g.
after significant organ movement). The process results in
an adaption of the virtual scene (homogeneous transformation) to the coordinate systems of the patient. After
registration, the transformation can be applied throughout
the procedure to align virtual and real data within a single
coordinate system. Registering the patient to the image via
an external tracking system also allows the position of the
tracked laparoscope and laparoscopic instruments to be
registered and displayed within the virtual scene.
While alternative methods of patient-to-image registration exist [4], the majority of image guidance systems
utilize a form of registration that determines the best fit
transformation between the patient and the image data
by minimizing the error between the locations of three or
more corresponding features (Figure 4.5). Such features
can consist of any object visible in both the real and virtual
scenes. Often anatomical structures or landmarks such as
the falciform ligament or the hepatic vein are used, but
artificial objects positioned specifically for the registration
process, known as fiducials, can also be utilized. While the
latter are often more easily identified in image data, they
can be difficult to place, and they are thus less effective in
the registration of anatomy during HPB laparoscopic
interventions.
The positions of features are typically selected in the
virtual scene by the user via a touch screen or input device
(e.g. a computer mouse), and the corresponding point on
the patient is recorded by a position measurement system
while the surgeon points at the location with a tracked
laparoscopic instrument. From the corresponding point
pairs, the registration transformation can be calculated
using a least squares fitting algorithm.
While not indicative of the absolute accuracy, the use of
a matching algorithm allows a measure of the alignment
error to be expressed as the mean error between registered landmark sets (i.e. fiducial registration error – FRE).
This value can be used as an indication of the level of
accuracy of the registration and therefore indicates when
a registration should be repeated.
Figure 4.5 Registration of the image data to the patient is most commonly achieved via the matching of three or more corresponding
features in both the real and virtual scenes in a process termed registration.
52 Chapter 4
In orderto provide better registration accuracy of moveable structures, virtual data may be generated from intraoperative imaging modalities, such as ultrasound, which
then may be continuously registered to the real organ
using internal landmarks such as vessel structures. This
technique,described by Konishi et al. [5], is less commonly
employed because of the difficulty of generating virtual
registered data at sufficient quality and speed.
4.3.3 Laparoscope image calibration
Tracking the 3D position of the laparoscope facilitates
automatic alignment of the data throughout the procedure. While the position measurement system can track
the position and orientation (pose) of a marker attached
to the laparoscope, the relation between the tracked pose
and the view of the laparoscope camera must be determined via a calibration process. Calibration is achieved by
using camera calibration techniques which, using captured images of a known pattern at a known position in
space, determine the transformation between each laparoscopic image pixel and the real world [6]. Calibration is
performed intraoperatively after the application of tracking markers and by using semiautomatic techniques; it
can be performed in less than one minute (Figure 4.6).
Utilizing the tracked position of the laparoscope, the calibration of the laparoscope’s pose and view, and the patientto-image registration transformation, the preoperative 3D
virtual model can be automatically rendered at the same
point of view as the laparoscope and can subsequently be
overlaid on the laparoscopic image (according to the transformation chain depicted in Figure 4.6). This method can
adjust for movements of the laparoscope automatically at
a speed greater than 20 Hz and does not require additional
staff.
4.3.4 Instrument tracking
To maintain a correct alignment and correct representation of the surgical scene during the procedure, any
dynamic component within the visualized scenes, be it
anatomy, tools or the laparoscopic view itself, must be
tracked in space (Figure 4.7). If the position of the virtual
scene is not updated at the same speed as the real scene, a
delay error effect known as lag is experienced in the
augmented view. The real-time tracking of instruments
is often accomplished, as in traditional image-guided
surgery, via an optical or electromagnetic tracking system
that monitors the positions of specific markers (optical) or
sensor coils (electromagnetic)attachedtothesurgical
instruments. An in-depth description of tracking techniques
and technologies for image-guided procedures is outside the
scope of this chapter. Readers are instead directed to the
comprehensive overviews provided by Cleary and Peters
and Konishi et al. [1,5]. Alternatively, objects can be tracked
directly within image data by identifyingtheir positionusing
Figure 4.6 Calibration of a laparoscope. The relationship between the tracked marker and the laparoscopic images must be
performed before a properly aligned AR view can be generated. Calibration is achieved by taking multiple images of a known
pattern and determining the transformation from the image to the real world. Calibration devices such as this guide the calibration
process and when combined with automatic calibration algorithms, allow calibration to be performed intraoperatively, quickly,
and intuitively.
Augmented reality for laparoscopic liver surgery 53
Figure 4.7 To maintain a correct alignment and correct representation of the surgical scene during the procedure, any dynamic
components within the visualized scenes (e.g. anatomy, tools or the laparoscopic view itself) must be tracked in space.
image processing techniques. Although such techniques
reduce the amount of external equipment required, the
processing requirements increase significantly and tracking
accuracy may be compromised.
4.3.5 Image overlay
Once the virtual scene is captured at the same view as the
real patient, virtual data can be automatically merged
with the real view. While virtual data promise to provide
additional information to the operating surgeon, it is
imperative that augmentation interferes minimally
with the surgeon’s laparoscopic view. Virtual data must
possess sufficient contrast and clarity to be easily visible in
the augmented view while not masking instruments or
anatomy in the real patient view. Typically, virtual data
are displayed in strong primary colors, in order to
enhance contrast, while transparency is applied to ensure
that information under the overlay can be seen. Additionally, functionality via a user interface that allows
virtual models to be displayed or turned off ensures
that augmentation is used only when needed.
Because of the introduction of an additional view in
augmented reality, compared with the classic laparoscopic view, issues relating to visual depth perception
or interference of laparoscopic instruments with the augmented view must be considered. This remains an
unsolved challenge for developers of augmented reality
systems (Figure 4.8). The created augmented laparoscopic image is most naturally displayed on a laparoscope
monitor, although alternatives such as head-mounted
displays or projection directly onto the patient are possible. Direct projection onto the patient, while not requiring
3D models to be overlaid onto a 2D image, suffers from
the parallax error which, owing to the projection of 3D
virtual data onto a 2D viewing plane, causes the perception of the location of projected underlying anatomy to
change with the viewing angle (see Figure 4.8). This effect
is reduced for superficial anatomical structures and can be
eliminated by deconstructing 3D information into 2D
guidance information [7]. Anatomy or guidance can be
projected onto the patient in a geometrically correct
manner by employing a tracked projection device that
can be calibrated using a method similar to that described
for the calibration of a laparoscope [8].
4.4 Application of augmented
reality for laparoscopic
hepatopancreatobiliary
surgery
The need to target underlying anatomical structures or
lesions while preserving surrounding critical anatomical
structures with few external navigation cues presents
significant challenges in HPB surgery, particularly when
liver surgery is performed by novice HPB surgeons. Visualization of the 3D surgical scene within a 2D laparoscopic
image further increases the difficulty in determining target
locationand distance required forsafe resection margins or
anatomical structure preservation. These challenges
54 Chapter 4
Figure 4.8 (Left) Endoscopic image overlay AR with virtual anatomical structures superimposed onto the endoscopic image and
virtual tool model displayed within the virtual anatomy. (Right) Projection AR used with volume-rendered underlying anatomy
displayed directly on the patient.
renderHPB surgery an idealcandidate for AR visualization.
Specifically, the resection of tumors buried within liver or
pancreas parenchyma, located in tissue containing large
vessels, may be significantly aided by the AR visualization
of underlying anatomical structures, a surgical resection
plane or distance information. Additionally, numerous
HPB laparoscopic procedures require the mental representationof functional tissue regions such as liver segments. In
such cases, the AR visualization of preoperatively determined segments can aid in anatomical understanding and
reduce intraoperative cognitive workload.
Identified benefits of AR guidance for laparoscopic HPB
surgery have led a number of groups to develop and
clinically test AR systems. Although validation of the
systems has been performed with only a small number
of clinical cases, studies have proved the feasibility of the
approaches and highlighted potential clinical applications.
In this section, we review a number of AR systems that
have been developed specifically for use in laparoscopic
HPB surgery, demonstrating advantages and disadvantagesof differentAR approachesand exploring the breadth
of clinical application within HPB laparoscopic surgery.
4.4.1 Head-mounted AR displays for
laparoscopic surgery
In 2008, the concept of AR visualization for laparoscopic
surgery was presented by Fuchs et al., who described
direct visualization characteristics similar to those experienced during open surgery [9]. The development of a
head-mounted display system for the visualization of a
merged view of the real patient with synthetic images
aimed at improving, first, the intuitiveness of the laparoscopic visualization and second, depth perception using
cues superimposed on the laparoscopic image. Additionally, the head-mounted device widened the surgeon’s
range of view by displaying multiple laparoscopic imaging
datasets. The proposed system suffered from delay in
image generation and registration that was deemed
unacceptable for clinical use. Despite these drawbacks,
Fuchs et al. described the possible improvements in depth
perception and field of view that could be achieved with
AR for laparoscopic surgery. The group also proposed the
use of registered preoperative images, surgical planning
data, and intraoperative image data for the generation of a
more comprehensive visualization of the operating field.
4.4.2 Laparoscopic image overlay AR for
structure localization
A number of groups have attempted to overcome the
problems of depth perception and spatial understanding
experienced during laparoscopic HBP surgery by superimposing segmented anatomical models from imaging
data onto the laparoscopic image. This technique enables
underlying structures, normally hidden by overlying
Augmented reality for laparoscopic liver surgery 55
anatomy, to be visualized and localized. Using this
method, pioneering work in the development and clinical
use of an AR system for soft tissue laparoscopic procedures was conducted by Soler et al. [10]. By manually
aligning preoperatively obtained semitransparent 3D
models of tumors and vessels with intraoperative laparoscopic imaging, the authors reported more intuitive
understanding of patient anatomy.
Similar AR viewing modalities were investigated by
Konishi et al. [5]. They reported the use of a newly developed AR navigation system for laparoscopic surgery in
which 3D ultrasound reconstructed models acquired by
an operator and 3D models from preoperative CT were
overlaid onto the laparoscopic view. The system was developed primarily for thoracic surgery but its evaluation during
two liver procedures (radiofrequency thermal ablation for
hepatoma and a partial hepatectomy) was also reported. In
the two cases, the superimposition of ultrasound data
increased understanding of spatial relationships between
tumors and intrahepatic vessels, according to the authors.
The use of 3D reconstructed intraoperative ultrasound
imaging,unlike models based on preoperativeCT, allowed
deformation and motion of the target organ to be incorporated into the virtual data. Registering the laparoscopic
image to the intraoperative ultrasound data reduced
discrepancies due to motion but required an additional
electromagnetic tracking system in addition to the optical
tracking system used for position detection of the laparoscope. The same group later continued with preoperative
CT overlay, reporting six cases of pediatric laparoscopic
splenectomy, specifically aiming at aiding understanding
of anatomyin patients with anatomical anomalies. During
a case of accessory spleen buried in fat tissue, the overlay
helped with the localization of the splenic artery and vein
and the pancreatic tail (Figure 4.9).
We report the use of an image guidance system for
laparoscopic liver surgery during a portal vein ligation
with in situ liver split procedure. An interactive view of
the preoperative plan was displayed throughout the
entire procedure on a touch monitor that enabled anatomy to be assessed from varying views (Figure 4.10). On a
second monitor, an augmented view consisting of the
laparoscopic image overlaid with 3D models of structures
of interest, such as tumors, liver segments, and vascular
structures, was displayed (Figure 4.11). Superimposed
anatomical models suffered from a loss of depth perception relative to the laparoscopic image, and thus, to
determine the relative position of deep-lying anatomy
from the tool, a virtual model of the optically tracked
instrument was included in the augmented view. Virtual
Figure 4.9 CT-based AR image-guided navigation. (a) Detection of isolated accessory spleen under fat tissue. (b) Visualization of
splenic artery. (c) Visualization of splenic vein. (d) Visualization of pancreas. (e) Visualization of pancreas and vascular anatomy
hidden under the large spleen. (f) Identification of pancreatic tail to protect it from injury during the stapling of the splenic vessels.
56 Chapter 4
Figure 4.10 Virtual anatomy display alongside the augmented endoscopic display during a laparoscopic liver surgery.
anatomical models, segmented from preoperative CT
images, were registered to the liver by selecting corresponding anatomical landmarks on the liver surface with
the tracked tool. The laparoscope was calibrated prior to
application using the automatic algorithm described in
Fusaglia et al. [6]
This experience aimed to test the technological integration within a real-world surgical scenario and to evaluate a possible impact on the routine surgical workflow.
The surgeons who evaluated the access to additional
information noted benefits for anatomical structure
and tumor localization. Based on this experiment, future
Figure 4.11 Endoscopic augmented view displayed during liver surgery with 3D models of vascular structures and tumor (left) and
the virtual tool (right).
Augmented reality for laparoscopic liver surgery 57
applications of our AR guidance system will include
features that allow the display of distances from tracked
tools to vascular structures, of liver segments, and encoding of depth information.
4.4.3 Console-integrated laparoscopic
image overlay AR for
robotic surgery
Advantages of robotic systems for minimally invasive surgery include a 3D view of the surgical field, instrument
tremor reduction, better ergonomics, and the possibility of
remote-controlled surgery. However, identification of deep
structures and definition of spatial relationships are still a
challenge. In order to compensate for these deficits, Buchs
et al. reported the integration of the above described AR
laparoscopic system with the da Vinci surgical console, and
they employed this technology in two cases of hepatic lesion
resection [11]. Patient-to-image registration was achieved
by determining common anatomical landmarks between
the virtual organ and the real liver, using the optically
tracked robotic arm. Using two types of AR visualization
(overlay of 3D anatomical models and a targeting guidance
viewer), the operator was able to determine accurate surgical margins for tumor resections. The targeting guidance
viewer, representing the real-time depth and lateral
distance of a specific structure of interest (e.g. tumor) relative
to the tip of a tracked instrument, provided depth information in a clear and measurable manner (Figure 4.12).
Although 3D models provide the general location of tumors,
deconstructing position information into 2D guidance overcomes problems of depth perception, while enabling more
accurate targeting. The author reported adequate resection
margins and an increase in confidence during the resections.
4.4.4 Image overlay projection for
structure localization and
port placement
While AR for laparoscopic procedures finds its primary
advantages in laparoscopic image overlay, projection-based
overlay, in which virtual data are overlaid directly onto the
patient’s body, has also been investigated for use in laparoscopic procedures. For example, using a beamer attached
over the operating table, Sugimoto et al. projected 3D recon-
structed anatomy onto the patient’s body during three
cholecystectomies, two gastrectomies, and two colectomies [12] (Figure 4.13). By overlaying structures such as
virtual cholangiography and surrounding vessels, they
reported reduced operative times, reduced number of ports,
and anticipation of structures such as the gallbladder and
cystic ducts. A similar approach was reported by Volonté
et al. [13]. The projection of computer-generated images
allowed the adaptation of port placements (see Figure 4.6).
Furthermore, the group suggested that projection of anatomical structures would assist inexperienced surgeons during port placement. While the image was only visually
aligned to the patient without any formal registration and
Figure 4.12 A targeting viewer superimposed onto the endoscopic image and displayed within the da Vinci console provides
qualitative feedback regarding the relative position of the tool from a selected structure of interest such as the tumor shown here.
58 Chapter 4
Figure 4.13 Image overlay projection of the virtual cholangiography on the abdomen during laparoscopic cholecystectomy.
tracking, incorporating registration and tracking methodologies such as those described for open liver surgery by
Gavaghan et al. [8] would increase accuracy of the approach.
As perception of the projection of underlying structures
(especially deep-lying structures) is influenced by the angle
of the viewer, the overlay of anatomy, without correction of
viewing angle, can be used only as a relatively coarse guide.
Alternatively, 2D rather than 3D surface projections of
planned port positions could be projected onto the patient
surface, allowing accurate AR guidance independent of
viewer angle.
4.5 Challenges
Most reports on image guidance and AR for laparoscopic
surgery have been positive, but AR technology and its
application remain in their infancy. Initial clinical studies
have highlighted possible usefulness, but they have also
identified a number of challenges facing developers and
users of AR technology. The two most prominent challenges, (i) organ movement and deformation, and
(ii) effective visualization methods, are discussed below.
4.5.1 Organ motion and deformation
Obtaining an optimal alignment of the real and virtual
scenes through registration, tracking, and image
processing techniques remains one of the greatest challenges in the realization of AR in surgery. Any misalignment is considerably more evident and distracting
in a merged scene than when additional data are displayed on a separate monitor. From a technological point
of view, tracking of organ motion and deformation
remains an unsolved challenge. Methods of nonrigid
registration, typically based on the reconstruction of a
3D liver model from images of the organ surface, have
been proposed, but technological limitations, such as
computational power, result in inapplicability of these
techniques in a real surgical scenario. Additionally, the
relationship between surface deformation and the misplacement of internal structures is unknown.
Current image guidance systems for open liver surgery
employ rigid registration techniques that function on the
premise that registering a small local area of interest
rigidly to preoperative data is sufficiently accurate. Studies support this hypothesis [14,15] and current research
on registration of the liver is focusing on the possibility of
increasing accuracy through the use of ultrasound imaging as the primary tool for registration. The use of tracked
ultrasound allows for the creation of 3D volumes of the
underlying vessels, which can then be used to align an
internal volume of the liver with the surface models
segmented from preoperative CT (Figure 4.14). The
main advantage of this technique lies in the fact that
Соседние файлы в папке Библиотека им академика М.И. Перельмана