322 |
M.P. Barry and G. Dagnelie |
Fig. 16.2 The effect of eye movements on stimulation of the visual system in natural vision (left panel) and in prosthetic vision with an external camera, both without (center panel) and with (right panel) compensation through gaze tracking
implants, only the multi-photodiode array (MPDA) of Retina Implant AG provides this capability. For all devices with an external camera the situation can be remedied by tracking the prosthesis wearer’s eye position and presenting a corresponding shift of the image to the implant. This would be done most easily by using a wideangle camera and instantly panning the section to be presented to the prosthesis wearer in accordance with the current direction of gaze. Such accurate and instantaneous gaze tracking is not currently used, however.
Accurate prosthetic vision simulations should therefore have the ability to mimic gaze stabilization. In the diagram of Fig. 16.1 this is implemented through a pupil-tracking video camera built into the HMD, eye-tracking software (Arrington Research, Scottsdale, AZ), and a resulting offset of the filtered imagery according to the updated gaze position; typically this is done at 30 or 60 frames per second, but more rapid systems are now available.
16.2.2 Filter Engine Parameters
In order to present imagery that closely resembles what a prosthesis wearer is expected to perceive, the filtering engine needs to transform the incoming video frames according to a number of important aspects. Roughly, these can be categorized into four groups: raster spatial properties, dot spatial and temporal properties, and dynamic background noise.
16.2.2.1 Raster Spatial Properties
Typically, the experimenter will have a specific implant configuration in mind and will sub-sample the incoming image to match that configuration. For a retinal
16 Simulations of Prosthetic Vision |
323 |
implant, the electrode arrangement will most likely be rectangular and regular, although hexagonal and/or radially expanding configurations could in principle be used, in order to conform more closely to the native properties of retinal processing. In all cases the incoming image is reduced in resolution by grouping the intensity and color values within the aperture of each prospective dot position. As an example, a typical 320 × 240 pixel camera image can be down-sampled to simulate a 10 × 6 implant by dividing it into 60 rectangular subfields of 32 × 40 pixels each, and averaging the pixel values within each rectangle to yield a single value that will be represented by the simulated phosphene. Color information will typically be discarded, since only grey scale values are thought to be meaningfully conveyed.
There are several instances where a regular grid of simulated phosphenes is not an adequate representation of what the implant recipient is expected to see. Most importantly, this is the case for implants beyond the retina. Stimulation of the optic nerve, LGN, or primary visual cortex should still provide a predictable phosphene array, depending on the accuracy of electrode placement, and these irregularities can be built into the simulated phosphene map.
Even for a retinal implant there may be distortions of the regular grid. In the normal retinal anatomy the centermost fovea does not contain any secondary neurons, so many neurons at 1°–2° eccentricity in the retina will correspond to locations much closer to fixation in the visual field, and stimulating those neurons will cause an apparent contraction of the image: phosphenes will be denser immediately around the point of fixation, and correspondingly sparser in a ring at 2°–4° eccentricity. In addition, the retinal rewiring process described in Chap. 3 will cause inner retinal neurons to migrate from their original positions, and may thus convey random scatter to the perceived phosphene positions. The magnitude of both effects can be estimated, but to our knowledge have not been taken into account in simulations of a retinal prosthesis. On the other hand the crude resolution of most current prostheses, with electrode separations of approximately 2°, reduces the need for such refinements.
In addition to the overall arrangement of dots in the raster, several parameters can specify raster properties:
•Dot number: This quantity corresponds to the number of electrodes in the implant.
•Dot density: This quantity determines the center-to-center distance between dots, and is typically chosen to correspond to the inter-electrode distance of the implant. For rectangular grids it is common for density to be equal in the two perpendicular directions. Note that density is the inverse of center-to-center distance.
•Dot spacing: When viewing the dot grid one can envisage each dot as being situate at the center of a “unit cell,” and the dot may or may not fill the entire cell. For round dots in a rectangular (rather than square) grating, dot spacing will be different in the two orthogonal directions. The space between dots and the background intensity light filling that space will be further discussed under
Sect. 16.2.2.2.
324 |
M.P. Barry and G. Dagnelie |
•Grid size: This quantity has a direct relationship to the previous two; it is common for one of the three parameters to be kept constant, and study the tradeoff between the remaining two.
•Dot drop-out: A subset of electrodes in an implant may prove non-functional after implantation or lose functionality over time; this loss of function can be caused by either the implant itself or by degeneration of the tissue substrate; to model this, a subset of dots may be omitted; typically this subset is chosen at random, and not altered while testing a given subject over multiple sessions, to investigate whether adaptation may occur to this localized absence of image information.
Effects of several grid parameter changes are shown in Fig. 16.3.
16.2.2.2 Dot Spatial Properties
Phosphenes elicited by localized electrical stimulation in blind individuals have generally been described as small round dots, varying in size from a pea to a quarter at arms length, and either sharp or fuzzy in appearance; some subjects have described rings or dark dots on a lighter background, depending on the stimulus conditions. This illustrates a basic problem when rendering images in even the simplest prosthetic vision simulation. The square pixelization commonly employed to hide a person’s identity in the media (see Fig. 16.4, left panel) lend themselves to rapid image rendering and have been used extensively by one research group [16, 17, 24, 26–28], but may not be an optimal representation of what is described by patients undergoing stimulation. Other groups have spent considerable effort on
Fig. 16.3 Illustration of the effects of grid and dot parameters on the display of a text fragment with pillbox-shaped dots. All changes are relative to the “standard condition” in the center of the figure. In the top right panel grid size is changed without increasing dot size or number, whereas in the bottom left panel the dot number is changed, and in the top left panel dot size is changed while keeping the gaps separating the dots equal
16 Simulations of Prosthetic Vision |
325 |
Fig. 16.4 Examples of pixelization used in prosthetic vision simulations. In both examples a rectangular raster was used. The left panel shows a 14 × 14 cell grid with square pixelization of a face (courtesy Dr. Wentai Liu), while the right panel shows a 4 × 4 grid with Gaussian dot profile as seen by a subject in our laboratory inspecting the scoop of a spoon
the creation of model phosphenes with precisely specified spatial properties. Generally, the following parameters are specified:
•Shape: Although most phosphenes seen by patients are not perfectly round, the most common shapes used in simulations have been bright circles on a dark background, as shown by the examples in Figs. 16.3 and 16.4 (right panel).
•Profile/size: There is a variety of ways in which the light representing the intensity in the scene can be distributed across the unit cell. The most common profiles chosen are rectangular and Gaussian; the extent to which the light in one cell merges with that in neighboring cells depends on the radius of the pillbox (0.495
in the example in Fig. 16.5; hence there is no overlap in the right half of the figure) or the value of s (four values shown). If a Gaussian profile is chosen there is always some overlap, making the use of Gaussians much more computationally intensive in a real-time simulation. The increased speed of general purpose processors and the use of dedicated hardware have led to more frequent use of Gaussian profiles in recent simulations, since they correspond more closely to the reports of prosthesis wearers [22].
•Intensity/contrast:Most simulations use bright dots and modulate the peak intensity of the dots to represent local brightness in the scene, on a black background. Yet it is unlikely that prosthesis wearers will experience such high contrast percepts: Patients blind from outer retinal degenerations describe their world as grey rather than black. For this reason some simulation studies have explored the dependence of subject performance on contrast. In some cases this was done by only changing dot brightness but leaving the black background; this reduces brightness rather than contrast and has very little effect as the subject adapts to the lower light level. Increasing the background intensity, with or without a reduction in dot intensity, is an appropriate way to reduce contrast.
Some studies (e.g., Chap. 17) have modulated the radius of pillbox dots rather than their intensity, but a systematic comparison of the two methods across