- •Contents
- •Figures
- •Tables
- •Preface
- •Acknowledgments
- •1. Raster images
- •Aspect ratio
- •Geometry
- •Image capture
- •Digitization
- •Perceptual uniformity
- •Colour
- •Luma and colour difference components
- •Digital image representation
- •Square sampling
- •Comparison of aspect ratios
- •Aspect ratio
- •Frame rates
- •Image state
- •EOCF standards
- •Entertainment programming
- •Acquisition
- •Consumer origination
- •Consumer electronics (CE) display
- •Contrast
- •Contrast ratio
- •Perceptual uniformity
- •The “code 100” problem and nonlinear image coding
- •Linear and nonlinear
- •4. Quantization
- •Linearity
- •Decibels
- •Noise, signal, sensitivity
- •Quantization error
- •Full-swing
- •Studio-swing (footroom and headroom)
- •Interface offset
- •Processing coding
- •Two’s complement wrap-around
- •Perceptual attributes
- •History of display signal processing
- •Digital driving levels
- •Relationship between signal and lightness
- •Algorithm
- •Black level setting
- •Effect of contrast and brightness on contrast and brightness
- •An alternate interpretation
- •Brightness and contrast controls in LCDs
- •Brightness and contrast controls in PDPs
- •Brightness and contrast controls in desktop graphics
- •Symbolic image description
- •Raster images
- •Conversion among types
- •Image files
- •“Resolution” in computer graphics
- •7. Image structure
- •Image reconstruction
- •Sampling aperture
- •Spot profile
- •Box distribution
- •Gaussian distribution
- •8. Raster scanning
- •Flicker, refresh rate, and frame rate
- •Introduction to scanning
- •Scanning parameters
- •Interlaced format
- •Interlace and progressive
- •Scanning notation
- •Motion portrayal
- •Segmented-frame (24PsF)
- •Video system taxonomy
- •Conversion among systems
- •9. Resolution
- •Magnitude frequency response and bandwidth
- •Visual acuity
- •Viewing distance and angle
- •Kell effect
- •Resolution
- •Resolution in video
- •Viewing distance
- •Interlace revisited
- •10. Constant luminance
- •The principle of constant luminance
- •Compensating for the CRT
- •Departure from constant luminance
- •Luma
- •“Leakage” of luminance into chroma
- •11. Picture rendering
- •Surround effect
- •Tone scale alteration
- •Incorporation of rendering
- •Rendering in desktop computing
- •Luma
- •Sloppy use of the term luminance
- •Colour difference coding (chroma)
- •Chroma subsampling
- •Chroma subsampling notation
- •Chroma subsampling filters
- •Chroma in composite NTSC and PAL
- •Scanning standards
- •Widescreen (16:9) SD
- •Square and nonsquare sampling
- •Resampling
- •NTSC and PAL encoding
- •NTSC and PAL decoding
- •S-video interface
- •Frequency interleaving
- •Composite analog SD
- •15. Introduction to HD
- •HD scanning
- •Colour coding for BT.709 HD
- •Data compression
- •Image compression
- •Lossy compression
- •JPEG
- •Motion-JPEG
- •JPEG 2000
- •Mezzanine compression
- •MPEG
- •Picture coding types (I, P, B)
- •Reordering
- •MPEG-1
- •MPEG-2
- •Other MPEGs
- •MPEG IMX
- •MPEG-4
- •AVC-Intra
- •WM9, WM10, VC-1 codecs
- •Compression for CE acquisition
- •AVCHD
- •Compression for IP transport to consumers
- •VP8 (“WebM”) codec
- •Dirac (basic)
- •17. Streams and files
- •Historical overview
- •Physical layer
- •Stream interfaces
- •IEEE 1394 (FireWire, i.LINK)
- •HTTP live streaming (HLS)
- •18. Metadata
- •Metadata Example 1: CD-DA
- •Metadata Example 2: .yuv files
- •Metadata Example 3: RFF
- •Metadata Example 4: JPEG/JFIF
- •Metadata Example 5: Sequence display extension
- •Conclusions
- •19. Stereoscopic (“3-D”) video
- •Acquisition
- •S3D display
- •Anaglyph
- •Temporal multiplexing
- •Polarization
- •Wavelength multiplexing (Infitec/Dolby)
- •Autostereoscopic displays
- •Parallax barrier display
- •Lenticular display
- •Recording and compression
- •Consumer interface and display
- •Ghosting
- •Vergence and accommodation
- •20. Filtering and sampling
- •Sampling theorem
- •Sampling at exactly 0.5fS
- •Magnitude frequency response
- •Magnitude frequency response of a boxcar
- •The sinc weighting function
- •Frequency response of point sampling
- •Fourier transform pairs
- •Analog filters
- •Digital filters
- •Impulse response
- •Finite impulse response (FIR) filters
- •Physical realizability of a filter
- •Phase response (group delay)
- •Infinite impulse response (IIR) filters
- •Lowpass filter
- •Digital filter design
- •Reconstruction
- •Reconstruction close to 0.5fS
- •“(sin x)/x” correction
- •Further reading
- •2:1 downsampling
- •Oversampling
- •Interpolation
- •Lagrange interpolation
- •Lagrange interpolation as filtering
- •Polyphase interpolators
- •Polyphase taps and phases
- •Implementing polyphase interpolators
- •Decimation
- •Lowpass filtering in decimation
- •Spatial frequency domain
- •Comb filtering
- •Spatial filtering
- •Image presampling filters
- •Image reconstruction filters
- •Spatial (2-D) oversampling
- •Retina
- •Adaptation
- •Contrast sensitivity
- •Contrast sensitivity function (CSF)
- •24. Luminance and lightness
- •Radiance, intensity
- •Luminance
- •Relative luminance
- •Luminance from red, green, and blue
- •Lightness (CIE L*)
- •Fundamentals of vision
- •Definitions
- •Spectral power distribution (SPD) and tristimulus
- •Spectral constraints
- •CIE XYZ tristimulus
- •CIE [x, y] chromaticity
- •Blackbody radiation
- •Colour temperature
- •White
- •Chromatic adaptation
- •Perceptually uniform colour spaces
- •CIE L*a*b* (CIELAB)
- •CIE L*u*v* and CIE L*a*b* summary
- •Colour specification and colour image coding
- •Further reading
- •Additive reproduction (RGB)
- •Characterization of RGB primaries
- •BT.709 primaries
- •Leggacy SD primaries
- •sRGB system
- •SMPTE Free Scale (FS) primaries
- •AMPAS ACES primaries
- •SMPTE/DCI P3 primaries
- •CMFs and SPDs
- •Normalization and scaling
- •Luminance coefficients
- •Transformations between RGB and CIE XYZ
- •Noise due to matrixing
- •Transforms among RGB systems
- •Camera white reference
- •Display white reference
- •Gamut
- •Wide-gamut reproduction
- •Free Scale Gamut, Free Scale Log (FS-Gamut, FS-Log)
- •Further reading
- •27. Gamma
- •Gamma in CRT physics
- •The amazing coincidence!
- •Gamma in video
- •Opto-electronic conversion functions (OECFs)
- •BT.709 OECF
- •SMPTE 240M OECF
- •sRGB transfer function
- •Transfer functions in SD
- •Bit depth requirements
- •Gamma in modern display devices
- •Estimating gamma
- •Gamma in video, CGI, and Macintosh
- •Gamma in computer graphics
- •Gamma in pseudocolour
- •Limitations of 8-bit linear coding
- •Linear and nonlinear coding in CGI
- •Colour acuity
- •RGB and R’G’B’ colour cubes
- •Conventional luma/colour difference coding
- •Luminance and luma notation
- •Nonlinear red, green, blue (R’G’B’)
- •BT.601 luma
- •BT.709 luma
- •Chroma subsampling, revisited
- •Luma/colour difference summary
- •SD and HD luma chaos
- •Luma/colour difference component sets
- •B’-Y’, R’-Y’ components for SD
- •PBPR components for SD
- •CBCR components for SD
- •Y’CBCR from studio RGB
- •Y’CBCR from computer RGB
- •“Full-swing” Y’CBCR
- •Y’UV, Y’IQ confusion
- •B’-Y’, R’-Y’ components for BT.709 HD
- •PBPR components for BT.709 HD
- •CBCR components for BT.709 HD
- •CBCR components for xvYCC
- •Y’CBCR from studio RGB
- •Y’CBCR from computer RGB
- •Conversions between HD and SD
- •Colour coding standards
- •31. Video signal processing
- •Edge treatment
- •Transition samples
- •Picture lines
- •Choice of SAL and SPW parameters
- •Video levels
- •Setup (pedestal)
- •BT.601 to computing
- •Enhancement
- •Median filtering
- •Coring
- •Chroma transition improvement (CTI)
- •Mixing and keying
- •Field rate
- •Line rate
- •Sound subcarrier
- •Addition of composite colour
- •NTSC colour subcarrier
- •576i PAL colour subcarrier
- •4fSC sampling
- •Common sampling rate
- •Numerology of HD scanning
- •Audio rates
- •33. Timecode
- •Introduction
- •Dropframe timecode
- •Editing
- •Linear timecode (LTC)
- •Vertical interval timecode (VITC)
- •Timecode structure
- •Further reading
- •34. 2-3 pulldown
- •2-3-3-2 pulldown
- •Conversion of film to different frame rates
- •Native 24 Hz coding
- •Conversion to other rates
- •Spatial domain
- •Vertical-temporal domain
- •Motion adaptivity
- •Further reading
- •36. Colourbars
- •SD colourbars
- •SD colourbar notation
- •Pluge element
- •Composite decoder adjustment using colourbars
- •-I, +Q, and Pluge elements in SD colourbars
- •HD colourbars
- •References
- •38. SDI and HD-SDI interfaces
- •Component digital SD interface (BT.601)
- •Serial digital interface (SDI)
- •Component digital HD-SDI
- •SDI and HD-SDI sync, TRS, and ancillary data
- •Analog sync and digital/analog timing relationships
- •Ancillary data
- •SDI coding
- •HD-SDI coding
- •Interfaces for compressed video
- •SDTI
- •Switching and mixing
- •Timing in digital facilities
- •Summary of digital interfaces
- •39. 480i component video
- •Frame rate
- •Interlace
- •Line sync
- •Field/frame sync
- •R’G’B’ EOCF and primaries
- •Luma (Y’)
- •Picture center, aspect ratio, and blanking
- •Halfline blanking
- •Component digital 4:2:2 interface
- •Component analog R’G’B’ interface
- •Component analog Y’PBPR interface, EBU N10
- •Component analog Y’PBPR interface, industry standard
- •40. 576i component video
- •Frame rate
- •Interlace
- •Line sync
- •Analog field/frame sync
- •R’G’B’ EOCF and primaries
- •Luma (Y’)
- •Picture center, aspect ratio, and blanking
- •Component digital 4:2:2 interface
- •Component analog 576i interface
- •Scanning
- •Analog sync
- •Picture center, aspect ratio, and blanking
- •R’G’B’ EOCF and primaries
- •Luma (Y’)
- •Component digital 4:2:2 interface
- •Scanning
- •Analog sync
- •Picture center, aspect ratio, and blanking
- •R’G’B’ EOCF and primaries
- •Luma (Y’)
- •Component digital 4:2:2 interface
- •43. HD videotape
- •HDCAM (D-11)
- •DVCPRO HD (D-12)
- •HDCAM SR (D-16)
- •JPEG blocks and MCUs
- •JPEG block diagram
- •Level shifting
- •Discrete cosine transform (DCT)
- •JPEG encoding example
- •JPEG decoding
- •Compression ratio control
- •JPEG/JFIF
- •Motion-JPEG (M-JPEG)
- •Further reading
- •46. DV compression
- •DV chroma subsampling
- •DV frame/field modes
- •Picture-in-shuttle in DV
- •DV overflow scheme
- •DV quantization
- •DV digital interface (DIF)
- •Consumer DV recording
- •Professional DV variants
- •47. MPEG-2 video compression
- •MPEG-2 profiles and levels
- •Picture structure
- •Frame rate and 2-3 pulldown in MPEG
- •Luma and chroma sampling structures
- •Macroblocks
- •Picture coding types – I, P, B
- •Prediction
- •Motion vectors (MVs)
- •Coding of a block
- •Frame and field DCT types
- •Zigzag and VLE
- •Refresh
- •Motion estimation
- •Rate control and buffer management
- •Bitstream syntax
- •Transport
- •Further reading
- •48. H.264 video compression
- •Algorithmic features, profiles, and levels
- •Baseline and extended profiles
- •High profiles
- •Hierarchy
- •Multiple reference pictures
- •Slices
- •Spatial intra prediction
- •Flexible motion compensation
- •Quarter-pel motion-compensated interpolation
- •Weighting and offsetting of MC prediction
- •16-bit integer transform
- •Quantizer
- •Variable-length coding
- •Context adaptivity
- •CABAC
- •Deblocking filter
- •Buffer control
- •Scalable video coding (SVC)
- •Multiview video coding (MVC)
- •AVC-Intra
- •Further reading
- •49. VP8 compression
- •Algorithmic features
- •Further reading
- •Elementary stream (ES)
- •Packetized elementary stream (PES)
- •MPEG-2 program stream
- •MPEG-2 transport stream
- •System clock
- •Further reading
- •Japan
- •United States
- •ATSC modulation
- •Europe
- •Further reading
- •Appendices
- •Cement vs. concrete
- •True CIE luminance
- •The misinterpretation of luminance
- •The enshrining of luma
- •Colour difference scale factors
- •Conclusion: A plea
- •Radiometry
- •Photometry
- •Light level examples
- •Image science
- •Units
- •Further reading
- •Glossary
- •Index
- •About the author
Boynton, Robert M. (1979),
Human Color Vision (New York:
Holt, Rinehart and Winston).
Wandell, Brian A. (1995),
Foundations of Vision (Sunderland, Mass.: Sinauer Associates).
Perception and
visual acuity |
23 |
Properties of human vision are central to image system engineering. They determine how many pixels need to be provided per degree of picture angle, and how many bits are necessary to represent luminance (or tristimulus) levels. This chapter introduces the luminance discrimination and spatial properties of vision that inform image system engineering choices.
Retina
The human retina has four types of photoreceptor cells that respond to incident radiation with different spectral response curves. A retina has about 100 million rod cells, effective only at extremely low light levels; and about 5 million cone cells, of three types, that mediate colour vision. Since there is only one type of rod cell, what is loosely called night vision cannot discern colours.
The cone cells are sensitive to longwave, mediumwave, and shortwave light – roughly, light in the red, green, and blue portions of the spectrum. Because there are just three types of colour photoreceptors, three numerical components are necessary and sufficient to describe colour: Colour vision is inherently trichromatic. To arrange for three components to mimic colour vision, suitable spectral sensitivity functions must be used; this topic will be discussed in The CIE system of colorimetry, on page 265.
Adaptation
Vision operates over a remarkable range of luminance levels – about eight orders of magnitude (decades),
247
Luminance of diffuse white |
sketched in Figure 23.1. For about four decades at the |
|||||||
low end of the range, the rods are active; vision at these |
||||||||
reflector in scene [cd·m-2] |
||||||||
SUNLIGHTTWILIGHT |
|
|
|
types)(3cellsCone visionPhotopic |
light levels is called scotopic. For the top five or six |
|||
30 k |
|
|
decades, the cones are active; vision at these light levels |
|||||
|
|
|
|
|
|
|||
|
3 k |
|
|
|
|
|
is called photopic. |
|
|
|
|
|
|
|
Mesopic vision takes place in the range of luminance |
||
|
|
|
|
|
|
|||
|
300 |
|
|
|
|
|
levels where there is some overlap between rods and |
|
|
|
|
|
|
|
cones. Considered from the bottom of the photopic |
||
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
||
|
30 |
|
|
|
|
|
region, this is called rod intrusion. It is a research topic |
|
|
|
|
|
|
|
whether the rods have significance to colour image |
||
|
|
|
|
|
|
|
||
STARLIGHTMOONLIGHT |
3 |
|
|
|
|
cellsRod(1 type) Scotopicvision |
reproduction at usual luminance levels (such as in the |
|
|
|
|
|
|||||
300 |
|
|
|
|
200,000; adaptation causes the sensitivity of the visual |
|||
|
|
|
|
|
|
|
cinema). For today’s engineering purposes, the effect of |
|
|
300 m |
|
|
|
|
|
rod intrusion is discounted. |
|
|
30 m |
|
|
|
|
|
During the course of the day we experience a wide |
|
|
|
|
|
|
|
range of illumination levels; adaptation adjusts accord- |
||
|
3 m |
|
|
|
|
|
ingly, as sketched in Figure 23.2. From moonlight to |
|
|
|
|
|
|
|
sunlight, illuminance changes by a factor of about |
||
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
system to reduce by about a factor of 1000. About one |
|
Figure 23.1 Luminance |
||||||||
decade of adaptation is effected by the eye’s iris – that |
||||||||
range of vision |
|
|
|
|||||
|
|
|
is, by changes in pupil diameter (from about 2 mm to |
|||||
|
|
|
|
|
|
|
||
|
30 k |
|
|
|
8 mm). The main mechanism of adaptation is |
|||
|
|
|
|
a photochemical process involving the visual pigment |
||||
|
|
|
|
|
|
|
||
|
3 k |
|
|
|
substance contained in the rods and the cones; it also |
|||
|
|
|
|
|
|
|
involves neural mechanisms in the visual pathway. |
|
|
300 |
|
|
|
|
100 |
Dark adaptation, to low luminance, is slow: Adapta- |
|
|
30 |
|
|
|
|
tion from a bright sunlit day to the darkness of a cinema |
||
|
|
|
|
|||||
|
|
|
|
|
|
|||
|
|
|
|
|
|
can take a few minutes. Adaptation to higher lumi- |
||
|
|
|
|
|
|
10 |
||
|
|
|
|
|
|
nance is rapid but can be discomforting, as you may |
||
|
3 |
|
|
|
|
|
||
|
|
|
|
|
|
have experienced when walking out of the cinema back |
||
|
|
|
|
|
|
1 |
||
|
|
|
|
|
|
into daylight. |
||
|
300 m |
|
|
|
||||
|
|
|
0.1 |
Adaptation is a low-level phenomenon within the |
||||
|
|
|
|
|
|
|||
|
|
|
|
|
||||
|
30 m |
|
|
|
visual system; it is mainly controlled by total retinal illu- |
|||
|
|
|
|
|
|
|
mination. Your adaptation state is closely related to the |
|
|
3 m |
|
|
|
mean luminance in your field of view. In a dark viewing |
|||
|
|
|
|
|
|
|
environment, such as a cinema, the image itself controls |
|
300 |
adaptation. |
|
|
||
Figure 23.2 Adaptation |
At a particular state of adaptation, vision can discern |
|
different luminances across about a 1000:1 range. |
||
|
||
|
When viewing a real scene, adaptation changes |
|
|
depending upon where in the scene your gaze is |
|
|
directed. In video and film, we are nearly always |
|
|
concerned with viewing at a known adaptation state, so |
|
|
a simultaneous contrast ratio of 1000:1 is adequate. |
248 |
DIGITAL VIDEO AND HD ALGORITHMS AND INTERFACES |
Diffuse white was described on page 117. This wide range of luminance levels is sometimes called dynamic range, but nothing is in motion!
Figure 23.3 A contrast sensitivity test pattern is presented to an observer in an experiment to determine the contrast sensitivity of human vision. The observer is adapted to background having luminance LB;
a bipartite patch is viewed. The experimenter adjusts ∆L; the observer reports whether he or she detects a difference in lightness between the two patches.
For image reproduction purposes, our ability to distinguish luminance differences ordinarily extends over a ratio of luminance of about three decades – 103, or 1000:1 – that is, down to about 0.1% of diffuse white as portrayed on the display. Loosely speaking, luminance levels less than 0.1% of diffuse white appear just “black”: Different luminances below that level are not ordinarily visually useful. Emergent high dynamic range (HDR) systems may increase that ratio.
Contrast sensitivity
Within the two-decade range of luminance that is useful for image reproduction, vision has a certain threshold of discrimination. It is convenient to express the discrimination capability in terms of contrast threshold, which is the ratio of a small test increment in luminance to the base luminance in a test stimulus having two adjacent patches of similar luminance.
Figure 23.3 below shows the pattern presented to an observer in an experiment to determine the contrast sensitivity of human vision. Most of the observer’s field of vision is filled by a background luminance level, LB, which fixes the observer’s state of adaptation. In the central area of the field of vision are placed two adjacent patches having slightly different luminance levels, L and L+∆L. The experimenter presents stimuli having a wide range of test values with respect to the surround, that is, a wide range of L/LB values. At each test luminance, the experimenter presents to the observer a range of luminance increments with respect to the test stimulus, that is, a range of ∆L/L values.
LB: Background luminance
L: Test luminance
∆L: Test luminance increment
LL+∆L
|
L: |
Test luminance |
|
∆L: |
Luminance test increment |
LB |
LB: |
Background luminance |
CHAPTER 23 |
PERCEPTION AND VISUAL ACUITY |
249 |
-0.8
L
∆L log
-1.0 

-1.2
-1.4
-1.6
-1.8
0.1 |
1 |
10 |
100 |
1000 |
104 |
Luminance [cd•m-2]
Figure 23.4 Contrast sensitivity. This graph is redrawn from Figure 3.4 of Schreiber’s Fundamentals of Electronic Imaging Systems. Over a range of luminance values of about 300:1, the discrimination threshold of vision is approximately a constant ratio of luminance. The flat portion of the curve shows that the perceptual response to luminance – termed lightness – is approximately logarithmic. At very low luminance values, the curve departs from logarithmic behaviour and approximates a square-root; this characteristic is called the de Vries-Rose law.
Schreiber, William F. (1993),
Fundamentals of Electronic
Imaging Systems, Third Edition
(Berlin: Springer-Verlag).
log 100 ≈ 463; 1.01463 ≈ 100 log 1.01
log 30 = 172 log 1.02
Fink, Donald G., ed. (1955),
Color Television Standards (New York: McGraw-Hill): 201.
When this experiment is conducted, the relationship graphed in Figure 23.4 above is found: Plotting log ∆L/L as a function of log L reveals an interval of a few decades of luminance over which the discrimination capability of vision is about 1% of the test luminance level. This experiment leads to the conclusion that – for threshold discrimination of two adjacent patches of nearly identical luminance – the discrimination capability is roughly logarithmic.
The contrast sensitivity function begins to answer this question: What is the minimum number of discrete codes required to represent relative luminance over
a particular range? In other words, what luminance codes can be thrown away without the observer noticing? On a linear luminance scale, to cover a 100:1
range with an increment of 0.01 takes 100/0.01, or about 10,000 codes, requiring about 14 bits. If codes
are spaced according to a ratio of 1.01, then only about 463 codes are required; codes can be represented in just 9 bits.(NTSC documents from the early 1950s used a contrast sensitivity of 2% and a contrast ratio of 30:1 to derive 172 steps; even today, 8 bits suffice for video distribution.)
The logarithmic relationship relates to contrast sensitivity at threshold: We are measuring the ability of the visual system to discriminate between two nearly identical luminances. If you like, call this a just noticeable
250 |
DIGITAL VIDEO AND HD ALGORITHMS AND INTERFACES |
