Figure 1.9 Comparison of aspect ratios between conventional television (now SD) and HD was attempted using various measures: equal height, equal width, equal diagonal, and equal area. All of these comparisons overlooked the fundamental improvement of HD: its increased pixel count. The correct comparison is based upon equal picture detail. It is the angular subtense of a pixel that should be preserved.
4:3 Aspect ratio |
16:9 Aspect ratio |
45.33
3 |
3 |
Equal |
|
Height |
|||
|
|
44
3 |
2.25 |
Equal |
|
Width |
|||
|
|
|
4 |
|
4.36 |
|
|
|
|
3 |
|
2.45 |
Equal |
|
Diagonal |
||
|
|
|
44.62
3 |
|
|
|
|
2.60 |
|
|
|
|
|
Equal |
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
|||
|
|
|
|
|
|
|
|
|
Area |
||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
12
4
3
6.75
Equal
Detail!
equal width, equal diagonal, and equal area. All of those measures overlooked the fundamental improvement of HD: Its “high definition” (or “resolution”) does not squeeze six times the number of pixels into the same visual angle! Instead, the angular subtense of
a single pixel should be maintained, and the entire image can now occupy a much larger area of the viewer’s visual field. HD allows a greatly increased picture angle. The correct comparison between conventional television and HD is not based upon picture aspect ratio; it is based upon picture detail.
Aspect ratio
With the advent of HD consumer television receivers, it became necessary to display 4:3 (SD) material on 16:9 (HD) displays and 16:9 material on 4:3 displays. During the standardization of HD, I proposed – not entirely facetiously – that SD content at 4:3 should be “pixelmapped” into the HD frame as sketched in Figure 1.10, preserving aspect ratio and equal detail. I anticipated
CHAPTER 1 |
RASTER IMAGES |
15 |
Figure 1.10 SD to HD pixel mapping is one way to convert 4:3 material to 16:9. The angular subtense of SD pixels is preserved. If CE vendors had adopted this approach at the introduction of HD, today’s aspect ratio chaos would have
been avoided.
4:3 SD
16:9 HD
Figure 1.11 Aspect ratio changes can compromise creative intent. Consider this frame at 1.78:1 aspect ratio. The two figures survey the water prior to embarking on an adventure.
Figure 1.12 When centre-cut to 4:3 aspect ratio, one character is deleted; the story has changed. Much drama and much comedy depends upon action at the edges of the frame.
that provisions would be made for the consumer to enlarge the SD image – but the consumer would have been aware of two qualitatively different image sources. (My idea wasn’t adopted!)
Widescreen 16:9 material can be adapted to 4:3 by cropping the image width; however, picture content is lost, and creative intent is liable to be compromised. Figures Figure 1.11 and 1.12 below show the result of centre-cropping 16:9 material. The plot might suffer!
Pan-and-scan, sketched in Figure 1.13 at the top of the facing page, refers to choosing on a scene-by-scene basis the 4:3 region to be maintained, to mitigate the creative loss that might otherwise result from cropping.
16 |
DIGITAL VIDEO AND HD ALGORITHMS AND INTERFACES |
4:3 |
16:9 |
Figure 1.13 Pan-and-scan crops the width of widescreen material – here, 16:9 – for
a 4:3 aspect ratio display.
16:9
4:3
Figure 1.14 Letterbox format fits widescreen material – here, 16:9 – to the width of a 4:3 display.
4:3
16:9
Figure 1.15 Pillarbox format
(sometimes called sidebar) fits narrow-aspect-ratio material to the height of a 16:9 display.
Some consumer HD receivers have nonlinear stretching where the horizontal expansion ratio is a function of position. The intended image geometry is distorted; horizontal panning looks wonky.
Many directors and producers refuse to allow their films to be altered by cropping; consequently, many movies on DVD are released in letterbox format, sketched in Figure 1.14 below. In letterbox format, the entirety of the widescreen image is maintained, and the top and bottom of the 4:3 frame are unused. (Typically, either grey or black is displayed.)
Conventional 4:3 material can be adapted to 16:9 in pillarbox format, shown in Figure 1.15. The full height of the display is used; the left and right of the widescreen frame are blanked. However, consumer electronics (CE) manufacturers were concerned about consumers complaining about unused screen area after upconversion of SD. So, CE vendors devised schemes to stretch the image horizontally to eliminate the side panels.
The centre panel below, Figure 1.17, shows an image with correct geometry. To its left (Figure 1.16), the image is squeezed horizontally to 75%; to its right (Figure 1.18), it is stretched horizontally to 133.3%. The distortion is so blatant that you may suspect that I have
Figure 1.16 Squeeze to 3/4 is necessary if 16:9 material is crudely resized to fit a 4:3 frame.
Figure 1.17 A normal image of Barbara Morris is shown here for comparison.
Figure 1.18 Stretch to 4/3 is necessary if 4:3 material is crudely resized to fit 16:9.
CHAPTER 1 |
RASTER IMAGES |
17 |
Details concerning frame rates and interlace are found in Flicker, refresh rate, and frame rate, on page 83.
exaggerated the effect – but the images here are distorted by exactly the amounts that would be used for SD-to-HD and HD-to-SD conversion to fit the frame width. Such shrinking and stretching is disastrous to picture integrity – but it has been commonplace since the introduction of HD to consumer television in North America. Failure of content distributors and consumer electronics manufacturers to properly respect picture aspect ratio has been, in my opinion, the most serious engineering error made in the introduction of HD systems to North America.
Frame rates
SD broadcast television historically used interlaced scanning. In 480i (“NTSC”) systems, a frame rate of
30/1.001 Hz (“29.97 Hz”) is standard; in 576i (“PAL”) systems, a frame rate of 25 Hz is standard. The frame
rates of composite NTSC and PAL video are rigid. Component video systems potentially have flexibility in the choice of frame rate. However, production and distribution infrastructure is generally locked-in to one of two frame rates, 25 Hz or 29.97 Hz. For international distribution of programming, frame-rate conversion is necessary either in the distribution infrastructure or in consumer equipment.
Frame rates have historically been chosen on
a regional basis to match the prevailing AC power line frequency. Efforts were made in the 1990s to establish a single worldwide frame rate for HD; these efforts were unsuccessful. Origination and broadcasting of HD typically takes place at the prevailing power-line frequency, 50 Hz or (nominally) 60 Hz. Certain lighting units used for acquisition flash at twice the AC power line frequency (though well above the perceptual flicker sensitivity). If a camera operates at a frame rate different from the AC line frequency, such flashing is liable to “beat” with the frame rate of the camera to produce an objectionable low-frequency strobing.
With distribution of video across commodity IP networks to consumer PCs, decoding recovers the native frame rate of the program, but generally no attempt is made to synchronize the display system. Poor motion portrayal often results.
18 |
DIGITAL VIDEO AND HD ALGORITHMS AND INTERFACES |
Image acquisition
and presentation |
2 |
The basic proposition of digital imaging is summarized in Figure 2.1. Image data is captured, processed, and/or recorded, then presented to a viewer. As outlined in the caption, and detailed later, appearance depends upon display and viewing conditions. Viewing ordinarily takes place in conditions different from those in effect at the time of capture of a scene. If those conditions differ,
a nontrivial mapping of the captured image data – picture rendering – must be imposed in order to achieve faithful portrayal, to the ultimate viewer, of the appearance of the scene (as opposed to its physical stimulus).
Figure 2.1 Image acquisition takes place in a camera, which
captures light from the scene, converts the
light to a signal, and – in most cameras – performs
certain image processing operations. The signal may then be recorded, further processed, and/or distributed. Finally, the signal
is converted to light at a display device. The appearance of the displayed image depends upon display conditions (such as peak luminance); upon viewing
conditions (such as the surroundings of the display surface); and upon conditions dependent upon both the display and its environment (such as contrast ratio). It is common for the scene to be much brighter than the displayed image: The scene may be captured in daylight, with white at 30,000 cd·m-2, but a studio display produces white of just
100 cd·m-2. The usual goal of imaging is not to match the physical stimulus associated with the scene (say, at daylight luminance levels), but to match the viewers’ expectation of the appearance of the scene. Producing an appearance match requires imposing a nontrivial mapping – termed picture rendering – that maps scene luminance to display luminance.
19