Video signal processing

Edge-replication is appropriate for motion-compensated interpolation in video compression: The replicated samples are used as predictions, and are not displayed.

a sequence of FIR filters. A strategy is necessary to deal with filtering at the edges of the image.

Many digital image-processing (DIP) textbooks suggest padding the area outside the pixel array with copies of the edge samples, replicated as many times as necessary. The assumption is unrealistic for virtually all imaging applications, because if a small feature happens to lie at the left edge of the image, upon replication it will effectively turn into a large feature and thereby exert undue influence on the filter result – that is, exert undue influence reaching into the interior of the pixel array.

Some textbooks advocate padding the image by mirroring as many left-edge samples as necessary. In the example above, padding would mirror the leftmost 12 image columns. This approach is also unrealistic: In general-purpose imaging, there is no reasonable possibility that the missing content is estimated by mirroring.

Many textbooks consider the image to wrap in a cylinder: Missing samples outside the left-hand edge of the image are copied from the right-hand edge of the image! This concept draws from Fourier transform theory, where a finite data set is treated as being cyclic (periodic). This assumption makes the math easy, but is not justified in practice, and the wrapping strategy is even worse than edge-pixel replication.

In video, we treat the image as lying on a field of black: Unavailable samples are taken to be black. With this strategy, repeated lowpass filtering causes the implicit black background to intrude to some extent into the image. In practice, few problems are caused by this intrusion. Video image data nearly always includes some black (or blanking) samples, as I outlined in the discussion of samples per picture width and samples per active line. (See Scanning parameters, on page 86.) In studio standards, a region lying within the pixel array is designated as the clean aperture, as sketched in Figure 8.4, on page 87. This region is supposed to remain subjectively free from artifacts that originate from filtering at the picture edges.

Transition samples

In Scanning parameters, on page 86, I mentioned that it is necessary to avoid an instantaneous transition from

100%

90%

50%

10%

0%

0 1 2 3 4 5 Y’

0 1 2 C

Figure 31.1 Transition samples.

The solid line, dots (), and light shading show the luma transition; the dashed line, open circles (), and colour shading show 4:2:2 chroma limits.

480i studio standards historically accommodated up to 487 image rows, as explained in 480i line assignment, on page 446. 576i studio standards provide 574 full lines and two halflines, as explained in 576i line assignment, on page 458.

blanking to picture at the start of a line. It is also necessary to avoid an instantaneous transition from picture to blanking at the end of a line. In studio video, the first and the last few active video samples on a line are blanking transition samples. I recommend that the first luma (Y’) sample of a line be black, and that this sample be followed by three transition samples clipped to 10%, 50%, and 90% of the full signal amplitude. In 4:2:2,

I recommend that the first three colour difference (C) samples on a line be transition samples, clipped to 10%, 50%, and 90%. Figure 31.1 sketches the transition samples. The transition values should be applied by clipping, rather than by multiplication, to avoid disturbing the transition samples of a signal that already has a proper blanking transition.

Picture lines

Historically, the count of image rows in 480i systems was poorly standardized. Various standards specified between 480 and 487 “picture lines.” It is pointless to carry picture on line 21/284 or earlier, because in NTSC transmission this line is reserved for closed caption data: 482 full lines, plus the bottom halfline, now suffice. With 4:2:0 chroma subsampling, as used in JPEG, MPEG-1, and MPEG-2, a multiple of 16 picture lines is required. DCT-based transform compression is now so ubiquitous that a count of 480 lines has become de rigeur for 480i MPEG video. In 576i scanning, a rigid standard of 576 picture lines has always been enforced; fortuitously for MPEG in 576i, the number 576 happens to be a multiple of 16.

MPEG-2 accommodates the 1920× 1080 image format; however, 1080 is not a multiple of 16. In MPEG-2 coding, the bottom of each 1920× 1080 picture is padded with eight image rows containing black to form a 1920× 1088 array that is coded. The extra eight lines are discarded upon decoding.

Traditionally, the image array of 480i and 576i systems had halflines, as sketched in Figures 13.3 and 13.4 on page 132: Halfline blanking was imposed on picture information on the top and bottom lines of each frame. Neither JPEG nor MPEG provides halfline blanking: When halfline-blanked image data is presented to a JPEG or MPEG compressor, the blank

Active lines (vertically) encompass the picture height. Active samples (horizontally) encompass not only the picture width, but also up to about a dozen blanking transition samples.

HD standards specify that the 50%-points of picture width must lie no further than six samples inside the production aperture.

image data is compressed. Thankfully, halflines have been abolished from HD.

Studio video standards have no transition samples on the vertical axis: An instantaneous transition from vertical blanking to full picture is implied. However, nonpicture vertical interval information coded like video – such as VITS or VITC – may precede the picture lines in a field or frame. Active lines comprise only picture lines (and exceptionally, in 480i systems, closed caption data). LA excludes vertical interval lines.

Computer display interface standards, such as those from VESA, make no provision for nonpicture (vertical interval) lines other than blanking.

Choice of SAL and SPW parameters

In Scanning parameters, on page 86, I characterized two video signal parameters, samples per active line (SAL) and samples per picture width (SPW). Active sample counts in studio standards have been chosen for the convenience of system design; within a given scanning standard, active sample counts standardized for different sampling frequencies are not exactly proportional to the sampling frequencies.

Historically, “blanking width” was measured instead of picture width. Through the decades, there has been considerable variation in blanking width of studio standards and broadcast standards. Also, blanking width was measured at levels other than 50%, leading to an unfortunate dependency upon frequency response.

Most modern video standards do not specify picture width: It is implicit that the picture should be as wide as possible within the production aperture, subject to reasonable blanking transitions. Figure 13.1, on

page 130 indicates SAL values typical of studio practice. For digital terrestrial broadcasting of 480i and 480p, the ATSC considered the coding of transition samples to be wasteful. Instead of specifying 720 SAL, ATSC established 704 SAL. This created an inconsistency between production standards and broadcast standards: MPEG-2

macroblocks are misaligned between the two. Computer display interface standards, such as those

from VESA, do not accommodate blanking transition samples and have no concept of clean aperture. In these standards, SPW and SAL are equal.