Hi444PP stands for High 4:4:4 predictive profile.

VP8 has three reference frames.

High profiles

The original H.264 features were augmented by the Fidelity range extensions (FRExt), which are available in the high profiles.

Ten bit sample depth is available in Hi10P and Hi422P; fourteen bit sample depth is available in Hi444P.

Hi422P and Hi444P offer 4:2:2 chroma subsampling: Y’CBCR 4:2:2 (loosely, Y’UV 4:2:2) can be coded. Hi444P offers 4:4:4 “chroma subsampling” – that is, no subsampling at all.

Hierarchy

The syntax elements in an H.264 bitstream have a hierarchical structure like that of MPEG-2. The bitstream hierarchy of H.264 – the syntax hierarchy – is as follows:

•sequence

•picture

•slice

•macroblock

•macroblock partition

•sub-macroblock partition

•block

•sample

The video coding layer (VCL) comprises elements at the slice level and below. A network abstraction layer (NAL) defines NAL units to convey coded data. Information at layers above the VCL – that is, at the sequence and picture levels – is conveyed in non-VCL NAL units. The two types of NAL units (VCL and non-VCL) can be transmitted in different streams, for example to achieve higher network robustness, though specification of such transmission mechanisms is outside the scope of H.264.

Supplemental enhancement information (SEI) and video usability information (VUI) are “messages” inserted into non-VCL NAL units of the coded bitstream. SEI comprises sequence and picture parameter sets (SPS and PPS). VUI conveys information comparable to the contents of the sequence display extension of MPEG-2.

Multiple reference pictures

MPEG-2 has two reference frames: one in the past, and one in the “future.” The “future” frame is available to predict B-pictures that lie earlier in display order.

Multiple reference pictures may be useful to predict “uncovered background” depending upon the encoder’s ability to discover it. Use of “future” reference pictures incurs latency, and may be impractical in some applications.

In H.264, multiple reference pictures are allowed – between 4 and 13, depending upon level. If the material being coded has a quick cut to a reverse shot, the encoder can instruct the decoder to retain the picture at the end of the first shot, and use it to predict the picture upon return from the reverse shot. Reference pictures can be addressed in arbitrary order.

Slices

Slices offer a decoder the option of parallelism: No intra prediction crosses a slice boundary. Decoder state effectively resets on slice boundaries, so slices limit the spatial extent of transmission-induced impairments. Slices can be coded redundantly to further mitigate against transmission error.

Spatial intra prediction

In MPEG-2, a macroblock may be coded entirely independently as an I-macroblock, or may exploit temporal prediction and be coded as a P-macroblock. In the development of H.264 it was realized that decoded intra macroblocks above the current one, and those to the left in the same slice, have prediction value in the spatial domain. H.264 implements intra prediction based upon that data, where image data above or to the left is copied directionally in several modes. The prediction can then be refined by transform-coded quantized residuals in the usual way. Intra prediction uses only information from intra-coded macroblocks.

There is also an intra-PCM mode, where I-macroblock pixel data is directly coded, bypassing the transform. The mode is potentially useful at very high data rates.

Flexible motion compensation

In MPEG-2, motion prediction is accomplished in units of 16× 16 blocks of luma pixels – that is, macroblocks. The encoder tries to find a 16× 16 region of a reference picture that forms a good predictor, then codes the relative coordinates of that block into the data stream as a motion vector.

In H.264, a macroblock can be partitioned into several shapes and sizes for prediction from different regions of a reference picture, even prediction from different reference pictures. An entire macroblock can

What is 1/4-pel for luma is 1/8-pel for 4:2:0 chroma.

be predicted from one 16× 16 source; alternatively, the macroblock can be partitioned into two 8× 16 macroblock partitions, two 16× 8 macroblock partitions, or four 8× 8 macroblock partitions, all predicted independently. In high profiles, if a macroblock is partitioned into four 8× 8 macroblock partitions, each of those can be partitioned into two 4× 8 sub-macroblock partitions, two 8× 4 sub-macroblock partitions, or four 4× 4 sub-macroblock partitions, again all predicted independently. A macroblock can be associated with up to 16 motion vectors.

Quarter-pel motion-compensated interpolation

In MPEG-2, motion vectors can have 1/2-pixel precision with respect to luma samples. In H.264, motioncompensated interpolation can be performed to quarter-pel precision – that is, motion vectors can be encoded in units of 1/4-pel (sometimes called quarterpel, or Qpel). The interpolation operation uses simple 6-tap FIR filters, and has the beneficial effect of lowpass-filtering the prediction signal in addition to delivering it at an optimal spatial position.

Weighting and offsetting of MC prediction

MPEG-2 behaves poorly in fades from one picture to another and in fades to black – or, in the case of Six Feet Under, fades to white. The DC terms of the transform coefficients are coded reasonably well, but in fade to black all of the AC terms scale down together; that stresses the quantizer. H.264 implements weighting and offsetting of MC prediction, to improve performance in fades and certain other circumstances.

16-bit integer transform

MPEG-2 followed JPEG in using the 8× 8 DCT, virtually always implemented in binary fixed-point arithmetic. The theoretical DCT matrix contains irrational numbers; encoders and decoders approximate them in fixedpoint binary integers, usually 16-bit. Neither the JPEG nor MPEG-2 standards specify the accuracy of the DCT. The encoder includes a simulation of the decoding process, but owing to different roundoff error in different implementations, the encoder’s DCT may not match the decoder’s DCT. When a decoded block is

H.264’s transform is sometimes termed HCT, which is either

H.264 cosine transform or high correlation transform depending upon whom you ask.

D11 111

Table 48.3 Two hypothetical coding schemes mapping symbols (A through D) into a bitstream are sketched. Scheme F allocates a fixed

number of bits to each symbol; Scheme V allocates a variable number of bits to symbols.

used as a prediction, the prediction formed at the decoder may not exactly match the prediction expected by the encoder. We assume that the encoder has more computational resources that the decoder, and is likely to have more accuracy, so we term the problem – perhaps unfairly to the decoder – as decoder drift.

In H.264, decoder drift is eliminated through use of a transform defined by a matrix of simple binary fractions whose inverse also comprises simple binary fractions. With 8-bit residuals and 16-bit arithmetic, no roundoff error occurs, so no drift occurs.

Quantizer

In MPEG-2, the transform coefficient quantizer levels are uniformly spaced. In H.264, the quantizer has 52 steps that are exponentially spaced: Each step increases the step size by a ratio of 1.122, that is, six

steps double the step size. (As a rough guide, increasing quantizer step size by +1 decreases bit rate by about 10%, and doubling halves the bit rate. This heuristic can be used for rate control at an encoder.)

Variable-length coding

Suppose you’re given sequences of four symbols (A, B, C, and D) to encode into a bitstream. Consider two simple coding schemes set out in Table 48.3 in the margin. Scheme F uses two bits for any of the four symbols. Scheme V uses one, two, or three bits, depending upon the symbol being coded. Both schemes faithfully encode any input sequence that is presented – that is, both encodings are lossless. However, if the input contains a lot of As, scheme V emits fewer bits than scheme F. Scheme V exemplifies the basic notion of variable-length coding: It’s advantageous to have an encoding that reflects the probabilities of the symbols being coded. In this example, scheme F is well adapted to inputs where A, B, C, and D have equal probabilities. Scheme V is well adapted to

probabilities [1/2,1/4,1/8,1/8] respectively.

In MPEG-2, a few dozen VLC coding schemes were devised for various syntax elements. H.264 required many additional syntax elements, and the developers got tired of constructing ad hoc tables. A systematic method, universal variable-length coding (UVLC) was

Table 48.4 An example of exponential Golomb coding of positive numbers 1 through 11 or integers ranging ±5 is shown.

int Coded bitstream

01

±1 01s

±2…±3 001xs

±4…±7 0001xxs

±8…±15 00001xxxs

±16…±31 000001xxxxs

±32…±63 0000001xxxxxs

±64…±127 00000001xxxxxxs

Table 48.5 Exp-Golomb coding can be generalized to signed integers represented in 1 bit, 2 bits, 3 bits, 4 bits, and more, indefinitely. The scheme favours inputs where small numbers are most likely: If inputs ±127 were equally likely, then fixed-length 8-bit two’s complement coding would be more efficient.

The pos example of Table 48.4 is constructed for ease of explanation; H.264’s unsigned integer (ue) codes are the indicated numbers less one. The int example of Table 48.4 corresponds to H.264’s signed integer (se) codes.

adopted. It is based upon the exponential Golomb scheme, an example of which is sketched in Table 48.4.

Decoding of the positive number (pos) symbols of the example proceeds as follows: If the datastream bit is 1, the coded value is 1. Otherwise, count leading zero bits, denoting the count n. Consider the following n+1 bits (including the leading 1 bit) to be the binarycoded positive number, most-significant bit first.

When used for signed integers (the int symbols of the example), decode as follows: If the datastream bit is 1, the coded value is 0. Otherwise, count leading zero bits, denoting the count n. Consider the following n bits (including the leading 1 bit) to be the absolute value of the coded number, expressed in binary, mostsignificant bit first. The trailing (n+1)th bit is the sign. The int example in Table 48.4 encodes signed inte-

gers such as those encountered in motion vector displacements. The code is easily adapted to nonnumeric symbols by simply assigning the required values or symbols to the appropriate number.

Table 48.5 shows how the coding extends to arbitrarily large numbers (or to a set of symbols of arbitrary size).

In H.264, UVLC is used at syntax levels above the transform coefficients, for data such as prediction modes and motion vectors. The UVLC scheme is not used for transform coefficients: either CAVLC or CABAC is used for those.