parameters for MB1 affects the coding performance of MB2 since, for example, the coding modes of MB2 (e.g. MV, intra prediction mode, etc.) may be differentially encoded from the coding modes of MB1.

Achieving near-optimum rate–distortion performance can be a very complex problem indeed, many times more complex than the video coding process itself. In a practical CODEC, the choice of optimisation strategy depends on the available processing power and acceptable coding latency. So-called ‘two-pass’ encoding is widely used in offline encoding, in which each frame is processed once to generate sequence statistics which then influence the coding strategy in the second coding pass (often together with a rate control algorithm to achieve a target bit rate or file size).

Many alternative rate–distortion optimisation strategies have been proposed (such as those based on Lagrangian optimisation) and a useful review can be found in [6]. Rate– distortion optimisation should not be considered in isolation from computational performance. In fact, video CODEC optimisation is (a least) a three-variable problem since rate, distortion and computational complexity are all inter-related. For example, rate–distortion optimised mode decisions are achieved at the expense of increased complexity, ‘fast’ motion estimation algorithms often achieve low complexity at the expense of motion estimation (and hence coding) performance, and so on. Coding performance and computational performance can be traded against each other. For example, a real-time coding application for a hand-held device may be designed with minimal processing load at the expense of poor rate–distortion performance, whilst an application for offline encoding of broadcast video data may be designed to give good rate–distortion performance, since processing time is not an important issue but encoded quality is critical.

7.5 RATE CONTROL

The MPEG-4 Visual and H.264 standards require each video frame or object to be processed in units of a macroblock. If the control parameters of a video encoder are kept constant (e.g. motion estimation search area, quantisation step size, etc.), then the number of coded bits produced for each macroblock will change depending on the content of the video frame, causing the bit rate of the encoder output (measured in bits per coded frame or bits per second of video) to vary. Typically, an encoder with constant parameters will produce more bits when there is high motion and/or detail in the input sequence and fewer bits when there is low motion and/or detail. Figure 7.35 shows an example of the variation in output bitrate produced by coding the Office sequence (25 frames per second) using an MPEG-4 Simple Profile encoder, with a fixed quantiser step size of 12. The first frame is coded as an I-VOP (and produces a large number of bits because there is no temporal prediction) and successive frames are coded as P-VOPs. The number of bits per coded P-VOP varies between 1300 and 9000 (equivalent to a bitrate of 32–225 kbits per second).

This variation in bitrate can be a problem for many practical delivery and storage mechanisms. For example, a constant bitrate channel (such as a circuit-switched channel) cannot transport a variable-bitrate data stream. A packet-switched network can support varying throughput rates but the mean throughput at any point in time is limited by factors such as link rates and congestion. In these cases it is necessary to adapt or control the bitrate produced by a video encoder to match the available bitrate of the transmission mechanism. CD-ROM

x 104 Encoder buffer contents (channel bitrate 100kbps)

Figure 7.37 Buffer example (encoder; channel bitrate 100 kbps)

4 kbits (and hence removes 4 kbits from the buffer) in every frame period. Figure 7.37 plots the contents of the encoder buffer (y-axis) against elapsed time (x-axis). The first I-VOP generates over 50 kbits and subsequent P-VOPs in the early part of the sequence produce relatively few bits and so the buffer contents drop for the first 2 seconds as the channel bitrate exceeds the encoded bitrate. At around 3 seconds the encoded bitrate starts to exceed the channel bitrate and the buffer fills up.

Figure 7.38 shows the state of the decoder buffer, filled at a rate of 100 kbps (4 kbits per frame) and emptied as the decoder extracts each frame. It takes half a second before the first complete coded frame (54 kbits) is received. From this point onwards, the decoder is able to extract and decode frames at the correct rate (25 frames per second) until around 4 seconds have elapsed. At this point, the decoder buffer is emptied and the decoder ‘stalls’ (i.e. it has to slow down or pause decoding until enough data are available in the buffer). Decoding picks up again after around 5.5 seconds.

If the decoder stalls in this way it is a problem for video playback because the video clip ‘freezes’ until enough data available to continue. The problem can be partially solved by adding a deliberate delay at the decoder. For example, Figure 7.39 shows the results if the decoder waits for 1 second before it starts decoding. Delaying decoding of the first frame allows the buffer contents to reach a higher level before decoding starts and in this case the contents never drop to zero and so playback can proceed smoothly2 .

2 Varying throughput rates from the channel can also be handled using a decoder buffer. For example, a widely-used technique for video streaming over IP networks is for the decoder to buffer a few seconds of coded data before commencing decoding. If data throughput drops temporarily (for example due to network congestion) then decoding can continue as long as data remain in the buffer.

These examples show that a variable coded bitrate can be adapted to a constant bitrate delivery medium using encoder and decoder buffers. However, this adaptation comes at a cost of buffer storage space and delay and (as the examples demonstrate) the wider the bitrate variation, the larger the buffer size and decoding delay. Furthermore, it is not possible to cope with an arbitrary variation in bitrate using this method, unless the buffer sizes and decoding delay are set at impractically high levels. It is usually necessary to implement a feedback mechanism to control the encoder output bitrate in order to prevent the buffers from overor under-flowing.

Rate control involves modifying the encoding parameters in order to maintain a target output bitrate. The most obvious parameter to vary is the quantiser parameter or step size (QP) since increasing QP reduces coded bitrate (at the expense of lower decoded quality) and vice versa. A common approach to rate control is to modify QP during encoding in order to (a) maintain a target bitrate (or mean bitrate) and (b) minimise distortion in the decoded sequence. Optimising the tradeoff between bitrate and quality is a challenging task and many different approaches and algorithms have been proposed and implemented. The choice of rate control algorithm depends on the nature of the video application, for example:

(a)Offline encoding of stored video for storage on a DVD. Processing time is not a particular constraint and so a complex algorithm can be employed. The goal is to ‘fit’ a compressed video sequence into the available storage capacity whilst maximising image quality and ensuring that the decoder buffer of a DVD player does not overflow or underflow during decoding. Two-pass encoding (in which the encoder collects statistics about the video sequence in a first pass and then carries out encoding in a second pass) is a good option in this case.

(b)Encoding of live video for broadcast. A broadcast programme has one encoder and multiple decoders; decoder processing and buffering is limited whereas encoding may be carried out in expensive, fast hardware. A delay of a few seconds is usually acceptable and so there is scope for a medium-complexity rate control algorithm, perhaps incorporating two-pass encoding of each frame.

(c)Encoding for two-way videoconferencing. Each terminal has to carry out both encoding and decoding and processing power may be limited. Delay must be kept to a minimum (ideally less than around 0.5 seconds from frame capture at the encoder to display at the decoder). In this scenario a low-complexity rate control algorithm is appropriate. Encoder and decoder buffering should be minimised (in order to keep the delay small) and so the encoder must tightly control output rate. This in turn may cause decoded video quality to vary significantly, for example it may drop significantly when there is an increase in movement or detail in the video scene.

Recommendation H.264 does not (at present) specify or suggest a rate control algorithm (however, a proposal for H.264 rate control is described in [39]). MPEG-4 Visual describes a possible rate control algorithm in an Informative Annex [40] (i.e. use of the algorithm is not mandatory). This algorithm, known as the Scalable Rate Control (SRC) scheme, is appropriate for a single video object (a rectangular V.O. that covers the entire frame) and a range of bit rates and spatial/temporal resolutions. The SRC attempts to achieve a target bit rate over a certain number of frames (a ‘segment’ of frames, usually starting with an I-VOP) and assumes the following model for the encoder rate R: