5

MPEG-4 Visual

5.1 INTRODUCTION

ISO/IEC Standard 14496 Part 2 [1] (MPEG-4 Visual) improves on the popular MPEG-2 standard both in terms of compression efficiency (better compression for the same visual quality) and flexibility (enabling a much wider range of applications). It achieves this in two main ways, by making use of more advanced compression algorithms and by providing an extensive set of ‘tools’ for coding and manipulating digital media. MPEG-4 Visual consists of a ‘core’ video encoder/decoder model together with a number of additional coding tools. The core model is based on the well-known hybrid DPCM/DCT coding model (see Chapter 3) and the basic function of the core is extended by tools supporting (among other things) enhanced compression efficiency, reliable transmission, coding of separate shapes or ‘objects’ in a visual scene, mesh-based compression and animation of face or body models.

It is unlikely that any single application would require all of the tools available in the MPEG-4 Visual framework and so the standard describes a series of profiles, recommended sets or groupings of tools for particular types of application. Examples of profiles include Simple (a minimal set of tools for low-complexity applications), Core and Main (with tools for coding multiple arbitrarily-shaped video objects), Advanced Real Time Simple (with tools for error-resilient transmission with low delay) and Advanced Simple (providing improved compression at the expense of increased complexity).

MPEG-4 Visual is embodied in ISO/IEC 14496-2, a highly detailed document running to over 500 pages. Version 1 was released in 1998 and further tools and profiles were added in two Amendments to the standard culminating in Version 2 in late 2001. More tools and profiles are planned for future Amendments or Versions but the ‘toolkit’ structure of MPEG-4 means that any later versions of 14496-2 should remain backwards compatible with Version 1.

This chapter is a guide to the tools and features of MPEG-4 Visual. Practical implementations of MPEG-4 Visual are based on one or more of the profiles defined in the standard and so this chapter is organised according to profiles. After an overview of the standard and its approach and features, the profiles for coding rectangular video frames are discussed (Simple,

H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia.

Iain E. G. Richardson. C 2003 John Wiley & Sons, Ltd. ISBN: 0-470-84837-5

Advanced Simple and Advanced Real-Time Simple profiles). These are by far the most popular profiles in use at the present time and so they are covered in some detail. Tools and profiles for coding of arbitrary-shaped objects are discussed next (the Core, Main and related profiles), followed by profiles for scalable coding, still texture coding and high-quality (‘studio’) coding of video.

In addition to tools for coding of ‘natural’ (real-world) video material, MPEG-4 Visual defines a set of profiles for coding of ‘synthetic’ (computer-generated) visual objects such as 2D and 3D meshes and animated face and body models. The focus of this book is very much on coding of natural video and so these profiles are introduced only briefly. Coding tools in the MPEG-4 Visual standard that are not included in any Profile (such as Overlapped Block Motion Compensation, OBMC) are (perhaps contentiously!) not covered in this chapter.

5.2 OVERVIEW OF MPEG-4 VISUAL (NATURAL VIDEO CODING)

5.2.1 Features

MPEG-4 Visual attempts to satisfy the requirements of a wide range of visual communication applications through a toolkit-based approach to coding of visual information. Some of the key features that distinguish MPEG-4 Visual from previous visual coding standards include:

Efficient compression of progressive and interlaced ‘natural’ video sequences (compression of sequences of rectangular video frames). The core compression tools are based on the ITU-T H.263 standard and can out-perform MPEG-1 and MPEG-2 video compression. Optional additional tools further improve compression efficiency.

Coding of video objects (irregular-shaped regions of a video scene). This is a new concept for standard-based video coding and enables (for example) independent coding of foreground and background objects in a video scene.

Support for effective transmission over practical networks. Error resilience tools help a decoder to recover from transmission errors and maintain a successful video connection in an error-prone network environment and scalable coding tools can help to support flexible transmission at a range of coded bitrates.

Coding of still ‘texture’ (image data). This means, for example, that still images can be coded and transmitted within the same framework as moving video sequences. Texture coding tools may also be useful in conjunction with animation-based rendering.

Coding of animated visual objects such as 2D and 3D polygonal meshes, animated faces and animated human bodies.

Coding for specialist applications such as ‘studio’ quality video. In this type of application, visual quality is perhaps more important than high compression.

5.2.2 Tools, Objects, Profiles and Levels

MPEG-4 Visual provides its coding functions through a combination of tools, objects and profiles. A tool is a subset of coding functions to support a specific feature (for example, basic

Table 5.1 MPEG-4 Visual profiles for coding natural video

video coding, interlaced video, coding object shapes, etc.). An object is a video element (e.g. a sequence of rectangular frames, a sequence of arbitrary-shaped regions, a still image) that is coded using one or more tools. For example, a simple video object is coded using a limited subset of tools for rectangular video frame sequences, a core video object is coded using tools for arbitrarily-shaped objects and so on. A profile is a set of object types that a CODEC is expected to be capable of handling.

The MPEG-4 Visual profiles for coding ‘natural’ video scenes are listed in Table 5.1 and these range from Simple Profile (coding of rectangular video frames) through profiles for arbitrary-shaped and scalable object coding to profiles for coding of studio-quality video. Table 5.2 lists the profiles for coding ‘synthetic’ video (animated meshes or face/body models) and the hybrid profile (incorporates features from synthetic and natural video coding). These profiles are not (at present) used for natural video compression and so are not covered in detail in this book.

Figure 5.1 MPEG-4 Visual profiles and objects

Figure 5.1 lists each of the MPEG-4 Visual profiles (left-hand column) and visual object types (top row). The table entries indicate which object types are contained within each profile. For example, a CODEC compatible with Simple Profile must be capable of coding and decoding Simple objects and a Core Profile CODEC must be capable of coding and decoding Simple and Core objects.

Profiles are an important mechanism for encouraging interoperability between CODECs from different manufacturers. The MPEG-4 Visual standard describes a diverse range of coding tools and it is unlikely that any commercial CODEC would require the implementation of all the tools. Instead, a CODEC designer chooses a profile that contains adequate tools for the target application. For example, a basic CODEC implemented on a low-power processor may use Simple profile, a CODEC for streaming video applications may choose Advanced Real Time Simple and so on. To date, some profiles have had more of an impact on the marketplace than others. The Simple and Advanced Simple profiles are particularly popular with manufacturers and users whereas the profiles for the coding of arbitrary-shaped objects have had very limited commercial impact (see Chapter 8 for further discussion of the commercial impact of MPEG-4 Profiles).

Profiles define a subset of coding tools and Levels define constraints on the parameters of the bitstream. Table 5.3 lists the Levels for the popular Simple-based profiles (Simple,