ALGORITHM

The Algorithm for Parameter Selection in 3D Shooting, Independent from Stereopair Registration Methods

A.S. Melkumov

Director, 3D Digital Department

Cinema and Photo Research Institute (NIKFI)

Abstract

There is a wide range of screen sizes (from 3 to 30 meters) used in movie theaters. How can we properly choose 3D shooting parameters so that the movie may be viewed equally well in digital 3D cinemas, on the Giant Screen, on home TV sets, and on mobile phones?

The proposed parameter selection algorithm for 3D shooting offers a selection independent of shooting methods or equipment by replacing absolute values with relative or average ones. The described procedure has been used for live shooting, stop-motion animation, and 3D CG. Movies made using this technique are viewed comfortably both on 3D home TV and in 3D movie theaters, including the Giant Screen.

One of the main objectives in shooting live 3D or 3D CG objects is the correct selection of 3D recording parameters to give the audience a comfortable binocular picture by separately (haploscopically) presenting two planar images shot from the right and left viewing angles. This condition is essential for securing an extended comfortable viewing of 3D movies.

In the 1950s, Professor A. G. Boltyansky from the Cinema and Photo Research Institute (NIKFI) proposed a stereoscopic parameter calculation theory for the big screen. Once converted from film to digital technology, this theory has demonstrated its universal applicability, independent from the capture methods or the equipment used. The present paper offers an algorithm for 3D shooting parameter selection based on Boltyansky’s theory (also known as the "parallax theory") and on our practical experience acquired over a 30-year span.

Methods of Calculating 3D Shooting Parameters

As in conventional film production, in classical dual-angle 3D cinema the viewer is shown flat photographic images, but these images are shot and presented separately for the right and left eyes from the right and left viewing angles respectively. When these images are superimposed on the screen, the disparity of these images is visualized as a horizontal shift of the object’s conjugate points, known as the horizontal parallax (Fig.1). While observing these separate images, the viewer gets a distinct sensation of a perceived depth (stereopsis). Horizontal parallaxes determine the convergence of visual axes and can be either positive or negative in value. If a given point of the object’s right image lies to the right of the respective point of the left image, the parallax is positive. In this case, the convergence point of the visual axes is behind the screen. If a given point of the object’s right image lies to the left of the respective point of the left image, the

1

parallax is negative, and the convergence point will be located in front of the screen. (Fig.2)

A

Positive parallax

Fig 1. Horizontal parallax

There are certain viewing restrictions imposed on the horizontal parallax value range.

Restriction One

The parallax of the farthest point of the 3D composition (Pbackground) should not exceed the interaxial distance between eyes (the viewer's eye basis, Вview):

Pbackground ≤ Вview

As in real life, the binocular perception of objects at infinity on the screen should occur on parallel axes (with a zero convergence angle). This is possible if the horizontal parallax value of the infinity point is equal to the viewer’s eye basis (Вview). The value of the infinity point’s parallax in the 3D composition is called

2

the infinity parallax on the screen (P∞screen). Any increase of this value will lead to the divergence of visual axes causing discomfort during an extended viewing of the stereoscopic image (Fig. 3).

А

Вview

Fig. 3 Divergence of visual axes

Restriction Two

When a distant and a close object are both located on the same viewing axis, the parallax of the close object (Pforeground) should not exceed the viewer's eye basis (Вview) taken with a negative sign.

- Pforeground ≤ Вview

While in reality the point of accommodation (the point of focusing) for the object varies and almost always coincides with the convergence point of the visual axes directed towards this object, in viewing the stereopair image the position of the accommodation point is fixed and lies within the screen plane, while the point of the visual axes convergence can lie either in front of or behind the screen, creating a gap between the accommodation and convergence points.

But the human eye has a certain depth interval which, in photography and cinema, is called the Depth of Field (DOF), the distance between the nearest and farthest objects in a scene that appear acceptably sharp. Consequently, in observing the 3D image the accommodation area is not limited to the screen plane on which the eye is focused, but to the space within which the permitted gap between the accommodation and the convergence of the visual axes may occur. Our experience indicates that fulfillment of the Restriction Two allows the viewer to converge the visual axes within the eye’s DOF.

Restriction Three

The foreground object cropped by the image frame should have its parallax equal to zero (a zero parallax).

3

In the binocular perception, the viewer watches the 3D composition on the screen through a window formed by the image boundaries. If the object's parallax is negative but the object is edged by a frame, such object cannot be perceived by the viewer within the space of the screen hall, for psychological reasons, and will be perceived only within the screen plane. We call this psychological phenomenon (the shut-off effect caused by the screen’s edges) "the stereoscopic window paradox". As a result, the whole 3D composition, as perceived by the viewer, will “move” behind the screen so that the foreground object would be placed within the mentioned screen plane. A prolonged discrepancy between the psychologically perceived position of the foreground object and its parallax parameter is one of the sources of viewer's discomfort.

The above restrictions allow us to define the permitted parallax range for the prolonged viewing of a 3D film:

D = (+Pbackground) - (-Pforeground) = 2Bview

Remembering that the horizontal parallax value is determined by the object’s spatial position relative to the 3D camera, it is possible, using the parallax range, to explain how the depth of the comfortably viewed space is formed on the set. This depth can be figuratively described as the Depth of Field. Similar to the hyperfocal distance, there is an estimated distance (known as the main ramp distance, Lmain ramp) for each combination of the shooting basis and the lens focal distance (regardless of the aperture value). If lenses converge at this distance, we get the depth interval within which the objects placed from half this distance to the infinity will be displayed within the allowed horizontal parallax range where the difference between the accommodation and convergence will not cause the viewer any discomfort.

Ramp

Вshooting

L main ramp

Fig. 4 The Ramp

4

Unlike the focal depth, the depth of the comfortably viewable space is divided into the front-screen zone and the back-screen zone which should be considered in building the image composition. The plane positioned within the 3D composition perpendicularly to the camera axis at the ramp distance is called the ramp and is defined as the zero parallax plane because the objects placed at this distance will have the zero parallax (Fig.4)

Thus, the stereopair image captured with correctly selected parameters, including the focal distance (F), the shooting basis (Bshooting) and the ramp distance (Lramp), should meet the following requirements:

the close object of the 3D composition cropped by the frame should lie at the ramp distance and have the zero parallax;

the infinity point parallax should not exceed the viewer's eye basis;

the parallax of the nearest point perceived by the viewer in the screen hall in the static 3D composition should not exceed the viewer's eye basis value taken with a negative sign (Photo 1)

Photo 1. 3D image recorded with properly selected parameters

The 3D shooting parameters above are linked by the following equation:

where F is the focal lens distance, Bshooting is the interaxial distance between the left and right lenses, Lramp is the main ramp distance, P∞screen is the on-screen

5

infinity parallax equal to the viewer's eye basis, Kpr is the projection ratio (the ratio between the screen width and the frame width on the sensor or the film).

The Algorithm for Averaging 3D Shooting Parameters

Due to a wide range of such characteristics as the infinity parallax on the screen (as defined by viewer’s eye basis values) and the projection ratio in the equation [1] the implementation of all the above requirements within mathematically strict limits would mean using individual shooting parameters for each viewer and each screen.

In practical 3D shooting we use an averaging algorithm starting with defining the average viewer’s eye basis. With the eye basis ranging from 52 to 72 mm, the average value of 65 mm is used to calculate the 3D shooting and play-back parameters. Accordingly, the parallax of the farthest on-screen point (the infinity parallax) Р∞screen should not exceed 65 mm. This way, the average viewer will see the point at infinity on parallel axes.

It has been determined empirically that, considering the human eye physiology, the divergence of visual axes up to 70´ can still be equivalent to viewing on parallel axes if the distance between the viewer and the screen is more than two image heights. In viewing the image with the aspect ratio 16:9 (1.78), the farthest point at infinity can be comfortably viewed on the screen with the horizontal parallax up to 1% of the image width. The maximum parallax value for the close object in the 3D composition can also extend up to 1% of the image width (with a negative sign). Consequently, when the image is displayed on a 6.5 m wide screen, the infinity parallax estimated as 1% of the screen width will be equal to the average viewer's eye basis, i.e. 65 mm. Similarly to the choice of the average viewer’s eye basis, the 6.5 m wide screen can be used as the average screen when analyzing the footage in terms of comfortable viewing.

Replacing the absolute figures of extreme parallax values with relative percentages of the image width (instead of using the average screen size) makes it possible to control the stereoscopic footage on any size monitor.

In the equation [1], the projection ratio and the infinity parallax on the screen can be replaced by one variable, i.e. the infinity parallax (Р∞film) on the sensor or the film:

The projection ratio Kpr is the ratio between the screen width and the frame width on the sensor or the film:

6

Thus, if the infinity parallax on the screen is taken as 1% of the screen width, Р∞film will amount to 1% of the sensor width and will become a constant used in defining the shooting parameters throughout the whole filming process with a specific 3D camera.

Therefore, the equation [1] is transformed into another equation with two variables and one constant (Р∞film):

Thus, the shooting parameter selection is reduced to three values: the lens focal distance, the shooting basis and the main ramp distance.

The Effect of Shooting Parameters on the Picture Composition Depth

The scale and the prominence of on-screen objects are dictated by the content of the shot and by its place in the editing sequence of the scene. The focal distance of the lens is directly related to its view angle (the longer the shot, the wider the angle, and, viсe versa, the closer the shot, the narrower the angle). Contrary to conventional filmmaking methods, in stereoscopy the focal distance selection is determined not so much by the size of the view field, but by the necessity to provide a desired intensity of the stereo effect.

Just like the color image is characterized by its chromatic saturation, the stereoscopic image is defined by its perspective rendering quality, via the depth factor Ө which describes the stereoscopic intensity of the 3D image. The value of Ө is defined as the ratio between the viewer’s distance from the screen (Lview) and the distance from the camera to the object (Lshooting):

We can obtain the same image scale (M) with various focal distances by changing the distance from the camera to the object (Lshooting). In different cases we will have different depth, characterized by a certain degree of compression (Lview<Lshooting) or expansion (Lview>Lshooting). Therefore, the selection of the focal distance should be based on its effect on the object’s depth representation.

The closer we are to the object, the higher is its stereoscopic intensity.

An important stage, done simultaneously with the lens selection in accordance with the required view angle, is the determination of the 3D shooting basis for filming.

7

м›1

Вsh = Вviewer

Lsh Lbase

L main ramp=K pr*F

Fig.6 Object’s real scale

Filming people with the shooting basis equal to the viewer's eye basis is possible only when the size of the on-screen image doesn't exceed the real size of the object placed at the ramp plane (М = 1). The object’s scale in stereo presentation can be real only in one case: when the ramp width on the set is equal to the width of the screen onto which the footage will be projected (Fig. 6). But this condition is rarely feasible due to a great variability of screen sizes. Normally, the basis of 65 mm or more should be used in long shots of landscapes and architecturals without any people in the foreground. Otherwise, on the screen the people will look tiny; this phenomenon is called the miniaturization effect.

In order to simplify the problem of shooting basis selection, especially when the Director of Photography (DP) has a mirror rig capable of changing this value from zero to hundreds of millimeters, the presented method introduces the notion of the average shooting basis.

More often, in filmmaking we deal with the projectional enlargement of people's images on the screen, and this is why the shooting basis value should be inversely proportional to the scale of the object on the screen Bshooting=Bview/M. If we assume that the average screen with the aspect ratio 16:9 is 6.5 m wide and 3.65 m high and the average human height is 1.65 m, the minimum projection enlargement scale in a full shot will be 2.2. Therefore, the maximum permissible shooting basis for scenes with people should not exceed 65/2.2≈30 mm. If we consider the more common case, when the actor is present at the ramp in an upper body shot, when the enlargement scale is closer to 2.5, we can accept 25-26 mm as an optimal shooting basis. Even while capturing a long shot with the actors in the background, in order to increase the depth effect, we place foreground objects

8

within the ramp plane, so these objects may define the image scale for the whole composition. Consequently, we can speak of the average shooting basis of 25-26 mm for capturing various compositions, including medium shots, close-ups, and long shots with foreground objects (Fig. 7).

Fig. 7

Is the ramp distance a variable (chosen by the DP) or is it a derivative of the selected proportion between the focal distance and the shooting basis?

Technically speaking, lenses can be converged at any distance. But in converging them at a distance smaller than the main ramp distance one should account for the change in the horizontal parallaxes of the distant areas of the spatial composition. For example, if the ramp distance is reduced by half with the same proportion between the shooting basis and the focal distance, the infinity parallax will double. So, if the ramp distance is reduced, the shot depth should be limited by the background placed at a finite distance Lbackground which will not cause the divergence of visual axes.

If the picture composition space is limited, the DP can reduce the ramp distance and shoot from an arbitrarily chosen ramp distance. In this case, when the DP has converged the lenses for the distance Lshooting (shorter than the main ramp distance) the shooting will be done under hypertrophic depth conditions with the H (hypertrophic) index defined as:

The hypertrophic depth representation occurs due to the fact that the infinity parallax is shifted to the objects lying at a finite distance. Thus, for example, if the main ramp distance derived from the equation [5] is 5 m and the lenses are converged on the foreground object at 2.5 m, the shooting will be done with H=2 because the objects lying at the distance of 5 m will be presented with parallaxes equal to the infinity parallax. The space distortion becomes visible with H>4 (in case of wide-angle lenses, the hypertrophy is apparent with H=3), so the shooting

9

with hypertrophy is a common case when the field size of the shot has to be doubled or tripled. It is important to remember that in this case the depth of the comfortably viewed space is reduced and the shot composition depth should be limited by the background beyond which the objects will be displayed with parallaxes exceeding the viewer's eye basis, causing the divergence of visual axes and double imaging.

In selecting the shooting basis we face a contradictory situation. As seen from the equation [5], the reduction of the shooting basis causes the reduction of the main ramp distance. When shooting lengthy compositions with a close foreground, the easiest solution seems to be a reduction of the value of the shooting basis, which is often used in modern 3D film production. This method makes it possible to get closer to the object and to shoot without limiting the background distance. But the reduction of the shooting basis results in the reduction of the stereoscopic effect. It is impossible to increase the depth of the comfortably viewed space by reducing the shooting basis, the same way we adjust the depth of field by using the aperture. This leads to “flattening” of the 3D composition viewed on the screen. The shooting basis value should be determined by the image scale rather than by an arbitrarily assigned depth of the comfortably viewed space. It is to be understood that the limitations of the range of the comfortably viewed space do not stem from the optical flaws of the 3D shooting equipment; they come from the physiology of stereoscopic vision, from the brain’s ability of simultaneous perception of unequally distant objects within a given location range. The DP encounters certain difficulties imposed by the limitations of the comfortably viewed space when he blindly transfers to 3D shooting the techniques of conventional planar picture composition with a prominent foreground set against an infinitely distant background. A long-term practice of Russian 3D film production has shown that shooting with a discrete shooting basis, most often equal to the average one (Bshooting=25 mm), makes the DP more disciplined in building 3D picture composition.

Our shooting experience shows that the formula “the larger is the scale of the object on the screen, the smaller should be the shooting basis” is not always true. For example, in transition from a medium shot to a close-up (especially when a longer-focus lens is used) the shooting should continue with the same average shooting basis. This will help render the natural stereo effect enhancement in zooming in on the object and compensate for the long-focus lens’s “flattening” tendency. Thus, if a medium shot with a following close-up are both shot with a gradual shooting basis reduction, the close up will appear flatter. At the same time, if the shooting is done in a limited indoor space with a wide-angle lens, the lens should be used with caution in zooming in. In this case, it would be preferable to use the shooting basis reduced to 75% of the average.

Please note that the stereoscopic image is sensitive to changes of the lenses and the shooting basis; so, in order to retain the visual unity of spatial compositions, it is recommended to shoot while keeping the same parameter (F and B) values within a unified sequence.

10