Performance Tip 22.1

Starting the Player takes less time if the Player has already prefetched the media before invoking start.

When the media clip ends, the Player generates a ControllerEvent of type EndOfMediaEvent. Most media players “rewind” the media clip after reaching the end so users can see or hear it again from the beginning. Method endOfMedia (lines 249– 253) handles the EndOfMediaEvent and resets the media clip to its beginning position by invoking Player method setMediaTime with a new Time (package javax.media) of 0 (line 251). Method setMediaTime sets the position of the media to a specific time location in the media, and is useful for “jumping” to a different part of the media. Line 252 invokes Player method stop, which ends media processing and places the Player in state Stopped. Invoking method start on a Stopped Player that has not been closed resumes media playback.

Often, it is desirable to configure the media before presentation. In the next section, we discuss interface Processor, which has more configuration capabilities than interface Player. Processors enable a program to format media and to save it to a file.

22.3 Formatting and Saving Captured Media

The Java Media Framework supports playing and saving media from capture devices such as microphones and video cameras. This type of media is known as captured media. Capture devices convert analog media into digitized media. For example, a program that captures an analog voice from a microphone attached to computer can create a digitized file from a that recording.

The JMF can access video capture devices that use Video for Windows drivers. Also, JMF supports audio capture devices that use the Windows Direct Sound Interface or the Java Sound Interface. The Video for Windows driver provides interfaces that enable Windows applications to access and process media from video devices. Similarly, Direct Sound and Java Sound are interfaces that enable applications to access sound devices such as hardware sound cards. The Solaris Performance Pack provides support for Java Sound and Sun Video capture devices on the Solaris platform. For a complete list of devices supported by JMF, visit JMF’s official Web site.

The SimplePlayer application presented in Fig. 22.1 allowed users to play media from a capture device. A locator string specifies the location of a capture device that the SimplePlayer demo accesses. For example, to test the SimplePlayer’s capturing capabilities, plug a microphone into a sound card’s microphone input jack. Typing the locator string javasound:// in the Open Location input dialog specifies that media should be input from the Java Sound-enabled capture device. The locator string initializes the MediaLocator that the Player needs for the audio material captured from the microphone.

Although SimplePlayer provides access to capture devices, it does not format the media or save captured data. Figure 22.2 presents a program that accomplishes these two new tasks. Class CapturePlayer provides more control over media properties via class

DataSource (package javax.media.protocol). Class DataSource provides the connection to the media source, then abstracts that connection to allow users to manipulate it. This program uses a DataSource to format the input and output media. The DataSource passes the formatted output media to a Controller, which will format it

further so that it can be saved to a file. The Controller that handles media is a Processor, which extends interface Player. Finally, an object that implements interface DataSink saves the captured, formatted media. The Processor object handles the flow of data from the DataSource to the DataSink object.

JMF and Java Sound use media sources extensively, so programmers must understand the arrangement of data in the media. The header on a media source specifies the media format and other essential information needed to play the media. The media content usually consists of tracks of data, similar to tracks of music on a CD. Media sources may have one or more tracks that contain a variety of data. For example, a movie clip may contain one track for video, one track for audio, and a third track for closed-captioning data for the hearing-impaired.

1// Fig. 22.2: CapturePlayer.java

2 // Presents and saves captured media

3

4 // Java core packages

5import java.awt.*;

6 import java.awt.event.*;

7import java.io.*;

8 import java.util.*;

9

10// Java extension packages

11import javax.swing.*;

12import javax.swing.event.*;

13import javax.media.*;

14import javax.media.protocol.*;

15import javax.media.format.*;

16import javax.media.control.*;

17import javax.media.datasink.*;

19 public class CapturePlayer extends JFrame {

20

21// capture and save button

22private JButton captureButton;

24// component for save capture GUI

25private Component saveProgress;

27// formats of device's media, user-chosen format

28private Format formats[], selectedFormat;

29

30// controls of device's media formats

31private FormatControl formatControls[];

33// specification information of device

34private CaptureDeviceInfo deviceInfo;

36// vector containing all devices' information

37private Vector deviceList;

38

Fig. 22.2 Formatting and saving media from capture devices (part 1 of 9).

92// initialize and configure capture device

93public void actionPerformed( ActionEvent actionEvent )

94{

Fig. 22.2 Formatting and saving media from capture devices (part 3 of 9).

247// show saving monitor dialog

248int result = JOptionPane.showConfirmDialog( this,

249saveProgress, "Save capture in progress...",

253// terminate saving if user presses "OK" or closes dialog

254if ( ( result == JOptionPane.OK_OPTION ) ||

260System.out.println ( "Capture closed." );

261}

262}

263

264// process captured media and save to file

265public void captureSaveFile()

266{

267// array of desired saving formats supported by tracks

268Format outFormats[] = new Format[ 1 ];

269

270 outFormats[ 0 ] = selectedFormat;

271

272// file output format

273FileTypeDescriptor outFileType =

274 new FileTypeDescriptor( FileTypeDescriptor.QUICKTIME );

275

276// set up and start processor and monitor capture

277try {

Fig. 22.2 Formatting and saving media from capture devices (part 6 of 9).

Chapter 22 Java Media Framework and Java Sound (on CD) 1259

Class CapturePlayer (Fig. 22.2) illustrates capturing, setting media formats and saving media from capture devices supported by JMF. The simplest test of the program uses a microphone for audio input. Initially, the GUI has only one button, which users click to begin the configuration process. Users then select a capture device from a pull-down menu dialog. The next dialog box has options for the format of the capture device and file output. The third dialog box asks users to save the media to a specific file. The final dialog box provides a volume control and the option of monitoring the data. Monitoring allows users to hear or see the media as it is captured and saved without modifying it in any way. Many media capture technologies offer monitoring capabilities. For example, many video recorders have a screen attachment to let users see what the camera is capturing without looking through the viewfinder. In a recording studio producers can listen to live music being recorded through headphones in another room. Monitoring data is different from playing data in that it does not make any changes to the format of the media or affect the data being sent to the Processor.

In the CapturePlayer program, lines 14–16 import the JMF Java extension packages javax.media.protocol, javax.media.format and javax.media.control, which contain classes and interfaces for media control and device formatting. Line 17 imports the JMF package javax.media.datasink, which contains classes for outputting formatted media. The program uses the classes and interfaces provided by these packages to obtain the desired capture device information, set the format of the capture device, create a Processor to handle the captured media data, create a DataSink to write the media data to a file and monitor the saving process.

CapturePlayer is capable of setting the media format. JMF provides class Format to describe the attributes of a media format, such as the sampling rate (which controls the quality of the sound) or whether the media should be in stereo or mono format. Each media format is encoded differently and can be played only with a media handler that supports its particular format. Line 28 declares an array of the Formats that the capture device supports and a Format reference for the user-selected format (selectedFormat).

After obtaining the Format objects, the program needs access to the formatting controls of the capture device. Line 31 declares an array to hold the FormatControls which will set the capture-device format. Class CapturePlayer sets the desired Format for the media source through the device’s FormatControls (line 31). The CaptureDeviceInfo reference deviceInfo (line 34) stores the capture device information, which will be placed in a Vector containing all of the device’s information.

Class DataSource connects programs to media data sources, such as capture devices. The SimplePlayer of Figure 22.1 accessed a DataSource object by invoking Manager method createPlayer, passing in a MediaLocator. However, class CapturePlayer accesses the DataSource directly. Line 40 declares two DataSources—inSource connects to the capture device’s media and outSource connects to the output data source to which the captured media will be saved.

An object that implements interface Processor provides the primary function that controls and processes the flow of media data in class CapturePlayer (line 46). The class also creates an object that implements interface DataSink to write the captured data to a file (line 43).

Clicking the Capture and Save File button configures the capture device by invoking method actionPerformed (lines 93–171) in private inner class Cap-

tureHandler (lines 90–173). An instance of inner class CaptureHandler is registered to listen for ActionEvents from captureButton (line 62). The program provides users with a list of available capture devices when lines 96–97 invoke CaptureDeviceManager static method getDeviceList. Method getDeviceList obtains all of the computer’s available devices that support the specified Format. Specifying null as the Format parameter returns a complete list of available devices. Class CaptureDeviceManager enables a program to access this list of devices.

Lines 109–119 copy the names of all capture devices into a String array (deviceNames) for display purposes. Lines 122-123 invoke CapturePlayer method getSelectedDeviceIndex (lines 195–210) to show a selector dialog with a list of all the device names stored in array deviceNames. The method call to showInputDialog (lines 198–201) has a different parameter list than earlier examples. The first four parameters are the dialog’s parent component, message, title, and message type, as earlier chapters use. The final three, which are new, specify the icon (in this case, null), the list of values presented to the user (deviceNames), and the default selection (the first element of deviceNames). Once users select a device, the dialog returns the string, which is used to return the integer index of the selected name in deviceNames. This String helps determine the parallel element in the deviceList. This element, which is an instance of CaptureDeviceInfo, creates and configures a new device from which the desired media can be recorded.

A CaptureDeviceInfo object encapsulates the information that the program needs to access and configure a capture device, such as location and format preferences. Calling methods getLocator (line 143) and getFormats (line 132) access these pieces of information. Lines 129–130 access the new CaptureDeviceInfo that the user specified in the deviceList. Next, lines 135–136 call inSource’s disconnect method to disengage any previously opened capture devices before connecting the new device.

Lines 142–143 invoke Manager method createDataSource to obtain the DataSource object that connects to the capture device’s media source, passing in the capture device’s MediaLocator object as an argument. CaptureDeviceInfo method getLocator returns the capture device’s MediaLocator. Method createDataSource in turn invokes DataSource method connect which establishes a connection with the capture device. Method createDataSource throws a NoDataSourceException if it cannot locate a DataSource for the capture device. An IOException occurs if there is an error opening the device.

Before capturing the media, the program needs to format the DataSource as specified by the user in the Format Selection dialog. Lines 146–147 use CaptureDevice method getFormatControls to obtain an array of FormatControls for DataSource inSource. An object that implements interface FormatControl specifies the format of the DataSource. DataSource objects can represent media sources other than capture devices, so for this example the cast operator in line 146 manipulates object inSource as a CaptureDevice and accesses capture device methods such as getFormatControls. Line 150 invokes method getSelectedFormat (lines 213–219) to display an input dialog from which users can select one of the available formats. Lines 176–192 call the method setDeviceFormat to set the media output format for the device to the user-selected Format. Each capture device can have several FormatControls, so setDeviceFormat uses FormatControl method setFormat to specify the format for each FormatControl object.

Line 287 invokes method makeDataWriter (lines 325–399) to create a DataSink object that can save the file. Manager method createDataSink requires the DataSource of the Processor and the MediaLocator for the new file as arguments. Within makeDataWriter, lines 229–242 invokes method getSaveFile to prompt users to specify the name and location of the file to which the media should be saved. The File object saveFile stores the information. Processor method getDataOutput (line 333) returns the DataSource from which it received the media. Lines 344–345 create a new MediaLocator for saveFile. Using this MediaLocator and the

DataSource, lines 349–350 create a DataSink object which writes the output media from the DataSource to the file in which the data will be saved, as specified by the

MediaLocator. Method createDataSink throws a NoDataSinkException if it cannot create a DataSink that can read data from the DataSource and output the data to the location specified by the MediaLocator. This failure may occur as a result of invalid media or an invalid MediaLocator.

The program needs to know when to stop outputting data, so lines 353–369 register a

DataSinkListener to listen for DataSinkEvents. DataSinkListener method dataSinkUpdate (lines 359–365) is called when each DataSinkEvent occurs. If the DataSinkEvent is an EndOfStreamEvent, indicating that the Processor has been closed because the capture stream connection has closed, line 364 closes the DataSink. Invoking DataSink method close stops the data transfer. A closed DataSink cannot be used again.

Common Programming Error 22.2

The media file output with a DataSink will be corrupted if the DataSink is not closed properly.

Software Engineering Observation 22.3

Captured media may not yield an EndOfMediaEvent if the media’s endpoint cannot be determined.

After setting up the DataSink and registering its listener, line 372 calls DataSink method open to connect the DataSink to the destination that the MediaLocator specifies. Method open throws a SecurityException if the DataSink attempts to write to a destination for which the program does not have write permission, such as a read-only file.

Line 373 calls DataSink method start to initiate data transfer. At this point, the program returns from method makeDataWriter back to method captureSaveFile (lines 265–322). Although the DataSink prepares itself to receive the transfer and indicates that it is ready by calling start, the transfer does not actually take place until the Processor’s start method is called. The invocation of Processor method start begins the flow of data from the capture device, formats the data and transfers that data to the DataSink. The DataSink writes the media to a file, which completes the process performed by class CapturePlayer.

While Processor encodes the data and the DataSink saves it to a file, CapturePlayer monitors the process. Monitoring provides a method of overseeing data as the capture device collects it. Lines 294–296 obtain an object that implements interface MonitorControl (package javax.media.control) from the Processor by calling method getControl. Line 299 calls MonitorControl method getControlCom-

ponent to obtain the GUI component that displays the monitored media. MonitorControls typically have a checkbox to enable or disable displaying media. Also, audio devices have a volume control and video devices have a control for setting the preview frame rate. Line 301 invokes method showSaveMonitor (lines 245–261) to display the monitoring GUI in a dialog box. To terminate capturing, users can press the OK button or close the dialog box (lines 254–261). For some video capture devices, the Processor needs to be in a Stopped state to enable monitoring of the saving and capturing process.

Thus far, we have discussed JMF’s capabilities to access, present and save media content. Our final JMF example demonstrates how to send media between computers using JMF’s streaming capabilities.

22.4 RTP Streaming

Streaming media refers to media that is transferred from a server to a client in a continuous stream of bytes. The client can begin playing the media while still downloading the media from the server. Audio and video media files are often many megabytes in size. Live events, such as concerts or football game broadcasts, may have indeterminable sizes. Users could wait until a recording of a concert or game is posted, then download the entire recording. However, at today’s Internet connection speeds, downloading such a broadcast could take days and typically users prefer to listen to live broadcasts as they occur. Streaming media enables client applications to play media over the Internet or over a network without downloading the entire media file at once.

In a streaming media application, the client typically connects to a server that sends a stream of bytes containing the media back to the client. The client application buffers (i.e. stores locally) a portion of the media, which the client begins playing after a certain portion has been received. The client continually buffers additional media, providing users with an uninterrupted clip, as long as network traffic does not prevent the client application from buffering additional bytes. With buffering, users experience the media within seconds of when the streaming begins, even though all of the material has not been received.

Performance Tip 22.3

Streaming media to a client enables the client to experience the media faster than if the client must wait for an entire media file to download.

Demand for real-time, robust multimedia is rising dramatically as the speed of Internet connections increase. Broadband Internet access, which provides high-speed network connections to the Internet for so many home users, is becoming more popular, though the number of users remains relatively small compared to the total number of Internet users. With faster connections, streaming media can provide a better multimedia experience. Users with slower connections can still experience the multimedia, though with lesser quality. The wide range of applications that use streaming media is growing. Applications that stream video clips to clients have expanded to provide real-time broadcast feeds. Thousands of radio stations stream music continuously over the Internet. Client applications such as RealPlayer have focused on streaming media content with live radio broadcasts. Applications are not limited to audio and video server-to-client streaming. For example, teleconferencing and video conferencing applications increase efficiency in everyday business by reducing the need for business people to travel great distances to attend meetings.

JMF provides a streaming media package that enables Java applications to send and receive streams of media in some of the formats discussed earlier in this chapter. For a complete list of formats, see the official JMF Web site:

java.sun.com/products/java-media/jmf/2.1.1/formats.html

JMF uses the industry-standard Real-Time Transport Protocol (RTP) to control media transmission. RTP is designed specifically to transmit real-time media data.

The two mechanisms for streaming RTP-supported media are passing it through a DataSink and buffering it. The easier mechanism to use is a DataSink, which writes the contents of a stream to a host destination (i.e., a client computer), via the same techniques shown in Fig. 22.2 to save captured media to a file. In this case, however, the destination MediaLocator’s URL would be specified in the following format:

rtp://host:port/contentType

where host is the IP address or host name of the server, port is the port number on which the server is streaming the media and contentType is either audio or video.

Using a DataSink as specified allows only one stream to be sent at a time. To send multiple streams (e.g., as a karaoke video with separate tracks for video and audio would) to multiple hosts, a server application must use RTP session managers. An RTPManager (package javax.media.rtp) provides more control over the streaming process, permitting specification of buffer sizes, error checking and streaming reports on the propagation delay (the time it takes for the data to reach its destination).

The program in Fig. 22.3 and Fig. 22.4 demonstrates streaming using the RTP session manager. This example supports sending multiple streams in parallel, so separate clients must be opened for each stream. This example does not show a client that can receive the RTP stream. The program in Fig. 22.1 (SimplePlayer) can test the RTP server by specifying an RTP session address

rtp://127.0.0.1:4000/audio

as the location SimplePlayer should open to begin playing audio. To execute the streaming media server on a different computer, replace 127.0.0.1 with either the IP address or host name of the server computer.

1// Fig. 22.3: RTPServer.java

2 // Provides configuration and sending capabilities

3 // for RTP-supported media files

4

5 // Java core packages

6 import java.io.*;

7 import java.net.*;

8

9 // Java extension packages

10import javax.media.*;

11import javax.media.protocol.*;

12import javax.media.control.*;

13import javax.media.rtp.*;

14import javax.media.format.*;

Fig. 22.3 Serving streaming media with RTP session managers (part 1 of 7).

68processor.configure();

69}

70

71// source connection error

72catch ( IOException ioException ) {

73 ioException.printStackTrace();

74return false;

75}

76

77// exception thrown when no processor could

78// be found for specific data source

79catch ( NoProcessorException noProcessorException ) {

80 noProcessorException.printStackTrace();

81return false;

82}

83

84 return true;

85

86 } // end method beginSession

87

88// ControllerListener handler for processor

89private class ProcessorEventHandler

90extends ControllerAdapter {

91

92// set output format and realize

93// configured processor

94public void configureComplete(

95ConfigureCompleteEvent configureCompleteEvent )

96{

103processor.realize();

104}

105

106// start sending when processor is realized

107public void realizeComplete(

108RealizeCompleteEvent realizeCompleteEvent )

109{

117System.out.println( "\nTransmission failed." );

118}

119

Fig. 22.3 Serving streaming media with RTP session managers (part 3 of 7).

120// stop RTP session when there is no media to send

121public void endOfMedia( EndOfMediaEvent mediaEndEvent )

122{

123 stopTransmission();

124System.out.println ( "Transmission completed." );

125}

126

127 } // end inner class ProcessorEventHandler

128

129// set output format of all tracks in media

130public void setOutputFormat()

131{

132// set output content type to RTP capable format

133processor.setContentDescriptor(

134 new ContentDescriptor( ContentDescriptor.RAW_RTP ) );

135

136// get all track controls of processor

137tracks = processor.getTrackControls();

139// supported RTP formats of a track

140Format rtpFormats[];

141

142// set each track to first supported RTP format

143// found in that track

144for ( int i = 0; i < tracks.length; i++ ) {

Fig. 22.3 Serving streaming media with RTP session managers (part 4 of 7).

173 } // end for loop

174

175 } // end method setOutputFormat

176

177// send media with boolean success value

178public boolean transmitMedia()

179{

180outSource = processor.getDataOutput();

182 if ( outSource == null ) {

183 System.out.println ( "No data source from media!" );

184

185return false;

186}

187

188// rtp stream managers for each track

189rtpManager = new RTPManager[ tracks.length ];

191// destination and local RTP session addresses

192SessionAddress localAddress, remoteAddress;

194// RTP stream being sent

195SendStream sendStream;

197// IP address

198 InetAddress ip;

199

200// initialize transmission addresses and send out media

201try {

Fig. 22.3 Serving streaming media with RTP session managers (part 5 of 7).

Fig. 22.3 Serving streaming media with RTP session managers (part 7 of 7).

Class RTPServer’s purpose is to stream media content. As in previous examples, the RTPServer (Fig. 22.3) sets up the media, processes and formats it, then outputs it. ControllerEvents and the various states of the streaming process drive this process. Processing of the media has three distinct parts—Processor initialization, Format configuration and data transmission. The code in this example contains numerous confirmation messages displayed to the command prompt and an emphasis on error checking. A problem during streaming will most likely end the entire process and it will need to be restarted.

To test RTPServer, class RTPServerTest (Fig. 22.4) creates a new RTPServer object and passes its constructor (lines 35–40) three arguments—a String representing the media’s location, a String representing the IP address of the client and a port number for streaming content. These arguments contain the information class RTPServer needs to obtain the media and set up the streaming process. Following the general approach outlined in SimplePlayer (Fig. 22.1) and CapturePlayer (Fig. 22.2), class RTPServer obtains a media source, configures the source through a type of Controller and outputs the data to a specified destination.

RTPServerTest calls RTPServer method beginSession (lines 44–86) to set up the Processor that controls the data flow. Line 47 creates a MediaLocator and initializes it with the media location stored in fileName. Line 58 creates a Processor for the data specified by that MediaLocator.

track stream. Line 211 assigns the port number to be two more than the previous port number, because the each track uses one port number for the stream control and one to actually stream the data. Lines 218–219 instantiate a new local session address where the stream is located (i.e., the RTP address that clients use to obtain the media stream) with the local IP address and a port number as parameters. Line 219 invokes InetAddress method getLocalHost to get the local IP address. Line 222 instantiates the client’s session address, which the RTPManager uses as the stream’s target destination. When line 225 calls RTPManager method initialize, the method directs the local streaming session to use the local session address. Using object remoteAddress as a parameter, line 228 calls RTPManager method addTarget to open the destination session at the specified address. To stream media to multiple clients, call RTPManager method addTarget for each destination address. Method addTarget must be called after the session is initialized and before any of the streams are created on the session.

The program can now create the streams on the session and start sending data. The streams are created in the current RTP session with the DataSource outSource (obtained at line 180) and the source stream index (i.e. media track index) in lines 234–235. Invoking method start on the SendStream (line 238) and on the Processor (line 246) starts transmission of the media streams, which may cause exceptions. An InvalidSessionAddressException occurs when the specified session address is invalid. An UnsupportedFormatException occurs when an unsupported media format is specified or if the DataSource’s Format has not been set. An IOException occurs if the application encounters networking problems. During the streaming process, RTPManagers can be used with related classes in package javax.media.rtp and package javax.media.rtp.event controls the streaming process and send reports to the application.

The program should close connections and stop streaming transmission when it reaches the end of the streaming media or when the program terminates. When the Processor encounters the end of the media, it generates an EndOfMediaEvent. In response, the program calls method endOfMedia (lines 121–125). Line 123 invokes method stopTransmission (lines 274–303) to stop and close the Processor (lines 279–282). After calling stopTransmission, streaming cannot resume because it disposes of the Processor and the RTP session resources. Lines 288–297 invoke RTPManager method removeTargets (lines 292–293) to close streaming to all destinations. RTPManager method dispose (line 296) is also invoked, releasing the resources held by the RTP sessions. The class RTPServerTest (Fig. 22.4), explicitly invokes method stopTransmission when users terminates the server application (line 40).

1 // Fig. 22.4: RTPServerTest.java

2 // Test class for RTPServer

3

4// Java core packages

5 import java.awt.event.*;

6 import java.io.*;

7 import java.net.*;

8

Fig. 22.4 Application to test class RTPServer from Fig. 22.3 (part 1 of 6).

131// get file from computer

132public File getFile()

133{

134JFileChooser fileChooser = new JFileChooser();

145return fileChooser.getSelectedFile();

146}

147

148// get media location from user

149public String getMediaLocation()

150{

151String input = JOptionPane.showInputDialog(

152this, "Enter MediaLocator" );

154// if user presses OK with no input

155if ( input != null && input.length() == 0 ) {

156System.err.println( "No input!" );

157return null;

158}

159

160return input;

161}

162

163// method getting IP string from user

164public String getIP()

165{

Fig. 22.4 Application to test class RTPServer from Fig. 22.3 (part 4 of 6).