Embedded Robotics (Thomas Braunl, 2 ed, 2006)

2 Sensors

2.8.3 Inclinometer

Inclinometers measure the absolute orientation angle within a specified range, depending on the sensor model. The sensor output is also model-dependent, with either analog signal output or PWM being available. Therefore, interfacing to an embedded system is identical to accelerometers (see Section 2.8.1).

Since inclinometers measure the absolute orientation angle about an axis and not the derivative, they seem to be much better suited for orientation measurement than a gyroscope. However, our measurements with the Seika inclinometer showed that they suffer a time lag when measuring and also are prone to oscillation when subjected to positional noise, for example as caused by servo jitter.

Especially in systems that require immediate response, for example balancing robots in Chapter 9, gyroscopes have an advantage over inclinometers. With the components tested, the ideal solution was a combination of inclinometer and gyroscope.

2.9 Digital Camera

Digital cameras are the most complex sensors used in robotics. They have not been used in embedded systems until recently, because of the processor speed and memory capacity required. The central idea behind the EyeBot development in 1995 was to create a small, compact embedded vision system, and it became the first of its kind. Today, PDAs and electronic toys with cameras are commonplace, and digital cameras with on-board image processing are available on the consumer market.

For mobile robot applications, we are interested in a high frame rate, because our robot is moving and we want updated sensor data as fast as possible. Since there is always a trade-off between high frame rate and high resolution, we are not so much concerned with camera resolution. For most applications for small mobile robots, a resolution of 60u80 pixels is sufficient. Even from such a small resolution we can detect, for example, colored objects or obstacles in the way of a robot (see 60u80 sample images from robot soccer in Figure 2.12). At this resolution, frame rates (reading only) of up to 30 fps (frames per second) are achievable on an EyeBot controller. The frame rate will drop, however, depending on the image processing algorithms applied.

The image resolution must be high enough to detect a desired object from a specified distance. When the object in the distance is reduced to a mere few pixels, then this is not sufficient for a detection algorithm. Many higher-level image processing routines are non-linear in time requirements, but even simple linear filters, for example Sobel edge detectors, have to loop through all pixels, which takes some time [Bräunl 2001]. At 60u80 pixels with 3 bytes of color per pixel this amounts to 14,400 bytes.

30

Digital Camera

Figure 2.12: Sample images with 60u80 resolution

2.9.1 Camera Sensor Hardware

In recent years we have experienced a shift in camera sensor technology. The previously dominant CCD (charge coupled device) sensor chips are now being overtaken by the cheaper to produce CMOS (complementary metal oxide semiconductor) sensor chips. The brightness sensitivity range for CMOS sensors is typically larger than that of CCD sensors by several orders of magnitude. For interfacing to an embedded system, however, this does not make a difference. Most sensors provide several different interfacing protocols that can be selected via software. On the one hand, this allows a more versatile hardware design, but on the other hand sensors become as complex as another microcontroller system and therefore software design becomes quite involved.

Typical hardware interfaces for camera sensors are 16bit parallel, 8bit parallel, 4bit parallel, or serial. In addition, a number of control signals have to be provided from the controller. Only a few sensors buffer the image data and allow arbitrarily slow reading from the controller via handshaking. This is an

31

2 Sensors

Figure 2.13: EyeCam camera module

ideal solution for slower controllers. However, the standard camera chip provides its own clock signal and sends the full image data as a stream with some frame-start signal. This means the controller CPU has to be fast enough to keep up with the data stream.

The parameters that can be set in software vary between sensor chips. Most common are the setting of frame rate, image start in (x,y), image size in (x,y), brightness, contrast, color intensity, and auto-brightness.

The simplest camera interface to a CPU is shown in Figure 2.14. The camera clock is linked to a CPU interrupt, while the parallel camera data output is connected directly to the data bus. Every single image byte from the camera will cause an interrupt at the CPU, which will then enable the camera output and read one image data byte from the data bus.

Figure 2.14: Camera interface

Every interrupt creates considerable overhead, since system registers have to be saved and restored on the stack. Starting and returning from an interrupt takes about 10 times the execution time of a normal command, depending on the microcontroller used. Therefore, creating one interrupt per image byte is not the best possible solution. It would be better to buffer a number of bytes and then use an interrupt much less frequently to do a bulk data transfer of image data. Figure 2.15 shows this approach using a FIFO buffer for intermediate storing of image data. The advantage of a FIFO buffer is that it supports unsynchronized read and write in parallel. So while the camera is writing data

32

Digital Camera

to the FIFO buffer, the CPU can read data out, with the remaining buffer contents staying undisturbed.The camera output is linked to the FIFO input, with the camera’s pixel clock triggering the FIFO write line. From the CPU side, the FIFO data output is connected to the system’s data bus, with the chip select triggering the FIFO read line. The FIFO provides three additional status lines:

•Empty flag

•Full flag

•Half full flag

These digital outputs of the FIFO can be used to control the bulk reading of data from the FIFO. Since there is a continuous data stream going into the FIFO, the most important of these lines in our application is the half full flag, which we connected to a CPU interrupt line. Whenever the FIFO is half full, we initiate a bulk read operation of 50% of the FIFO’s contents. Assuming the CPU responds quickly enough, the full flag should never be activated, since this would indicate an imminent loss of image data.

Figure 2.15: Camera interface with FIFO buffer

2.9.2 Camera Sensor Data

We have to distinguish between grayscale and color cameras, although, as we will see, there is only a minor difference between the two. The simplest available sensor chips provide a grayscale image of 120 lines by 160 columns with 1 byte per pixel (for example VLSI Vision VV5301 in grayscale or VV6301 in color). A value of zero represents a black pixel, a value of 255 is a white pixel, everything in between is a shade of gray. Figure 2.16 illustrates such an image. The camera transmits the image data in row-major order, usually after a certain frame-start sequence.

Creating a color camera sensor chip from a grayscale camera sensor chip is very simple. All it needs is a layer of paint over the pixel mask. The standard Bayer pattern technique for pixels arranged in a grid is the Bayer pattern (Figure 2.17). Pixels in odd rows (1, 3, 5, etc.) are colored alternately in green and red, while pixels in even rows (2, 4, 6, etc.) are colored alternately in blue and green.

33

2 Sensors

Figure 2.16: Grayscale image

With this colored filter over the pixel array, each pixel only records the intensity of a certain color component. For example, a pixel with a red filter will only record the red intensity at its position. At first glance, this requires 4 bytes per color pixel: green and red from one line, and blue and green (again) from the line below. This would result effectively in a 60u80 color image with an additional, redundant green byte per pixel.

However, there is one thing that is easily overlooked. The four components red, green1, blue, and green2 are not sampled at the same position. For example, the blue sensor pixel is below and to the right of the red pixel. So by treating the four components as one pixel, we have already applied some sort of filtering and lost information.

Figure 2.17: Color image

Demosaicing A technique called “demosaicing” can be used to restore the image in full 120u160 resolution and in full color. This technique basically recalculates the three color component values (R, G, B) for each pixel position, for example by averaging the four closest component neighbors of the same color. Figure 2.18 shows the three times four pixels used for demosaicing the red, green, and blue components of the pixel at position [3,2] (assuming the image starts in the top left corner with [0,0]).

34

Digital Camera

Figure 2.18: Demosaic of single pixel position

Averaging, however, is only the simplest method of image value restoration and does not produce the best results. A number of articles have researched better algorithms for demosaicing [Kimmel 1999], [Muresan, Parks 2002].

2.9.3 Camera Driver

There are three commonly used capture modes available for receiving data from a digital camera:

•Read mode: The application requests a frame from the driver and blocks CPU execution. The driver waits for the next complete frame from the camera and captures it. Once a frame has been completely read in, the data is passed to the application and the application continues. In this mode, the driver will first have to wait for the new frame to start. This means that the application will be blocked for up to two frames, one to find the start of a new frame and one to read the current frame.

•Continuous capture mode: In this mode, the driver continuously captures a frame from the camera and stores it in one of two buffers. A pointer to the last buffer read in is passed to the application when the application requests a frame.

•Synchronous continuous capture mode: In this mode, the driver is working in the background. It receives every frame from the camera and stores it in a buffer. When a frame has been completely read in, a trap signal/software interrupt is sent to the application. The application’s signal handler then processes the data. The processing time of the interrupt handler is limited by the acquisition time for one camera image.

Most of these modes may be extended through the use of additional buffers. For example, in the synchronous capture mode, a driver may fill more than a single buffer. Most high-end capture programs running on workstations use the synchronous capture mode when recording video. This of course makes

35

2 Sensors

sense, since for recording video, all frames (or as many frames as possible) lead to the best result.

The question is which of these capture modes is best suited for mobile robotics applications on slower microprocessors. There is a significant overhead for the M68332 when reading in a frame from the camera via the parallel port. The camera reads in every byte via the parallel port. Given the low resolution color camera sensor chip VLSI Vision VV6301, 54% of the CPU usage is used to read in a frame, most of which will not actually be used in the application.

Another problem is that the shown image is already outdated (one frame old), which can affect the results. For example, when panning the camera quickly, it may be required to insert delays in the code to wait for the capture driver to catch up to the robot motion.

Therefore, the “read” interface is considered the most suitable one for mobile robotics applications. It provides the least amount of overhead at the cost of a small delay in the processing. This delay can often be eliminated by requesting a frame just before the motion command ends.

2.9.4 Camera RoBIOS Interface

All interaction between camera and CPU occurs in the background through external interrupts from the sensor or via periodic timer interrupts. This makes the camera user interface very simple. The routines listed in Program 2.1 all apply to a number of different cameras and different interfaces (i.e. with or without hardware buffers), for which drivers have been written for the EyeBot.

Program 2.1: Camera interface routines

typedef BYTE image [imagerows][imagecolumns]; typedef BYTE colimage[imagerows][imagecolumns][3];

The only mode supported for current EyeCam camera models is NORMAL, while older QuickCam cameras also support zoom modes. CAMInit returns the

36

Digital Camera

code number of the camera found or an error code if not successful (see Appendix B.5.4).

The standard image size for grayscale and color images is 62 rows by 82 columns. For grayscale, each pixel uses 1 byte, with values from 0 (black) over 128 (medium-gray) to 255 (white). For color, each pixel comprises 3 bytes in the order red, green, blue. For example, medium green is represented by (0, 128, 0), fully red is (255, 0, 0), bright yellow is (200, 200, 0), black is (0, 0, 0), white is (255, 255, 255).

The standard camera read functions return images of size 62u82 (including a 1-pixel-wide white border) for all camera models, irrespective of their internal resolution:

This originated from the original camera sensor chips (QuickCam and EyeCam C1) supplying 60u80 pixels. A single-pixel-wide border around the image had been added to simplify coding of image operators without having to check image boundaries.

Function CAMGetColFrame has a second parameter that allows immediate conversion into a grayscale image in-place. The following call allows grayscale image processing using a color camera:

image buffer; CAMGetColFrame((colimage*)&buffer, 1);

Newer cameras like EyeCam C2, however, have up to full VGA resolution. In order to be able to use the full image resolution, three additional camera interface functions have been added for reading images at the camera sensor’s resolution (i.e. returning different image sizes for different camera models, see Appendix B.5.4). The functions are:

•CAMGetFrameMono (read one grayscale image)

•CAMGetFrameColor (read one color image in RGB 3byte format)

•CAMGetFrameBayer (read one color image in Bayer 4byte format)

Since the data volume is considerably larger for these functions, they may require considerably more transmission time than the CAMGetFrame/CAMGetColFrame functions.

Different camera models support different parameter settings and return different camera control values. For this reason, the semantics of the camera routines CAMSet and CAMGet is not unique among different cameras. For the camera model EyeCam C2, only the first parameter of CAMSet is used, allowing the specification of the camera speed (see Appendix B.5.4):

FPS60, FPS30, FPS15, FPS7_5, FPS3_75, FPS1_875

For cameras EyeCam C2, routine CAMGet returns the current frame rate in frames per second (fps), the full supported image width, and image height (see Appendix B.5.4 for details). Function CAMMode can be used for switching the camera’s auto-brightness mode on or off, if supported by the camera model used (see Appendix B.5.4).

37

2 Sensors

Program 2.2: Camera application program

5int main()

6{ camera=CAMInit(NORMAL);

7LCDMenu("","","","End");

8while (KEYRead()!=KEY4)

9{ CAMGetFrame (&grayimg);

10LCDPutGraphic(&grayimg);

11}

12return 0;

13}

2.10References

BARSHAN, B., AYRULU, B., UTETE, S. Neural network-based target differentiation using sonar for robotics applications, IEEE Transactions on Robotics and Automation, vol. 16, no. 4, August 2000, pp. 435-442 (8)

BRÄUNL, T. Parallel Image Processing, Springer-Verlag, Berlin Heidelberg, 2001

DINSMORE, Data Sheet Dinsmore Analog Sensor No. 1525, Dinsmore Instrument Co., http://dinsmoregroup.com/dico, 1999

EVERETT, H.R. Sensors for Mobile Robots, AK Peters, Wellesley MA, 1995

38

References

JÖRG, K., BERG, M. Mobile Robot Sonar Sensing with Pseudo-Random Codes, IEEE International Conference on Robotics and Automation 1998 (ICRA ‘98), Leuven Belgium, 16-20 May 1998, pp. 2807-2812 (6)

KIMMEL, R. Demosaicing: Image Reconstruction from Color CCD Samples, IEEE Transactions on Image Processing, vol. 8, no. 9, Sept. 1999, pp. 1221-1228 (8)

KUC, R. Pseudoamplitude scan sonar maps, IEEE Transactions on Robotics and Automation, vol. 17, no. 5, 2001, pp. 767-770

MURESAN, D., PARKS, T. Optimal Recovery Demosaicing, IASTED International Conference on Signal and Image Processing, SIP 2002, Kauai

Hawaii, http://dsplab.ece.cornell.edu/papers/conference/ sip_02_6.pdf, 2002, pp. (6)

PRECISION NAVIGATION, Vector Electronic Modules, Application Notes, Precision Navigation Inc., http://www.precisionnav.com, July 1998

SHARP, Data Sheet GP2D02 - Compact, High Sensitive Distance Measuring Sensor, Sharp Co., data sheet, http://www.sharp.co.jp/ecg/, 2006

SICK, Auto Ident Laser-supported sensor systems, Sick AG, http:// www.sick.de/de/products/categories/auto/en.html, 2006

SMITH, C. Vision Support System for Tracked Vehicle, B.E. Honours Thesis, The Univ. of Western Australia, Electrical and Computer Eng., supervised by T. Bräunl, 2002

STAMATIOU, N. Sensor processing for a tracked vehicle, B.E. Honours Thesis, The Univ. of Western Australia, Electrical and Computer Eng., supervised by T. Bräunl, 2002

39