Embedded Systems Design - An Introduction to Processes Tools and Techniques (A. Berger, 2002)

Figure 8.4: Emulation control system.

Schematic diagram of a memory-mapper circuit, in which 18 address bits map the 18 high-order address bits from the microprocessor to the addresses of physical memory, either target-system memory or emulation memory. The remaining six bits are used to assign unique personalities to each of the 16KB blocks.

Note

address range assigned, for example, to a particular 16KB block. The advantage of this method is that it doesn’t require a large, fast memory array, as is required in Figure 8.4. In fact, a comparator-based design is very much like the trigger circuit that you use in the logic analyzer to compare the address, data, and status states to your trigger condition. The disadvantage is that it’s harder to explain in a simple picture how this kind of a memory-mapper system works, so this discussion uses the RAM-based architecture.

The mapper memory is actually wider than it needs to be to perform the memorymapping function. The extra memory bits assign “personalities” to each of the 16KB memory blocks. Thus, you can map memory as emulation RAM or target RAM. This means that the memory system steering logic sends the address, data, and status information to the appropriate memory space, either in the target system or in emulation memory. The other bits allow you to detect error conditions in the code flow. If, for example, you assign an emulation ROM attribute to a 16KB memory block, you can cause the emulation system to stop execution, trigger a trace, or both, if a write signal is asserted to a memory address in that block. However, because each block of emulated ROM really is RAM memory located within the emulator, each block can be easily loaded with new software, much like the ROM emulator.

Team-Fly®

Figure 8.4 points out an important difference between overlay memory and substitution memory. Because overlay memory is mappable and can be assigned a personality, it can be given precedence over memory in the target system by assigning the selected block of address space to emulation memory instead of the target memory. Substitution memory can be used only to replace a block of memory in the target system that is accessed through the ROM socket.

The two other personality bits of interest in Figure 8.4 allow you to protect a region of memory from being written to, even though it’s assigned to be target or emulation RAM. This feature allows you to track any attempts to overwrite a variable defined as const, as usually happens when a pointer goes haywire. The coverage bit is discussed in the next chapter with coverage testing. For now, the coverage bit enables you to gather statistics about what distinct memory locations in that 16KB block have been accessed by the user’s program code.

aOverlay memory completes the design of the generic emulator. In contrast to the individual tools described in earlier chapters, the emulator offers:

A single connection to the target board

Reliable run control even in unreliable systems

Cross-triggering of the run control system, allowing trace and breakpoint facilities to work together

Real-time monitoring of illegal memory accesses

Despite these advantages, the emulator is only slightly more complex than a logic analyzer.

Timing Constraints

Unfortunately, emulators can’t be used with every system. The two main obstacles are timing constraints and physical constraints. The physical constraints are similar to those limiting the use of logic analyzers and were discussed in the “Making the Connection” sidebar. The timing constraints, however, are a direct consequence of the emulator’s steering logic.

The fact that a RAM circuit is needed to map the memory regions has a serious implication for the target system’s capability to work properly with the emulator. If the target system is designed with very little timing margin between the processor and the target memory, or other memory-mapped I/O devices in the target, you could have a problem.

Note

Hardware designers aren’t wrong to design very close to the edge with respect to timing constraints. For example, according to the data book, a particular memory chip might have a minimum access time of 45ns, a typical access time of 55ns, and a maximum access time of 70ns. This data represents a statistical sampling by the manufacturer that, over the appropriate temperature range for that version of the device, most of the devices would be able to work at 55ns, and almost all of them (six sigma) would work at 70ns. This also implies that almost none of them would work at access times less than 45ns.

However, it would be possible to get a lot of parts and painstakingly sort through them until you find some that actually worked reliably below 45ns. Alternatively, if you could guarantee that the part never got warmer than room temperature, then a processor with an access time of 41ns might actually work with a memory chip that had a maximum access time of 70ns. Thus, you could save some money by

buying a part that is less costly but slower than the part that a conservatively designed system might demand.

Recall that the hardware designer is trying to do the right thing and not overdesign the hardware. Thus, if the processor-to-memory-system-interface timing specification calls for a minimum of 55ns and you use a memory chip with a typical specification of 50ns, you should be safe. The design might not pass a stringent hardware design review, but it will most likely work. However, your emulator might need to steal 10ns from the 50ns because the memory-mapper RAM needs 10ns to do the address translation. After all, it’s RAM memory as well. Suddenly, the 55ns access time becomes 45ns, and the system stops working when the emulator is substituted for the actual processor.

Several possible solutions exist. If the hardware designer added the capability to the design or the feature was already built into the microcontroller, you could add one or more wait states to the memory access time. Another possibility is to slow down the processor clock so that the access times increase. This might work, but it probably won’t.

Note

A wait state is a method of stretching the memory access time by one or more clock cycles. For example, the Motorola MF5206e ColdFire processors that are used in my Embedded Systems class have an onchip wait state generator that can add from one to 16 wait states onto a memory access. This is convenient because the experiment boards and cables tend to mess things up a bit, and three wait states are required for reliable access to the experiment boards.

aLast, you might be able to use faster parts in your development boards and go with the slower ones in the released product. Of course, the downside is that you’ve started on a slippery slope. Your emulator now has intruded upon your target system by forcing it to run slower (a wait state takes time). Your target system might not run properly because the time to execute critical code segments has increased beyond some critical threshold. The emulator designers are aware of this, and they can sometimes choose processors that have been selected for speed to compensate for the access time used up by the mapping memory. However, sometimes it just won’t work.

Distributed Emulators

Earlier in this chapter, I mentioned the concept of constructing an emulator from these separate pieces:

Run control

Real-time instruction and data trace

Memory substitution

The generic emulator used a debug kernel instead of a debug core. You can still build the ICE and integrate the three functions even if you have a debug core. In this case, much of the run control circuitry is simplified, but the three elements pretty much function in the same relative way.

A JTAG or BDM debugger, a ROM emulator, and a logic analyzer can be brought together and used to provide similar functionality to an ICE. When properly

Chapter 9: Testing

Embedded systems software testing shares much in common with application software testing. Thus, much of this chapter is a summary of basic testing concepts and terminology. However, some important differences exist between application testing and embedded systems testing. Embedded developers often have access to hardware-based test tools that are generally not used in application development. Also, embedded systems often have unique characteristics that should be reflected in the test plan. These differences tend to give embedded systems testing its own distinctive flavor. This chapter covers the basics of testing and test case development and points out details unique to embedded systems work along the way.

Why Test?

Before you begin designing tests, it’s important to have a clear understanding of why you are testing. This understanding influences which tests you stress and (more importantly) how early you begin testing. In general, you test for four reasons:

To find bugs in software (testing is the only way to do this)

To reduce risk to both users and the company

To reduce development and maintenance costs

To improve performance

To Find the Bugs

One of the earliest important results from theoretical computer science is a proof (known as the Halting Theorem) that it’s impossible to prove that an arbitrary program is correct. Given the right test, however, you can prove that a program is incorrect (that is, it has a bug). It’s important to remember that testing isn’t about proving the “correctness” of a program but about finding bugs. Experienced programmers understand that every program has bugs. The only way to know how many bugs are left in a program is to test it with a carefully designed and measured test plan.

To Reduce Risk

Testing minimizes risk to yourself, your company, and your customers. The objectives in testing are to demonstrate to yourself (and regulatory agencies, if appropriate) that the system and software works correctly and as designed. You want to be assured that the product is as safe as it can be. In short, you want to discover every conceivable fault or weakness in the system and software before it’s deployed in the field.

Developing Mission-Critical Software Systems

Incidents such as the Therac-25 radiation machine malfunction — in which several patients died due to a failure in the software monitoring the patients — should serve as a sobering reminder that the lives of real people might depend on the quality of the code that you write. I’m not an expert on writing safety-critical code,

but I’ve identified some interesting articles on mission-critical software development:

Brown, Doug. “Solving the Software Safety Paradox.” Embedded Systems Programming, December 1998, 44.

Cole, Bernard. “Reliability Becomes an All-Consuming Goal.” Electronic Engineering Times, 13 December 1999, 90.

Douglass, Bruce Powel. “Safety-Critical Embedded Systems.”

Embedded Systems Programming, October 1999, 76.

Knutson, Charles and Sam Carmichael. “Safety First: Avoiding Software Mishaps.” Embedded Systems Programming, November 2000, 28.

Murphy, Niall. “Safe Systems Through Better User Interfaces.”

Embedded Systems Programming, August 1998, 32.

Tindell, Ken. “Real-Time Systems Raise Reliability Issues.” Electronic Engineering Times, 17 April 2000, 86.

To Reduce Costs

The classic argument for testing comes from Quality Wars by Jeremy Main.

In 1990, HP sampled the cost of errors in software development during the year. The answer, $400 million, shocked HP into a completely new effort to eliminate mistakes in writing software. The $400M waste, half of it spent in the labs on rework and half in the field to fix the mistakes that escaped from the labs, amounted to one-third of the company’s total R&D budget…and could have increased earnings by almost 67%.[5]

The earlier a bug is found, the less expensive it is to fix. The cost of finding errors and bugs in a released product is significantly higher than during unit testing, for example (see Figure 9.1).

Figure 9.1: The cost to fix a problem.

Simplified graph showing the cost to fix a problem as a function of the time in the product life cycle when the defect is found. The costs associated with finding and fixing the Y2K problem in embedded systems is a close approximation to an infinite cost model.

To Improve Performance

Testing maximizes the performance of the system. Finding and eliminating dead code and inefficient code can help ensure that the software uses the full potential of the hardware and thus avoids the dreaded “hardware re-spin.”

When to Test?

It should be clear from Figure 9.1 that testing should begin as soon as feasible. Usually, the earliest tests are module or unit tests conducted by the original developer. Unfortunately, few developers know enough about testing to build a thorough set of test cases. Because carefully developed test cases are usually not employed until integration testing, many bugs that could be found during unit testing are not discovered until integration testing. For example, a major network equipment manufacturer in Silicon Valley did a study to figure out the key sources of its software integration problems. The manufacturer discovered that 70 percent of the bugs found during the integration phase of the project were generated by code that had never been exercised before that phase of the project.

Unit Testing

Individual developers test at the module level by writing stub code to substitute for the rest of the system hardware and software. At this point in the development cycle, the tests focus on the logical performance of the code. Typically, developers test with some average values, some high or low values, and some out-of-range values (to exercise the code’s exception processing functionality). Unfortunately, these “black-box” derived test cases are seldom adequate to exercise more than a fraction of the total code in the module.

Regression Testing

It isn’t enough to pass a test once. Every time the program is modified, it should be retested to assure that the changes didn’t unintentionally “break” some unrelated behavior. Called regression testing, these tests are usually automated through a test script. For example, if you design a set of 100 input/output (I/O) tests, the regression test script would automatically execute the 100 tests and compare the output against a “gold standard” output suite. Every time a change is made to any part of the code, the full regression suite runs on the modified code base to insure that something else wasn’t broken in the process.

From the Trenches

I try to convince my students to apply regression testing to their course projects; however, because they are students, they never listen to me. I’ve had more than a few projects turned in that didn’t work because the student made a minor change at 4:00AM on the day it was due, and the project suddenly unraveled. But, hey, what do I know?

When to Test?

It should be clear from Figure 9.1 that testing should begin as soon as feasible. Usually, the earliest tests are module or unit tests conducted by the original developer. Unfortunately, few developers know enough about testing to build a thorough set of test cases. Because carefully developed test cases are usually not employed until integration testing, many bugs that could be found during unit testing are not discovered until integration testing. For example, a major network equipment manufacturer in Silicon Valley did a study to figure out the key sources of its software integration problems. The manufacturer discovered that 70 percent of the bugs found during the integration phase of the project were generated by code that had never been exercised before that phase of the project.

Unit Testing

Individual developers test at the module level by writing stub code to substitute for the rest of the system hardware and software. At this point in the development cycle, the tests focus on the logical performance of the code. Typically, developers test with some average values, some high or low values, and some out-of-range values (to exercise the code’s exception processing functionality). Unfortunately, these “black-box” derived test cases are seldom adequate to exercise more than a fraction of the total code in the module.

Regression Testing

It isn’t enough to pass a test once. Every time the program is modified, it should be retested to assure that the changes didn’t unintentionally “break” some unrelated behavior. Called regression testing, these tests are usually automated through a test script. For example, if you design a set of 100 input/output (I/O) tests, the regression test script would automatically execute the 100 tests and compare the output against a “gold standard” output suite. Every time a change is made to any part of the code, the full regression suite runs on the modified code base to insure that something else wasn’t broken in the process.

From the Trenches

I try to convince my students to apply regression testing to their course projects; however, because they are students, they never listen to me. I’ve had more than a few projects turned in that didn’t work because the student made a minor change at 4:00AM on the day it was due, and the project suddenly unraveled. But, hey, what do I know?

Which Tests?

Because no practical set of tests can prove a program correct, the key issue becomes what subset of tests has the highest probability of detecting the most errors, as noted in The Art of Software Testing by Glen Ford Myers[6]. The problem of selecting appropriate test cases is known as test case design. Although dozens of strategies exist for generating test cases, they tend to fall into two fundamentally different approaches: functional testing and coverage testing. Functional testing (also known as black-box testing) selects tests that assess how well the implementation meets the requirements specification. Coverage testing (also known as white-box testing) selects cases that cause certain portions of the code to be executed. (These two strategies are discussed in more detail later.) Both kinds of testing are necessary to test rigorously your embedded design. Of the two, coverage testing implies that your code is stable, so it is reserved for testing a completed or nearly completed product. Functional tests, on the other hand, can be written in parallel with the requirements documents. In fact, by starting with the functional tests, you can minimize any duplication of efforts and

rewriting of tests. Thus, in my opinion, functional tests come first. Everyone agrees that functional tests can be written first, but Ross[7], for example, clearly believes they are most useful during system integration … not unit testing.

The following is a simple process algorithm for integrating your functional and coverage testing strategies:

1.Identify which of the functions have NOT been fully covered by the functional tests.

2.Identify which sections of each function have not been executed.

3.Identify which additional coverage tests are required.

4.Run new additional tests.