6BPart III: Liveness, Performance, and Testing 24BChapter 12. Testing Concurrent Programs 167

you're unlucky, dead code elimination will just speed up your program by some factor that could be explained by other means.

Dead code elimination is a problem in benchmarking statically compiled languages too, but detecting that the compiler has eliminated a good chunk of your benchmark is a lot easier because you can look at the machine code and see that a part of your program is missing. With dynamically compiled languages, that information is not easily accessible.

Many micro benchmarks perform much "better" when run with HotSpot's -server compiler than with -client, not just because the server compiler can produce more efficient code, but also because it is more adept at optimizing dead code. Unfortunately, the dead code elimination that made such short work of your benchmark won't do quite as well with code that actually does something. But you should still prefer -server to -client for both production and testing on multiprocessor systems you just have to write your tests so that they are not susceptible to dead code elimination.

Writing effective performance tests requires tricking the optimizer into not optimizing away your benchmark as dead code. This requires every computed result to be used somehow by your program in a way that does not require synchronization or substantial computation.

In PutTakeTest, we compute the checksum of elements added to and removed from the queue and combine these checksums across all the threads, but this could still be optimized away if we do not actually use the checksum value.

We happen to need it to verify the correctness of the algorithm, but you can ensure that a value is used by printing it out. However, you should avoid doing I/O while the test is actually running, so as not to distort the run time measurement.

A cheap trick for preventing a calculation from being optimized away without introducing too much overhead is to compute the hashCode of the field of some derived object, compare it to an arbitrary value such as the current value of System. nanoTime, and print a useless and ignorable message if they happen to match:

if (foo.x.hashCode() == System.nanoTime()) System.out.print(" ");

The comparison will rarely succeed, and if it does, its only effect will be to insert a harmless space character into the output. (The print method buffers output until println is called, so in the rare case that hashCode and System.nanoTime are equal no I/O is actually performed.)

Not only should every computed result be used, but results should also be unguessable. Otherwise, a smart dynamic

optimizing compiler is allowed to replace actions with pre computed results. We addressed this in the construction of

PutTakeTest, but any test program whose input is static data is vulnerable to this optimization.

12.4. Complementary Testing Approaches

While we'd like to believe that an effective testing program should "find all the bugs", this is an unrealistic goal. NASA devotes more of its engineering resources to testing (it is estimated they employ 20 testers for each developer) than any commercial entity could afford toand the code produced is still not free of defects. In complex programs, no amount of testing can find all coding errors.

The goal of testing is not so much to find errors as it is to increase confidence that the code works as expected. Since it is unrealistic to assume you can find all the bugs, the goal of a quality assurance (QA) plan should be to achieve the greatest possible confidence given the testing resources available. More things can go wrong in a concurrent program than in a sequential one, and therefore more testing is required to achieve the same level of confidence. So far we've focused primarily on techniques for constructing effective unit and performance tests. Testing is critically important for building confidence that concurrent classes behave correctly, but should be only one of the QA metholologies you employ.

Different QA methodologies are more effective at finding some types of defects and less effective at finding others. By employing complementary testing methodologies such as code review and static analysis, you can achieve greater confidence than you could with any single approach.

12.4.1. Code Review

As effective and important as unit and stress tests are for finding concurrency bugs, they are no substitute for rigorous code review by multiple people. (On the other hand, code review is no substitute for testing either.) You can and should design tests to maximize their chances of discovering safety errors, and you should run them frequently, but you should not neglect to have concurrent code reviewed carefully by someone besides its author. Even concurrency experts make

168 Java Concurrency In Practice

mistakes; taking the time to have someone else review the code is almost always worthwhile. Expert concurrent programmers are better at finding subtle races than are most test programs. (Also, platform issues such as JVM implementation details or processor memory models can prevent bugs from showing up on particular hardware or software configurations.) Code review also has other benefits; not only can it find errors, but it often improves the quality of comments describing the implementation details, thus reducing future maintenance cost and risk.

12.4.2. Static Analysis Tools

As of this writing, static analysis tools are rapidly emerging as an effective complement to formal testing and code review. Static code analysis is the process of analyzing code without executing it, and code auditing tools can analyze classes to look for instances of common bug patterns. Static analysis tools such as the open source FindBugs[9] contain bug pattern detectors for many common coding errors, many of which can easily be missed by testing or code review.

[9] http://findbugs.sourceforge.net

Static analysis tools produce a list of warnings that must be examined by hand to determine whether they represent actual errors. Historically, tools like lint produced so many false warnings as to scare developers away, but tools like

FindBugs have been tuned to produce many fewer false alarms. Static analysis tools are still somewhat primitive

(especially in their integration with development tools and lifecycle), but they are already effective enough to be a valuable addition to the testing process.

As of this writing, FindBugs includes detectors for the following concurrency related bug patterns, and more are being

added all the time:

Inconsistent synchronization. Many objects follow the synchronization policy of guarding all variables with the object's intrinsic lock. If a field is accessed frequently but not always with the this lock held, this may indicate that the synchronization policy is not being consistently followed.

Analysis tools must guess at the synchronization policy because Java classes do not have formal concurrency specifications. In the future, if annotations such as @GuardedBy are standardized, auditing tools could interpret annotations rather than having to guess at the relationship between variables and locks, thus improving the quality of analysis.

Invoking Thread.run. Thread implements Runnable and therefore has a run method. However, it is almost always a mistake to call Thread.run directly; usually the programmer meant to call Thread.start.

Unreleased lock. Unlike intrinsic locks, explicit locks (see Chapter 13) are not automatically released when control exits the scope in which they were acquired. The standard idiom is to release the lock from a finally block; otherwise the lock can remain unreleased in the event of an Exception.

Empty synchronized block. While empty synchronized blocks do have semantics under the Java Memory Model, they are frequently used incorrectly, and there are usually better solutions to whatever problem the developer was trying to solve.

Double checked locking. Double checked locking is a broken idiom for reducing synchronization overhead in lazy initialization (see Section 16.2.4) that involves reading a shared mutable field without appropriate synchronization.

Starting a thread from a constructor. Starting a thread from a constructor introduces the risk of subclassing problems, and can allow the this reference to escape the constructor.

Notification errors. The notify and notifyAll methods indicate that an object's state may have changed in a way that would unblock threads that are waiting on the associated condition queue. These methods should be called only when the state associated with the condition queue has changed. A synchronized block that calls notify or notifyAll but does not modify any state is likely to be an error. (See Chapter 14.)

Condition wait errors. When waiting on a condition queue, Object.wait or Condition. await should be called in a loop, with the appropriate lock held, after testing some state predicate (see Chapter 14). Calling Object.wait or Condition.await without the lock held, not in a loop, or without testing some state predicate is almost certainly an

error.

Misuse of Lock and Condition. Using a Lock as the lock argument for a synchronized block is likely to be a typo, as is

calling Condition.wait instead of await (though the latter would likely be caught in testing, since it would throw an

IllegalMonitorStateException the first time it was called).