Advanced PHP Programming

450 Chapter 19 Synthetic Benchmarks: Evaluating Code Blocks and Functions

To answer the aforementioned questions and others, you need to write synthetic benchmarks as test cases. Synthetic benchmarks provide a means for testing small portions of code or individual functions to evaluate (and, by comparison, minimize) their resource usage. By incorporating benchmarks into unit tests, you can also track performance changes in libraries over time.

Synthetic benchmarks differ from application benchmarks in that they do not attempt to simulate a realistic use of the application but instead focus simply on measuring the performance of a particular piece of code. Synthetic benchmarks have a rich history in computer science. In the 1950s, programmers used benchmarks with the goal of optimizing physical systems’ implementations. One of the original and most famous synthetic benchmarks is the Whetstone benchmark, designed to benchmark floating-point operations. Other common examples include calculating Fibonacci Sequences, using the Towers of Hanoi to test the speed of recursive function calls in a language, and using matrix multiplication to test linear algebra algorithms.

The results of synthetic benchmarks often have little bearing on the overall performance of an application.The real issue is that nothing is intrinsically broken with the idea of benchmarking; rather, it is simply an issue of optimizing the wrong parts of an application. A critical companion to benchmarking is profiling, which allows you to pinpoint the sections of an application that can benefit most from optimization.

In creating a good synthetic benchmark, you need to address the following two issues:

nDoes it test what you intend?—This might sound obvious, but it is very important to make sure a benchmark is really designed to test what you are looking for. Remember:You are not testing the whole application, but just a small component. If you do not succeed in testing that component alone, you have reduced the relevance of the benchmark.

nDoes it use the function the way you will?—Algorithms often vary dramatically, depending on the structure of their input. If you know something about the data that you will be passing to the function, it is beneficial to represent that in the test data set. Using a sample of live data is optimal.

Intentionally missing from this list is the question “Is it relevant?” Benchmarking can be a useful exercise in and of itself to help familiarize you with the nuances of PHP and the Zend Engine. Although it might not be useful to optimize array iteration in a seldomused script, having a general knowledge of the performance idioms of PHP can help you develop a coding style that needs less optimization down the road.

Benchmarking Basics

When comparing benchmarks, you need to make sure they differ with only one degree of freedom.This means that you should vary only one independent factor at a time in a test, leaving the rest of the data and algorithms as a control. Let’s say, for example, that you are writing a class that reads in a document and calculates its Flesch readability

score. If you simultaneously change the algorithms for counting words and counting sentences, you will be unable to determine which algorithm change accounts for the performance difference.

You should also keep in mind that benchmarks are highly relative. If I compare array_walk() on my laptop versus a for loop on my development server, I will likely just prove that a for loop on a more powerful machine is faster than array_walk() on a less powerful machine.This is not a very useful statement.To make this into a benchmark that has relevance, I should run my tests on the same machine unless the goal is to have a laptop versus server shootout, in which case I should fix the functions I am comparing. Standardized initial data is also extremely important. Many functions (regular expressions being a prime example) exhibit extremely different performance characteristics as

the size and disposition of their operands change.To make a fair comparison, you need to use similar data sets for all the functions you want to compare. If you are using statically specified data for the test, it should be reused between functions. If you are using random data, you should use statistically equivalent data.

Building a Benchmarking Harness

Because you plan on benchmarking a lot of code, you should build a benchmarking harness to help automate the testing process. Having a benchmarking infrastructure not only helps to standardize benchmarks, it also makes it easy to incorporate benchmarks into a unit testing framework so that you can test the performance effects of library changes and PHP version changes.

The following are some of the features required in a usable harness:

nEase of use—Obviously, if the suite is hard to use, you will not use it. In particular, the benchmarking suite should not require you to modify your code in order to test it.

nLow or measurable overhead—The benchmarking harness itself takes resources to run.You need the ability to either minimize this overhead or (better yet) measure it so that you can remove it from the measured results.

nGood ability to select initial data—A benchmark is only as good as the data you use to run it against.The ability to be able to specify arbitrary input data is crucial.

nExtensibility—It would be nice to be able to extend or modify the statistics that are gathered.

PEAR’s Benchmarking Suite

PEAR has a built-in benchmarking suite, Benchmark_Iterate, that satisfies almost all the needs described in the preceding section. Benchmark_Iterate is suitable for many simple benchmarking tasks.

452 Chapter 19 Synthetic Benchmarks: Evaluating Code Blocks and Functions

Benchmark_Iterate works by running a function in a tight loop, recording execution times around each execution, and providing accessors for getting summary information on the results.

To start, you need to install the Benchmark libraries. Prior to PHP 4.3, the Benchmark class suite was packaged with PHP. After version 4.3, you need to either download the classes from http://pear.php.net or use the PEAR installer for a one-step installation:

# pear install Benchmark

To benchmark the performance of the function foo() over 1,000 iterations, you create a Benchmark_Iterate object, invoke the run method that specifies 1,000 iterations, and report the average runtime:

require ‘Benchmark/Iterate.php’; $benchmark = new Benchmark_Iterate; $benchmark->run(1000, foo); $result = $benchmark->get();

print “Mean execution time for foo: $result[mean]\n”;

A simple example of this is to use the suite to compare the speed of the built-in function max() with the PHP userspace implementation my_max().This is a simple example of how iterating over arrays with built-in functions can be significantly faster than using a userspace implementation.

The my_max() function will work identically to the built-in max() function, performing a linear search over its input array and keeping track of the largest element it has seen to date:

Function my_max(&$array) { $max = $array[0]; Foreach ($array as $el) { If($element > $max) {

$max = $element;

}

}

return $max;

}

For testing array functions, it is nice to have random test data.You can write a convenience function for generating such arrays and add it to the include test_data.inc so that you can reuse it later down the road:

Function random_array($size) {

For($I=0; $I<$size; $I++) {

$array[] = mt_rand();

}

return $array;

}

454 Chapter 19 Synthetic Benchmarks: Evaluating Code Blocks and Functions

function sort_max($array) {

return array_pop(asort($array));

}

Many sorting algorithms (including quicksort, which is the sorting algorithm used internally in all the PHP sorting functions) exhibit very different best-case and worstcase times. An unlucky “random” data choice can generate misleading results. One solution to this problem is to run benchmarks multiple times to eliminate edge cases. Of course, a robust benchmarking suite should handle that for you.

Benchmark_Iterate is slow.Very slow.This is because Benchmark_Iterate does much more work than is strictly necessary.The main loop of the run() method looks like this:

for ($i = 1; $i <= $iterations; $i++) { $this->setMarker(‘start_’ . $i);

call_user_func_array($function_name, $arguments); $this->setMarker(‘end_’ . $i);

}

setMarker(), in this case, is a method inherited from Benchmark_Timer, which basically just calls microtime() (which is a front end for the system call gettimeofday()). Accessing the system clock is not a particularly cheap operation in any language.You recognize this overhead here, and it is unnecessary. Unless you are interested in calculating more complex statistical metrics than the mean runtime, you do not need to record the runtime for every individual iteration.

Benchmark_Iterate returns wall clock timings. Sometimes you might like to collect more detailed information, such as augmenting the collected statistics with getrusage() statistics.

Calling userspace functions and class methods is not cheap. For extremely quickly executing functions, or for testing a code block that is not contained in a function, the act of calling a userspace wrapper for the timing functions may introduce overhead that obscures the result.

Building a Testing Harness

Because this book is decidedly not about reinventing the wheel, I presume that you would like to address as many issues as possible without writing a harness by hand. Fortunately, Benchmark_Iterate has a clean object-oriented design that makes extending its functionality relatively quick and easy.

First, you should look closer at the Benchmark_Timer and Benchmark_Iterate class diagram. Figure 19.1 is a stripped-down version of the UML diagram for Benchmark_Iterate and its parent classes. Attributes and methods not used by Benchmark_Iterate have been culled from the figure.

Benchmark_Timer

_construct(autoStart) _destruct() getOutput() getProfiling() display()

start()

stop()

timeElapsed(startMarker, endMarker)

Benchmark_Iterate

run(iterations, functionName) get()

Figure 19.1 A class diagram of Benchmark_Iterate that shows the major class methods you might want to override to build a custom testing harness.

As you can see in Figure 19.1, the main methods used in a benchmarking case are run() and get(). Under the hood, run() calls setMarker() immediately before and after every call to the function being benchmarked. setMarker() calls microtime to get the current time, with microsecond accuracy, and adds a marker to the markers array with that time.

The get() method uses the timeElapsed() method to track the time changes between markers. get() returns an array consisting of the execution time for every iteration, plus two additional keys: iterations, which is the number of times the function was executed, and mean, which is the mean execution time across all the iterations.

Adding Data Randomization on Every Iteration

Random data is a good thing.When you are authoring a function, you can seldom be sure of exactly what data is going to be passed to it. Being able to test random data minimizes the chance of hitting performance edge cases.The problem with the stock benchmark classes, though, is that they require you to specify your inputs before you enter the execution loop. If you generate random data once and pass it to your function, you are not testing a range of data at all, but just a single (albeit random) case.This does little but create confusing and inconsistent initial conditions.What you would like is to be able to randomize the data on every iteration.This way you could really test a wide distribution of potential inputs.

The ideal API would be if you could specify your own random data-generation function and have it called before each iteration. Here is an extension of Benchmark_Iterate that allows for randomized data:

456Chapter 19 Synthetic Benchmarks: Evaluating Code Blocks and Functions

require ‘Benchmark/Iterate.php’;F

class RandomBench extends Benchmark_Iterate {

function run_random() {

$function_name = array_shift($arguments); $argument_generator = array_shift($arguments); if (strstr($function_name, ‘::’)) {

$function_name = explode(‘::’, $function_name); $objectmethod = $function_name[1];

}

if (strstr($function_name, ‘->’)) { $function_name = explode(‘->’, $function_name); $objectname = $function_name[0];

global ${$objectname}; $objectmethod = $function_name[1];

for ($i = 1; $i <= $iterations; $i++) { $random_data = $argument_generator(); $this->setMarker(‘start_’ . $i);

call_user_method_array($function_name[1], ${$objectname}, $random_data); $this->setMarker(‘end_’ . $i);

}

return(0);

}

for ($i = 1; $i <= $iterations; $i++) { $random_data = $argument_generator(); $this->setMarker(‘start_’ . $i);

call_user_func_array($function_name, $random_data); $this->setMarker(‘end_’ . $i);

}

}

}

Removing Harness Overhead

To remove the overhead of the harness itself, you just need to measure the time it takes to benchmark nothing and deduct that from your averages.You can accomplish this by creating your own class that extends Benchmark_Iterate and replaces the run method with your own, which also calculates the overhead of doing a no-op (that is, no operation) between setting the start and stop timers. Here’s how it would look:

<?

require_once ‘Benchmark/Iterate.php’;

class MyBench extends Benchmark_Iterate {

public function run() {

parent::run($iterations, $function_name, $arguments); $oh = new Benchmark_Iterate;

for ($i = 1; $i <= $iterations; $i++) { $oh->setMarker(‘start_’ . $i); $oh->setMarker(‘end_’ . $i);

}

$oh_result = $oh->get(); $this->overhead = $oh_result[‘mean’] ; return(0);

}

public function get() { $result = parent::get();

$result[‘mean’] -= $this->overhead; $result[‘overhead’] = $this->overhead; return $result;

}

}

?>

You can use your new class by simply changing all the Benchmark_Iterate references in the sample test script:

require “test_data.inc”; require “MyBench.inc”;

foreach (array(10, 100, 1000) as $size) {

//Generate a test array. Benchmark_Iterate does not

//support generating random data for each iteration,

//so we need to be careful to use the same $test_array

//for testing both functions.

$test_array = random_array($size);

foreach (array(‘my_max’, ‘max’) as $func ) { $benchmark->run(1000, $func, $test_array); $result = $benchmark->get(); $summary[$func][$size] = $result[‘mean’] ;

$summary[‘my_max’][$size], $summary[‘max’][$size], $summary[‘my_max’][$size]/$summary[‘max’][$size]);

}