Amdahl's Law Defined

A program (or algorithm) which can be parallelized can be split up into two parts:

• A part which cannot be parallelized

• A part which can be parallelized

Imagine a program that processes files from disk. A small part of that program may scan the directory and create a list of files internally in memory. After that, each file is passed to a separate thread for processing. The part that scans the directory and creates the file list cannot be parallelized, but processing the files can.

The total time taken to execute the program in serial (not in parallel) is called T. The time T includes the time of both the non-parallelizable and parallelizable parts. The non-parallelizable part is called B. The parallizable part is referred to as T - B. The following list sums up these definitions:

• T = Total time of serial execution

• B = Total time of non-parallizable part

• T - B = Total time of parallizable part (when executed serially, not in parallel)

From this follows that:

T = B + (T-B)

It may look a bit strange at first that the parallelizable part of the program does not have its own symbol in the equation. However, since the parallelizable part of the equation can be expressed using the total time T and B (the non-parallelizable part), the equation has actually been reduced conceptually, meaning that it contains less different variables in this form.

It is the parallelizable part, T - B, that can be sped up by executing it in parallel. How much it can be sped up depends on how many threads or CPUs you apply to execute it. The number of threads or CPUs is called N. The fastest the the parallelizable part can be executed is thus:

(T - B) / N

Another way to write this is:

(1/N) * (T - B)

Wikipedia uses this version in case you read about Amdahl's law there.

According to Amdahl's law, the total execution time of the program when the parallelizable part is executed using N threads or CPUs is thus:

T(N) = B + (T - B) / N

T(N) means total execution with with a parallelization factor of N. Thus, T could be written T(1) , meaning the total execution time with a parallelization factor of 1. Using T(1) instead of T, Amdahl's law looks like this:

T(N) = B + ( T(1) - B ) / N

It still means the same though.

A Calculation Example

To better understand Amdahl's law, let's go through a calculation example. The total time to execute a program is set to 1. The non-parallelizable part of the programs is 40% which out of a total time of 1 is equal to 0.4 . The parallelizable part is thus equal to 1 - 0.4 = 0.6 .

The execution time of the program with a parallelization factor of 2 (2 threads or CPUs executing the parallelizable part, so N is 2) would be:

T(2) = 0.4 + ( 1 - 0.4 ) / 2

= 0.4 + 0.6 / 2

= 0.4 + 0.3

= 0.7

Making the same calculation with a parallelization factor of 5 instead of 2 would look like this:

T(5) = 0.4 + ( 1 - 0.4 ) / 5

= 0.4 + 0.6 / 5

= 0.4 + 0.12

= 0.52

Amdahl's Law Illustrated

To better understand Amdahl's law I will try to illustrate how the law is derived.

First of all, a program can be broken up into a non-parallelizable part B, and a parallelizable part 1-B, as illustrated by this diagram:

The line with the delimiters on at the top is the total time T(1).

Here you see the execution time with a parallelization factor of 2:

Here you see the execution time with a parallelization factor of 3: