Figure 1

(b) In Example 1(c) we saw that n² < 2ⁿ for n > 4. In fact, given any

positive constant m, we have

n^m < 2ⁿ for all sufficiently large n.

To see this, first note that by (a), and

so

.

Now for n  4m², and so

log₂ n^m < n for n  4m².

Hence we have

n^m < 2ⁿ for n  4m².

(c) Let m be a positive integer. From part (b) we have

log₂ n^m < n for n  4m².

This inequality holds even if n is not an integer, so we can replace n by

to obtain log₂ ( )^m < for  4m², i.e., |

log₂ n< for n  (4m²)^m.

Thus log₂ n < for sufficiently large n. ■

Roughly speaking, Example 2(b) shows that 2ⁿ grows faster than any power of n, and Example 2(c) tells us that log; n grows more slowly than any root of n. Before making these ideas precise, we observe some more inequalities.

example 3 (a) We have

2ⁿ < n! < nⁿ for n  4. I

For n = 4 these inequalities are evident: 16 < 24 < 256. For n > 4 we

have n! == (4!)  5  6  (n-1)n. The first factor 4! exceeds 2⁴ and the

remaining n – 4 factors each exceed 2. So n! > 2⁴  2^n-4= 2ⁿ.

The inequality n! > nⁿ is clear since n! is a product of integers all but

one of which less than n.

(b) Let’s be greedy and claim that 40ⁿ < n! for sufficiently large n.

This will be a little trickier than 2ⁿ < n! to verify. Observe that for

n > 80 we can write

provided that 2ⁿ > 80⁸⁰ or n > log₂ (80⁸⁰) = 80  log₂ 80  505.8. This

was a "crude" argument in the sense that we lost a lot [80! in fact]

when we wrote n! > n(n – 1)    81. If we had wanted better

information about where n! actually overtakes 40ⁿ, we could have

made more careful esti­mates. Our work was enough, though, to

show that 40ⁿ < n! for all sufficiently large n. ■

To be more precise about what we mean when we say "... grows like ... for large n,” we need develop some new notation, called the big-oh notation. Our main use for the notation will be to describe algorithm running times.

Given two sequences, say s and t, with nonnegative real values, the statement “s(n) = O(t(n))” [read “s(n) is big oh of t(n)” is intended to mean that as n gets large than the values of s are no larger than the values of some constant multiple of t.

EXAMPLE 4 (a) Example 1(a) tells us that = O(n), n =O(n²), etc. Example

1(b) tells us that n =O(2ⁿ), and Example 1(c) tells us that n² =O(2ⁿ). In

fact, n^m = O(2ⁿ) for each m by Example 2(b). Example 3(a) shows

that 2ⁿ = O(n!) and that n! = O(nⁿ).

(b) The dominant term of 6n⁴ + 20n² + 2000 is 6n⁴, since for large

n the value of n⁴ is much greater than n² or the constant 2000. We will

write this observation as

6n⁴ + 20n² + 2000 = 0(n⁴),

to indicate that the expression on the left “grows no worse than a mul-

ti­ple of n⁴.” It actually grows a little bit faster than 6n⁴. What matters,

though, is that for large enough n [in this case, n  8 is big enough]

20n² + 2000 < n⁴, so 6n⁴ + 20n² + 2000 < 7n⁴; i.e., the total quantity is

no larger than some fixed multiple of n⁴. We could also say, of

course, that it grows no faster than a multiple of n⁵, but that’s not as

useful a piece of information as the one we’ve given. ■

Here is the precise definition. Let  and g be sequences of real numbers. We write

f(n) = O(g(n))

in case there is some positive constant C such that

|(n)|  C  |g(n)| for all sufficiently large values of n.

Here we are willing to have the inequality fail for a few small values of n, perhaps because (n) or g(n) fails to be defined for those values. All we really care about are the large values of n. In practice, (n) will represent some sequence of current interest [such as an upper bound on the time some algorithm will run], while g(n) will be some simple sequence, like n, log₂ n, n³, etc., whose growth we understand. The next theorem lists some of what we have learned in Examples 1 to 3.

T

Here is the hierarchy of several familiar sequences in the sense that each sequence is big-oh of any sequence to its right:

heorem 1

The constant sequence 1 in the theorem is defined by l(n) = 1 for all n. It doesn’t grow at all.

EXAMPLE 5 (a) Suppose that g(n) = n for all n  N. The statement f(n) = O(n)

means that |f(n)| is bounded by a constant multiple of n, i.e., that there is

some C > 0 so that |f(n)|  C_n for all large enough n  N.

(b) Suppose that g(n) = 1 for all n  N. We say that f(n) is O(1) if

there is a constant C such that |f(n)|  C for all large n, that is, in case

the values of | f | are bounded above by some constant.

(c) The sequence s defined by s_n = 3n² + 15n satisfies s_n = O(n²),

because n  n² for n  1, and thus | s_n|  3n² + 15n² = 18n² for all large

enough n.

(d) The sequence t given by t_n = 3n² + (- 1)ⁿ15n also satisfies

t_n = O(n²). As in part (c), we have | t_n|  3n² + 15n²  18n² for n  1.

(e) We can generalize the examples in parts (c) and (d). If f(n) =

is a polynomial in n of degree m with a_m  0,

then |a_k n^k |  |a_k|  n^m for k = 0,1, ..., m – 1, so

and hence f(n) = O(n^m). The first inequality holds because

for any finite sequence x₁, x₂, …, x_i in R. ■

Because of Example 5(e) a sequence f(n) is said to have polynomial growth if f(n) = O(n^m) for some positive integer m. From a theoretical point of view, algorithms whose time behaviors have polynomial growth are regarded as manageable. They certainly are, compared to those with time behavior O(2ⁿ), say. In practice, efficient time behavior like O(n) or O(n  log₂ n) is most desirable. The next example concerns two sequences that arise in estimating the time behavior of algorithms.

example 6 (a) Let for n  1. We claim that

s_n = O(log₂ n).

Observe that

etc. In general < k + 1.

Now consider any integer n > 2.

Trap n between powers of 2, say 2^k < n  2^k+1. Since k < log₂ n, we

have

If n  4, then log₂ n  2, and so

s_n < 2 log₂ n for n  4. S |

Hence s_n = O(log₂ n).

(b) If for n  1, then we have

t_n = n  s_n, so for n  4, by (a). Therefore, we have

t_n = O(n  log₂ n). ■

The next theorem lists some general facts about big-oh notation. Remember that writing f(n) = O(g(n) just signifies that f(n) is some sequence that is O(g(n)).

T

(a) If f{n) = O(g(n)) and if c is a constant, then c  (n) = O(g{n)).

(b) If f(n) = O(g{n)) and h(n) = O(g(n)), then f(n) + h(n) = O(g(n)).

(c) If f(n) = O(a(n)) and g(n) = O(b(n)), then we have f(n)  g(n) = O(a(n)b(n)).

(d) If a(n) = O(b(n)) and b(n) = O(c(n)), then a(n) = O(c(n)).

heorem 2

Proof. Parts (a) and (c) are left to Exercise 13.

(b) If f(n) = O(g(n)) and h(n) = O(g(n)), there exist positive constants C and D such that

|f(n)|  C  |g(n)| for all sufficiently large n,

and

|h(n)|  D  |g(n)| for all sufficiently large n.

Since |x + y|  |x| + |y| for x, y  R we conclude that

|f(n) + h(n)|  |f(n)| + |h(n)|  (C + D)  |g{n)|

for all sufficiently large n. Consequently, we have f(n) + h{n) = O(g(n)).

(d) If a(n) = O(b(n)) and b(n) == O(c(n)), there exist positive constants C and D such that

|a(n)|  C  |b(n)| and |b(n)|  D  |c(n)|

for all sufficiently large n. Thus

|a(n)|  C  |b(n)|  C  D  |c(n)| for all sufficiently large n,

and hence a(n) = O(c(n)). ■

The general principles in Theorem 2 would have shortened some arguments in Examples 5 and 6. For instance, we know that n^k = O(n^m) if k  m, so Theorem 2 gives

We are able to use (b) here because the number of summands, m + 1, does not depend on n.

In Example 6(a) we were really done when we got s_n < log₂ n + 2, since

log₂ n + 2 = O(log₂ n) + O(1) = O(log₂ n).

Example 6(b) is immediate, because

t_n = n  s_n = O(n  log₂ n)

by Theorem 2(c).

Looking at Example 4(b) and Example 5, we see that it might be useful to describe a sequence by giving its dominant term plus terms that contribute less for large n. Thus we might write

6n⁴ + 20n² + 1000 = 6n⁴ + O(n²),

where the big-oh term here stands for the expression 20n² + 1000, which we know satisfies the condition 20n² + 1000 = O(n²). This usage of the big-oh notation in expressions such as f(n) + O(g(n)) is slightly different in spirit from the usage in equations like f(n) = O(g(n)), but both are con­sistent with an interpretation that O(g(n)) represents “a sequence whose values are bounded by some constant multiple of g{n) for all large enough n.” It is this meaning that we shall use from now on.

Just as we made sense out of a(n) + O(b(n)), we can write a(n)  O(b(n)) to mean “a(n)  f(n) where f(n) = O(b(n)).” With this interpretation we can write “equations” like those in the next theorem.

T

For any sequences a(n) and b(n), we have

(a) O(a(n)) + O(b(n)) == O(max{|a(n)|, |b(n)|}).

(b) O(a(n))O(b(n))=O(a(n)-b(n)).

eorem 3

These statements simply say: If f(n) = O(a(n)) and a(n) = O(b(n)) then f(n) + g(n) = O(max{|a(n)|, |b(n)|}) and f(n)  g(n) = O(a(n)  b(n)). For instance, we have O(n³) + O(n⁴) = O(n⁴), since max{n³, n⁴} = n⁴, and O(n³) O(n⁴) = O(n⁷). Unlike true equations, statements like these really don’t mean much if we read them from right to left.

Proof of Theorem 3. (a) Let f(n) = O(a(n)) and g(n) = O(b(n)). Then there exist positive constants C and D such that

|f(n)|  C|a(n)| and |g(n)|  D  |b(n)| for sufficiently large n.

Then we have

|f(n) + g(n)|  |f(n)| + |g(n)|  C  |a(n)| + D  |b(n)|

 C  max{|a(n)|, |b(n)|} + D  max{|a(n)|, |b(n)|}

= (C + D)  max{|a(n)|,|b(n)|}

for sufficiently large n. Therefore, f(n) + g(n) = O(max{|a(n)|, |b(n)|}). Part (b) is immediate from Theorem 2(c). ■

EXAMPLE 7 (a) Since i + 13n = O(n²) and (n + I)³ = O(n³) [think about this],

(n² + 13n) + (n + 1)³ = O(n³) by Theorem 3(a), and (n² + 13n)(n + I)³ =

O(n⁵) by Theorem 3(b).

(b) If a(n) = O(n⁴) and b(n) = O(log₂ n), then we have a(n)-b(n) =

O(n⁴  log₂ n), a(n)² = O(n⁸) and b(n)² = O(log₂ n). Note that

denotes (log₂ n)². ■

(c) As a consequence of Example 5(e), we can also write

if a_m  0. ■

EXERCISES 1.6

1. For each sequence below find the smallest number k such that f(n) = 0(n^k).

(a) f(n) = 13n² + 4n - 73 (b) f(n) = (n² + l)(2n⁴ + 3n - 8)

(c) f(n) = (n³ + 3n - 1)⁴ (d)

2. Repeat Exercise 1 for

(a) f(n) = (n² - 1)⁷ (b) f(n) =

(c) f(n) = (d) f(n) =(n²+n+1)² •(n³ + 5)

3. For each sequence below give the sequence a(n) in the hierarchy of Theorem 1 such that f(n) = O(a(n)) and such that a(n) is as far to the left as possible in the hierarchy.

(a) f(n) = 3ⁿ (b) f(n) = n³  log₂ n (c) f(n) =

4. Repeat Exercise 3 for

(a) f(n) = n + 3  log₂ n (b) f(n) = (n  log₂ n + 1)²

(c) f(n) =(n+1)!

5. State whether each of the following is true or false. In each case give a reason for your answer.

(a) 2^{n+ 1} = O(2ⁿ) (b) (n + I)² = O(n²)

(c) 2^{n+ 1} = O(2ⁿ) (d) (200n)² = O(n²)

6. Repeat Exercise 5 for

(a) log n = O( ) (b) log₂ (n⁷³) = O(log₂ n)

(c) log₂ nⁿ = O(log₂ n) (d) ( + 1)⁴ = O(n²)

7. True or false. In each case give a reason.

(a) 40ⁿ = O(2ⁿ) (b) (40n)² = O(n²)

(c) (2n)! = O(n!) (d) (n + 1)⁴⁰ = O(n⁴⁰)

8. Let A be a positive constant. Show that Aⁿ < n! for sufficiently large n. Hint:

Analyze Example 3(b).

9. (a) For n  P let , so that for n  4.

Show that . Hint: Show that for k  2, so that

(b) Show that t_n = O(n²) where t_n = = 1 + 2 + … + n.

10. (a) Show that if s_n = , then s_n = O(n³).

(b) Fix m in P and define t_n = . Show that t_n = O(n^m+1).

11. Show that if f(n) = 3n⁴ + O(n) and g(n) = 2n³ + O(n), then

(a) f(n) + g(n) = 3n⁴ + O(n³) (b) f(n)  O(n) = 6n⁷ + O(n⁵)

12. Show that

(a) (5n³ + O(n²)  (3n⁴+ O(n³)) = 15n⁷ + O(n⁶)

(b) (5n³ + O(n))  (3n⁴ + O(n²)) = 15n⁷ + O(n⁵)

13. Explain why parts (a) and (c) of Theorem 2 are true.

14. (a) Prove directly from Theorem 2 that if a(n) = O(c(n)) and b(n) = 0(c(n)) then

O(a(n)) + O(b(n)) = O(c(n)).

(b) Observe that this gives another proof of Theorem 3(a).

15. This exercise shows that one must be careful using division in big-oh calculations.

(a) Let a(n) = n⁵ and b(n) = n. Observe that a(n) = O(n⁵) and b(n) = O(n²) but that

a(n)b(n) is not O(n³).

(b) Give examples of sequences a(n) and b(n) such that a(n) = O(n⁶) and b(n) =

O(n²) but a(n)/b(n) is not O(n⁴).

16. Show that log₁₀ n = O(log₂ n).

17. For each n  P let digit(n) be the number of digits in the decimal expansion of n.

(a) Show that 10^digit(n)-1  n < 10^digit(n).

(b) Show that log₁₀ n is O(digit(n)).

(c) Show that digit(n) is O(log₁₀ n).

(d) Let digit2(n) be the number of digits in the binary expansion of n. How are

O(digit(n)) and O(digit2(n)) related?

CHAPTER HIGHLIGHTS

T o check your understanding of the material in this chapter, we recom­mend that you consider each item listed below and:

(a) Satisfy yourself that you can define each concept and notation and can describe each method.

(b) Give at least one reason why the item was included in this chapter.

(c) Think of at least one example of each concept and at least one situation in which each fact or method would be useful.

This chapter is introductory and contains a relatively large number of fundamental definitions and notations. It is important to be comfortable with this material now, since the rest of the book is built on it.

CONCEPTS

set [undefined]

member = element, subset

equal, disjoint

set operations

universe, complement

Venn diagram

ordered pair, product of sets

alphabet, language, word, length of word

function = map == mapping

domain, codomain

image of x, image of f = Im(f)

graph of a function

one-to-one, onto, one-to-one correspondence

composition of functions

inverse function

restriction, f(A)

pre-image, f ^(B)

sequence, finite sequence

“big-oh” notation

EXAMPLES AND NOTATION

N, P, Z, Q, R

, , { : }, , 

 = { } = empty set

[a, b], (a, b), [a, b), (a, b] notation for intervals

P(S), , , A\B, A  B, A^c

(s, t) notation for ordered pairs, S  T, Sⁿ

|S| = number of elements in the set S

 notation for sums,  notation for products

n! for n factorial

^,  = empty word

special functions l_S, , log_b

f(n) = O(g(n)), f(n) = g(n) + O(h(n))

FACTS

Basic laws of set algebra [Table 1 of 1.2].

Composition of functions is associative.

A function is invertible if and only if it is a one-to-one correspondence.

Comparative growth rates of common sequences [Figure 2 of 1.5, i Theorem 1 of 1.6].

METHODS

Use of Venn diagrams.

Reasoning from definitions and previously established facts.

1 We will use ■ to signify the end of an example or proof.

1 We thank our colleague Richard M. Koch for supplying the larger values in this table. He used Mathematica.