6Vectors

In economics, an ordered n-tuple often describes a bundle of commodities such that the ith value represents the quantity of the ith commodity. This leads to the concept of a vector which we will introduce next.

6.1 PRELIMINARIES

Definition 6.1 A vector a is an ordered n-tuple of real numbers a1, a2, . . . , an.

The numbers a1, a2, . . . , an are called the components (or coordinates) of vector a.

for a column vector. For the so-called transposed vector aT which is a row vector obtained by taking the column as a row, we write:

We use letters in bold face to denote vectors. We have defined above a vector a always as a column vector, and if we write this vector as a row vector, this is indicated by an upper-case T which stands for ‘transpose’. The convenience of using the superscript T will become clear when matrices are discussed in the next chapter. If vector a has n components, we say that vector a has dimension n or that a is an n-dimensional vector or n-vector. If not noted differently, we index components of a vector by subscripts while different vectors are indexed by superscripts.

Vectors 233

6.2 OPERATIONS ON VECTORS

We start with the operations of adding two vectors and multiplying a vector by a real number (scalar).

Definition 6.4 Let a, b Rn with a = (a1, a2, . . . , an)T and b = (b1, b2, . . . , bn)T. The sum of the two vectors a and b is the n-dimensional vector a + b obtained by adding each component of a to the corresponding component of b:

Definition 6.5 Let a Rn with a = (a1, a2, . . . , an)T and λ R. The product of number (or scalar) λ and vector a is the n-dimensional vector λa whose components are λ times the corresponding components of a:

The operation of multiplying a vector by a scalar is known as scalar multiplication.

Using Definitions 6.4 and 6.5, we can now define the difference of two vectors.

Definition 6.6 Let a, b Rn be n-dimensional vectors. Then the difference between the vectors a and b is defined by

a − b = a + (−1)b.

According to Definition 6.6, the difference vector is obtained by subtracting the components of vector b from the corresponding components of vector a. Notice that the sum and the difference of two vectors are only defined when both vectors a and b have the same dimension.

The sum and difference of the two vectors, as well as the scalar multiplication, are geometrically illustrated in Figure 6.2. The sum of the two vectors a and b is the vector obtained when adding vector b to the terminal point of vector a. The resulting vector from the origin to the terminal point of vector b gives the sum a + b.

Figure 6.2 Vector operations.

We see that multiplication of a vector by a positive scalar λ does not change the orientation of the vector, while multiplication by a negative scalar reverses the orientation of the vector. The difference a − b of two vectors means that we add vector b with opposite orientation to the terminal point of vector a, i.e. we add vector −b to vector a.

Next, we summarize some rules for the vector operations introduced above. Let a, b, c Rn and λ, µ R.

Vectors 235

Rules for vector addition and scalar multiplication

The validity of the above rules follows immediately from Definitions 6.4 and 6.5 and the validity of the commutative, associative and distributive laws for the set of real numbers.

Definition 6.7 The scalar product of two n-dimensional vectors a = (a1, a2, . . . , an)T and b = (b1, b2, . . . , bn)T is defined as follows:

The scalar product is also known as the inner product. Note also that the scalar product of two vectors is not a vector, but a number (i.e. a scalar) and that aT · b is defined only if a and b are both of the same dimension. In order to guarantee consistence with operations in later chapters, we define the scalar product in such a way that the first vector is written as a row vector aT and the second vector is written as a column vector b. The commutative and distributive laws are valid for the scalar product, i.e.

aT · b = bT · a and aT · (b + c) = aT · b + aT · c

for a, b, c Rn. It is worth noting that the associative law does not necessarily hold for the scalar product, i.e. in general we have

a · (bT · c) = (aT · b) · c.

Example 6.3 Assume that a firm produces three products with the quantities x1 = 30, x2 = 40 and x3 = 10, where xi denotes the quantity of product i. Moreover, the cost of production is 20 EUR per unit of product 1, 15 EUR per unit of product 2 and 40 EUR per unit of product 3. Let c = (c1, c2, c3)T be the cost vector, where the ith component describes the cost per unit of product i and x = (x1, x2, x3)T. The total cost of production is obtained

= 600 + 600 + 400 = 1, 600.

We have found that the total cost of production for the three products is 1,600 EUR.

Definition 6.8 Let a R with a = (a1, a2, . . . , an)T. The (Euclidean) length (or norm) of vector a, denoted by |a|, is defined as

|a| = a21 + a22 + . . . + a2n.

A vector with length one is called a unit vector (remember that we have already introduced the specific unit vectors e1, e2, . . . , en which obviously have length one). Each non-zero n-dimensional vector a can be written as the product of its length |a| and an n-dimensional unit vector e(a) pointing in the same direction as the vector a itself, i.e.

a = |a| · e(a).

be given. We are looking for a unit vector pointing in the same direction as vector a. Using

|a − b| = (a1 − b1)2 + (a2 − b2)2 + . . . + (an − bn)2.

Vectors 237

The distance between the two-dimensional vectors a and b is illustrated in Figure 6.3. It corresponds to the length of the vector connecting the terminal points of vectors a and b.

Figure 6.3 Distance between the vectors a and b.

be given. The distance between both vectors is given by

|a − b| = [3 − (−1)]2 + (2 − 1)2 + (−3 − 5)2 = √81 = 9.

Next, we present some further rules for the scalar product of two vectors and the length of a vector. Let a, b Rn and λ R.

Further rules for the scalar product and the length

(3)|λa| = |λ| · |a|;

(4)|a + b| ≤ |a| + |b|;

(5)aT · b = |a| · |b| · cos(a, b);

(6)|aT · b| ≤ |a| · |b| (Cauchy–Schwarz inequality).

238 Vectors

In rule (5), cos(a, b) denotes the cosine value of the angle between vectors a and b. We illustrate the Cauchy–Schwarz inequality by the following example.

The scalar product of vectors a and b is obtained as

aT · b = 2 · 5 + (−1) · (−4) + 3 · (−1) = 10 + 4 − 3 = 11.

Cauchy–Schwarz’s inequality says that the absolute value of the scalar product of two vectors a and b is never greater than the product of the lengths of both vectors. For the example, this inequality turns into

|aT · b| = 11 ≤ |a| · |b| = √14 · √42 ≈ 24.2487.

Example 6.7 Using rule (5) above, which can also be considered as an alternative equivalent definition of the scalar product, and the definition of the scalar product, according to Definition 6.7, one can easily determine the angle between two vectors a and b of the same dimension. Let

Then we obtain

We have to find the smallest positive argument of the cosine function which gives the value 0.80178. Therefore, the angle between vectors a and b is approximately equal to 36.7◦.

Next, we consider orthogonal vectors. Consider the triangle given in Figure 6.4 formed by the three two-dimensional vectors a, b and a − b. Denote the angle between vectors a and b by γ . From the Pythagorean theorem we know that angle γ is equal to 90◦ if and only if the

Vectors 239

sum of the squared lengths of vectors a and b is equal to the squared length of vector a − b. Thus, we have:

γ = 90◦ |a|2 + |b|2 = |a − b|2

aT · a + bT · b = (a − b)T · (a − b)

aT · a + bT · b = aT · a − aT · b − bT · a + bT · b

aT · b = 0.

The latter equality has been obtained since aT · b = bT · a. For two-dimensional vectors we have seen that the angle between them is equal to 90o if and only if the scalar product is equal to zero. We say in this case that vectors a and b are orthogonal (or perpendicular) and write a b. The above considerations can be generalized to the n-dimensional case and we define orthogonality accordingly:

a b aT · b = 0,

where a, b Rn.

Figure 6.4 Triangle formed by vectors a, b and a – b.

Example 6.8 The three-dimensional vectors

are orthogonal since

aT · b = 3 · 4 + (−1) · 6 + 2 · (−3) = 0.

Since two vectors are equal if they coincide in all components, the above equation (6.2) represents n linear equations with the variables λ1, λ2, . . . , λm. In Chapter 8, we deal with the solution of such systems in detail.

Remark From Definition 6.10 we obtain the following equivalent characterization of linearly dependent and independent vectors.

(1)A set of m vectors a1, a2, . . . , am Rn is linearly dependent if and only if at least one of the vectors can be written as a linear combination of the others.

(2)A set of m vectors a1, a2, . . . , am Rn is linearly independent if and only if none of the vectors can be written as a linear combination of the others.

In this case, we have a2 = −3a1 which can be written as

3a1 + 1a2 = 0.

We can conclude that equation (6.2) holds with λ1 = 3 and λ2 = 1, and thus vectors a1 and a2 are linearly dependent (see Figure 6.6).

In this case, equation λ1a1 + λ2b2 = 0 reduces to

3λ1 − λ2 = 0

λ1 + 2λ2 = 0.

242 Vectors

This is a system of two linear equations with two variables λ1 and λ2 which can easily be solved. Multiplying the first equation by two and adding it to the second equation, we obtain λ1 = 0 and then λ2 = 0 as the only solution of this system. Therefore, both vectors a1 and b2 are linearly independent (see Figure 6.6).

Figure 6.6 Linearly dependent and independent vectors.

The above examples illustrate that in the case of two vectors of the two-dimensional Euclidean space R2, we can easily decide whether they are linearly dependent or independent. Two two-dimensional vectors are linearly dependent if and only if one vector can be written as a multiple of the other vector, i.e.

(see Figure 6.6). On the other hand, in the two-dimensional space every three vectors are linearly dependent. This is also illustrated in Figure 6.6. Vector c can be written as a linear combination of the linearly independent vectors a1 and b2, i.e.

c = λ1a1 + λ2b2,

from which we obtain

By Definition 6.10, these vectors are linearly dependent since e.g. the scalar of vector c in representation (6.3) is different from zero.

Considering 3-vectors, three vectors are linearly dependent if one of them can be written as a linear combination of the other two vectors which means that the third vector belongs to the plane spanned by the other two vectors. If the three vectors do not belong to the same plane,

Vectors 243

these vectors are linearly independent. Four vectors in the three-dimensional Euclidean space are always linearly dependent. In general, we can say that in the n-dimensional Euclidean space Rn, there are no more than n linearly independent vectors.

Example 6.11 Let us consider the three vectors

and investigate whether they are linearly dependent or independent. Using Definition 6.10, we obtain

Considering the third component of the above vectors, we obtain λ3 = 0. Substituting λ3 = 0 into the second component, we get from λ2 + 2λ3 = 0 the only solution λ2 = 0 and considering finally the first component, we obtain from 2λ1 + λ2 + 3λ3 = 0 the only solution λ1 = 0. Since vector λT = (λ1, λ2, λ3) = (0, 0, 0) is the only solution, the above three vectors are linearly independent.

Example 6.12 The set {e1, e2, . . . , en} of n-dimensional unit vectors in the space Rn obviously constitutes a set of linearly independent vectors, and any n-dimensional vector aT = (a1, a2, . . . , an) can be immediately written as a linear combination of these unit vectors:

In this case, the scalars of the linear combination of the unit vectors are simply the components of vector a.

244 Vectors

6.4 VECTOR SPACES

We have discussed several properties of vector operations so far. In this section, we introduce the notion of a vector space. This is a set of elements (not necessarily only vectors of real numbers) which satisfy certain rules listed in the following definition.

Definition 6.11 Given a set V = {a, b, c, . . . } of vectors (or other mathematical objects), for which an addition and a scalar multiplication are defined, suppose that the following properties hold (λ, µ R):

(1)a + b = b + a;

(2)(a + b) + c = a + (b + c);

(3)there exists a vector 0 V such that for all a V the equation

a + 0 = a holds (0 is the zero or neutral element with respect to addition);

(4)for each a V there exists a uniquely determined element x V such that a + x = 0 (x = −a is the inverse element of a with respect to addition);

(5)(λµ)a = λ(µa);

(6)1 · a = a;

(7)λ(a + b) = λa + λb;

(8)(λ + µ)a = λa + µa.

If for any a, b V , inclusion a + b V and for any λ R inclusion λa V hold, then V is called a linear space or vector space.

As mentioned before, the elements of a vector space do not necessarily need to be vectors since other ‘mathematical objects’ may also obey the above rules (1) to (8). Next, we give some examples of vector spaces satisfying the rules listed in Definition 6.11, where in each case an addition and a scalar multiplication is defined in the usual way.

Examples of vector spaces

(1)the n-dimensional space Rn;

(2)the set of all n-vectors a Rn that are orthogonal to some fixed n-vector b Rn;

(3)the set of all sequences {an};

(4)the set C[a, b] of all continuous functions on the closed interval [a, b];

(5)the set of all polynomials

Pn(x) = anxn + an−1xn−1 + · · · + a1x + a0

of a degree of at most n.

To prove this, one has to verify the validity of the rules (1) to (8) given in Definition 6.11 and to show that the sum of any two elements as well as multiplication by a scalar again gives an

Vectors 245

element of this space. For instance, consider the set of all polynomials of degree n. The sum of two such polynomials

Pn1(x) = anxn + an−1xn−1 + · · · + a1x + a0

Pn2(x) = bnxn + bn−1xn−1 + · · · + b1x + b0

gives

Pn1(x) + Pn2(x) = (an + bn)xn + (an−1 + bn−1)xn−1 + · · · + (a1 + b1)x + (a0 + b0),

i.e. the sum of these polynomials is again a polynomial of degree n. By multiplying a polynomial Pn1 of degree n by a real number λ, we obtain again a polynomial of degree n:

λPn1 = (λan)xn + (λan−1)xn−1 + · · · + (λa1)x + λa0.

Basis of a vector space; change of basis

Next, we introduce the notion of the basis of a vector space.

Definition 6.12 A set B = {b1, b2, . . . , bn} of linearly independent vectors of a vector space V is called a basis of V if any vector a V can be written as a linear combination

a = λ1b1 + λ2b2 + · · · + λnbn

of the basis vectors b1, b2, . . . , bn. The number n = |B| of vectors contained in the basis gives the dimension of vector space V .

For an n-dimensional vector space V , we also write: dim V = n. Obviously, the set Bc = {e1, e2, . . . , en} of unit vectors constitutes a basis of the n-dimensional Euclidean space Rn (see also Example 6.12). This basis Bc is also called the canonical basis. The notion of a basis of a vector space is a fundamental concept in linear algebra which we will need again later when discussing various algorithms.

Remark (1) An equivalent definition of a basis is that a maximal set of linearly independent vectors of a vector space V constitutes a basis and therefore, the dimension of a vector space is given by the maximal number of linearly independent vectors.

(2)We say that V is spanned by the vectors b1, b2, . . . , bn of B since any vector of the vector space V can be ‘generated’ by means of the basis vectors.

(3)The dimension of a vector space is not necessarily equal to the number of components of its vectors. If a set of n-dimensional vectors is given, they can contain at most n linearly independent vectors. However, any n linearly independent vectors of an n-dimensional vector space constitute a basis.

Next, we establish whether an arbitrary vector of a vector space can be written as a linear combination of the basis vectors in a unique way.

246 Vectors

THEOREM 6.1 Let B = {b1, b2, . . . , bn} be a basis of an n-dimensional vector space V . Then any vector c V can be uniquely written as a linear combination of the basis vectors from set B.

PROOF We prove the theorem indirectly. Assume there exist two different linear combinations of the given basis vectors from B which are equal to vector c:

i.e. there exists an index i with 1 ≤ i ≤ n such that λi = µi . By subtracting equation (6.5) from equation (6.4), we get

0 = (λ1 − µ1)b1 + (λ2 − µ2)b2 + · · · + (λn − µn)bn

Since the basis vectors b1, b2, . . . , bn are linearly independent by Definition 6.12, we must have

i.e. we have obtained a contradiction. Thus, any vector of V can be uniquely written as a linear combination of the given basis vectors.

While the dimension of a vector space is uniquely determined, the basis is not uniquely determined. This leads to the question of whether we can replace a particular vector in the basis by some other vector not contained in the basis such that again a basis is obtained. The following theorem shows that there is an easy way to answer this question, and from the proof of the following theorem we derive an algorithm for the replacement of a vector in the basis by some other vector (provided this is possible). The resulting procedure is a basic part of some algorithms for the solution of systems of linear equations or linear inequalities which we discuss in Chapter 8.

ak = λ1b1 + λ2b2 + · · · + λk bk + · · · + λnbn

with λk = 0. Then the set B = {b1, b2, . . . , bk−1, ak , bk+1, . . . , bn} is also a basis of V , i.e. vector bk contained in basis B can be replaced by the vector ak to obtain another basis.

Vectors 247

By substituting the linear combination ak = λ1b1 + λ2b2 + · · · + λnbn

into equation (6.6), we obtain:

(µ1 + µk λ1)b1 + (µ2 + µk λ2)b2 + · · · + (µk−1 + µk λk−1)bk−1

+ (µk λk )bk + (µk+1 + µk λk+1)bk+1 + · · · + (µn + µk λn)bn = 0.

Since the vectors of the set B = {b1, b2, . . . , bn} constitute a basis, they are linearly independent and all the scalars in the above linear combination must be equal to zero, i.e. we get

(µi + µk λi ) = 0 for i = 1, 2, . . . , n, i = k and µk λk = 0.

Since by assumption λk = 0, we get first µk = 0 and then, using the latter result, µi = 0

for all i with 1 ≤ i ≤ n, i = k, i.e. all the scalars in the linear combination (6.6) must be equal to zero. Hence, the vectors of set B = {b1, b2, . . . , bk−1, ak , bk+1, . . . , bn} are linearly

independent by Definition 6.10 and they constitute a basis.

We know that any basis of a certain vector space consists of the same number of vectors. So, if we want to remove one vector from the current basis, we have to add exactly one other vector to the remaining ones such that the resulting set of vectors is again linearly independent.

We now look for a procedure for performing such an interchange of two vectors described in the proof of Theorem 6.2. To this end, assume that B = {b1, . . . , bk−1, bk , bk+1, . . . , bn} is a basis and B = {b1, . . . , bk−1, ak , bk+1, . . . , bn} is another basis, where vector bk has

been replaced by vector ak . According to Theorem 6.2, we must have λk = 0 in the linear combination of vector

of the vectors of basis B since otherwise a replacement of vector bk by vector ak is not possible. Let us consider an arbitrary vector c and its linear combinations of the basis vectors of bases B and B , respectively:

c= β1b1 + · · · + βk−1bk−1 + βk (λ1b1 + · · · + λk bk + · · · + λnbn)

+βk+1bk+1 + · · · + βnbn

=(β1 + βk λ1) b1 + · · · + (βk−1 + βk λk−1) bk−1 + βk λk bk

+(βk+1 + βk λk+1) bk+1 + · · · + (βn + βk λn) bn.

248 Vectors

Comparing now the scalars of the vectors from the basis B = {b1, b2, . . . , bn} in both representations of vector c (i.e in representation (6.8) and the last equality above) first for k and then for all remaining i = k, we first obtain

it follows by means of equality (6.10)

In order to transform the linear combination of vector c of the basis vectors of B into the linear combination of the basis vectors of B , we can therefore use the scheme given in Table 6.1. The last column describes the operation that has to be performed in order to get the elements of the current row: e.g. in row n + 2, the notation ‘row 2 – (λ2/λk ) row k’ means that we have to take the corresponding element of row 2 (i.e. α2 in the c column), and then we have to subtract λ2/λk times the corresponding element of row k (i.e. αk in the c column) which gives the new element β2 in row n + 2 and the c column. The transformation formula above for all rows different from k is also called the rectangle formula since exactly four elements forming a rectangle are required to determine the corresponding new element. This is illustrated in the scheme above for the determination of element β2, where the corresponding four elements in row 2 and row k are underlined.

Table 6.1 Tableau for Steinitz’s procedure