# Tensors 2

## Tensor as an element of tensor product of vector spaces

Before presenting another way of definition a tensor, we define a notation. A linear map and a bilinear form are respectively written as a linear combination of and . Any of these (for any the dimensions of the corrresponding spaces) can be considered as one new object and denoted as for example. and . The writing of the basis vectors and/or basis covectors adjacent to each other is usually denoted by and . This notation is referred to as tensor product of (basis) vectors. A general definition will be presented later. Using this notation, for now, we can write a linear map and a bilinear form as,

This notation can be extended to be used with any finite linear combination of tensor products of basis vectors and/or covectors where the combination coefficients takes indices following the index level convension. For example we can write,

Let’s define tensor product of vectors and covectors and their rules.

### Tensor product of vectors and covectors

Let , and be vectors or covectors (not necessarily basis ones), the we define the tensor product of each pair as , and etc., and the following rules and operations,

0. Order matters:

1. Scalar multiplication: .

The above rules can be extended to tensor product of any number of vectors or covectors. For example,

1. Scalar multiplication:

The above rules can be recruited to construct vector spaces, called tensor-product vector spaces. For example, if and , then,

Any vector spaces can get into a tensor product. For example, with members like with and .

Note that tensor product of vector space can be done on totally different vector spaces over the same field, e.g. .

#### Basis for a tensor product space

Let and be vector spaces with bases and \{\zeta_j\}_1^m respectively. if and , we can write,

This states that any vector can be written as a linear combination of . Therefore,

1. The set of vectors \} is a basis for the vector space .
2. The dimension of the vector space is .

The above can be extended to tensor product of any number of vector spaces, i.e. the tensor product of the basis vectors of vector spaces creates a basis for the resultant tensor product space.

Example: Let and and .

Then, is a basis for , and,

### Tensor by tensor product

Definition (Tensor-product view): Tensor is a collection of vectors and covectors combined together by using the tensor product (of vectors and/or covectors). A tensor of type (r,s) is a member of the tensor product space,

and written as,

Note that collects the component or the coordinates of the tensor with respect to the basis vectors \{\varepsilon^{i_1}\otimes\cdots\otimes \varepsilon^{i_r}\otimes e_{j_1}\otimes\cdots\otimes e_{j_s}\} or inherently

In this view, a vector is a (0,1) tensor, a covector is a (1,0) tensor. A linear map is a (1,1) tensor. A bilinear form is a (2,0) tensor. A bilinear map is a (2,1) tensor.

# Tensors 1

#### Notations

Einstein summation convention is used here. A matrix M is denoted as and its ij-th element is referred to by . Quantities or coefficients are indexed as for example , or . These indices do not automatically pertain to row and column indices of a matrix, but the quantities can be presented by matrices through isomorphisms once their indices are freely interpreted as rows and columns of matrices.

## Coordinates of a vector

Let be a n-dimensional vector space and with be a basis for . Then, we define the coordinate function as,

such that for a vector written by its components (with respect to ) as the function acts as,

The coordinate function is a linear map.

## Change of basis for vectors

Let and be two basis for , then,

and

where the indices of the scalar terms and are intentionally set this way. So, if all are collected into a matrix , then the sum is over the rows of the matrix for a particular column. In other words, we can utilize the rule of matrix multiplication and write,

The same is true for . In above formulations, note that is a dummy index (i.e. we can equivalently write )

Setting as the initial (old) basis and writing the current (new) basis in terms of is referred to as forward transform denoted by . Relatively, is called backward transform.

The relation between that forward and backward transforms is obtained as follows,

We now find how vector coordinates are transformed relative to different bases. A particular can be expressed by its components according to any of or basis, therefore,

To find the relation between and we write,

As it can be observed, the old basis to new basis is transformed by the forward transform while the old coordinates are transformed to the new ones, , by the backward transform . Because the coordinates of behave contrary to the basis vectors in transformation, the coordinates or the scalar components are said to be contravariant. A vector can be called a contravariant object because its scalar components (coordinates) transforms differently from the basis vectors whose their linearly combination equals to the vector. Briefly,

Proposition: Let . Then, the scalar components/coordinates are transformed by if and only if the basis vectors are transformed by , such that .

Later, a vector is called a contravariant tensor. For the sake of notation and to distinguish between the transformations of the basis and the coordinates of a vector, in index of a coordinate is written as superscript to show it is contravariant. Therefore,

## Linear maps and linear functionals

Definition: is defined as the space of all linear maps where the domain and codomain are vectors spaces.

It can be proved that is a vector space , hence, for and

Note that the addition on the LHS is an operator in and the addition on the RHS is an operator in .

Proposition 1: Let , i.e a linear map from a vector space to another one . If is a basis for , and for and , then is uniquely defined over .

This proposition says a linear map over a space is uniquely determined by its action on the basis vectors of that space. In other words, if and then . proof: let (given by the nature of ), then for such that , we can write , therefore, . Because, ‘s are unique for (a particular) then is unique for and hence must be unique for any . In other word, there is only one unique over such that .

As a side remark, if is a basis for , hence spanning , then spans the range of ; The range of is a subset of .

By this proposition, a matrix completely determining a linear can be obtained for the linear map. let be n-dimensional with a basis , and be m-dimensional with a basis . Then there are coefficients such that,

In the notation , the index is superscript because for a fixed and hence a fixed , the term is the coordinate of and it is a contravariant (e.g ).

For , and , with the coordinates and , we can show that,

This expression can be written as a matrix multiplication of , where presented by its elements as,

As a remark, above can be viewed as columns of the matrix and written as,

### Linear functional (linear form or covector)

Definition: a linear functional on is a linear map . The space is called the dual space of .

Proposition: Let and be defined as . Then, called dual basis of , is a basis of , and hence .

Proof: first we show that ‘s are linearly independent, i.e. . Note that on the RHS, . For a we can write and assume . Then,

Since is arbitrary, ■ .

Now we prove that spans . I.e such that . To this end, we apply both sides to a basis vector of and write which implies or explicitly is found as . Consequently, ■.

Consider and . If , then the matrix of the linear functional/map is

So, for as we can write,

Result: if the coordinates of a vector is shown by a column vector or single-column matrix (which is a vector in the space of ), then a row vector or a single-row matrix represents the matrix of a linear functional.

Definition: a linear functional , which can be identified with a row vector as its matrix, is also called a covector.

Like vectors, a covector (and any linear map) is a mathematical object that is independent of a basis (i.e. invariant). The geometric representation of a vector in (or by an isomorphism in) is an arrow in . For a covector isomorphic to , the representation is a set (stack) of planes in that can be represented by iso lines in . A covector that is isomorphic to can be represented by iso surfaces in .

Example: Let be a basis of and be the matrix of a covector in some . Then, if , we can write,

which, for different values of , is a set of (iso) lines in a Cartesian CS defined by two axes and along and that are the geometric representations of and . The Cartesian axes are not necessarily orthogonal.

If we chose any other basis for , then the matrix of the covector changes. Also, the geometric representations of are different from and and hence the geometric representation of the covector stays the same shape.

Example: Let be a basis of and be a basis for . This means and . Then, the matrix of each dual basis vector is as,

### Change of basis for covectors

Let and be two bases for , and hence, and be two bases for . Each dual basis vector can be written in terms of the (old) dual basis vectors by using a linear transformation as . Now, the coefficients are to be determined as follows,

Using the formula regarding the change of basis of vectors, the above continues as,

This indicates that the dual basis are transformed by the backward transformation. Referring to the index convention, we use subscript for components that are transformed trough a backward transformation. Therefore,

meaning that dual basis vectors are contravariant because they behave contrary to the basis vectors in transformation from to .

Now let . Writing and using the above relation, we get,

meaning that they are transforming in a covariant manner when the basis of the vector space changes from to .

Briefly the following relations have been shown.

### Basis and change of basis for the space of linear maps

As can be proved is a linear vector space and any linear map is a vector. Therefore, we should be able to find a basis for this space. If is n-dimensional and is m-dimensional, the is mn-dimensional and hence its basis should have vectors, i.e. linear maps. Let’s enumerate the basis vectors of as for and , then any linear map can be written as,

By proposition 1, any linear map is uniquely determined by its action on the basis vectors of its codomain. If be a basis for , then for any basis vector ,

Setting a basis for as , the above equation becomes,

This equation holds if,

Therefore, we can choose a set of basis vectors for as,

By recruiting the basis of , the above can be written as,

The term is obviously a linear map . It can be readily shown that being a linear combinations of the derived basis vectors is linearly independent, i.e. for any (here, note that ).

So, a linear map can be written as a linear combination . Here, it is necessary to use the index level convention. To this end, we observe that for a fixed the term couples with and represents the coordinates of a covector. As coordinates of a covector are covariant, index is written as subscript. For a fixed though, the term couples with and represents the coordinates of a vector. As coordinates of a vector are contravariant, index should rise. Therefore, we write,

The coefficients can be determined as,

Stating that are the coordinates of with respect to the basis of , i.e. . Comparing with what was derived as , we can conclude that . Therefore,

The above result can also be derived from as follows.

Change of basis of is as follows.

For , let and be bases for , and and be bases for . Also, and are corresponding bases of . Forward and backward transformation pairs in and are denoted as and .

Note that the coordinates of a linear map need two transformations such that the covariant index of pertains to the forward transformation and the contravariant index pertains to the backward transformation.

Example: let , then,

If the matrices, , , and are considered, we can write,

## Bilinear forms

A bilinear form is a bilinear map defined as . Setting a basis for , a bilinear form can be represented by matrix multiplications on the coordinates of the input vectors. If is a basis for , then

which can be written as,

where with .

The expression indicates that a bilinear form is uniquely defined by its action on the basis vectors. This is the same as what was shown for linear maps by proposition 1. This comes from the fact that a bilinear form is a linear map with respect to one of its arguments at a time.

Now we seek a basis for the space of bilinear forms, i.e. . This is a vector space with the following defined operations.

The dimension of this space is , therefore, for any bilinear form there are bilinear forms such that,

From the result we can conclude that

Following the index level convention, the indices of should stay as subscripts because each index pertains to the covariant coordinates of a covector after fixing the other index.

If and are two bases for , then the change of basis of the space of bilinear forms are as follows.

Example: the metric bilinear map (metric tensor)

Dot/inner product on the vector space over is defined as a bilinear map such that, and . With this regard two objects (that can have geometric interpretations for Euclidean spaces) are defined as,

1- Length of a vector
2- Angle between two vectors

Let see how the dot product is expressed through the coordinates of vectors. With being a basis for , we can write,

The term is called the metric tensor and its components can be presented by an n-by-n matrix as .

If the basis is an orthonormal basis, i.e. , then and is the identity matrix. Therefore, and .

## Multilinear forms

A general multilinear form is a multilinear map defined as , where is a vector space. Particularly setting leads to a simpler multilinear form as .

Following the same steps as shown for a bilinear map, a multilinear form can be written as,

which implies,

showing that a multilinear form can be written as a linear combination of covectors.

## Multilinear maps

A multilinear map is a map . A multilinear map can be written in terms of vector and covector basis. For example, consider as with and being bases for and . We can write,

Because for each and we have , we can write,

We write the indices and in as subscripts in accordance with their position on the LHS; however, we’ll see that is a coordinate of a covector for each or when is fixed. Combining above, we can write,

The term collects coefficients and uniqully defines the multilinear map. We can imagine this term as a 3-dimensional array/matrix. Above also shows that the multilinear map can be written as a linear combination of basis covectors and basis vectors.

## Definition of a tensor

Defining the following terms,

• Vector space and basis and another basis .
• Basis transformation as , and therefore .
• The dual vector space of as .
• Vector space and basis and another basis .
• Basis transformation as , and therefore
• Linear map .
• Bilinear form .

we concluded that,

It is observed that if a vector is written in terms a single sum/linear combination of basis vectors of , then the components of the vectors change contravariantely with respect to a change of basis. Then, the covectors are considered and it is observed that their components change covariently upon change of basis of or . A linear map can be written as a linear combination of vectors and covectors. The coefficients of this combination is seen to change both contra- and covariantely when the bases (of and ) change. A bilinear form though can be written in terms of a linear combination of covectors. The corresponding coefficients change covariantly with change of basis. These results can be generalized toward an abstract definition of a mathematical object called a tensor. There are two following approaches for algebraically denfining a tensor.

### Tensor as a multilinear form

Motivated by how linear maps, bilinear forms, and multilinear forms and maps can be written by combining basis vectors and covectors, a generalized combination of these vectors can considered. For example,

This object consists of a linear combination of a unified (merged) set of basis vector and covectors (of and ) by scalar coefficients . According to the type of the basis vectors, the indices become sub- or superscript, and hence it determines the type of the transformation regarding that index. By reordering the basis vectors and covectors, we can write,

Recalling that vector components can be written as implines that there is a map for a particular vector such that,

And also the components of a covector determined as,

motivates defining as the array (collection of the coefficients) of a multilinear form,

Therefore, the object or the multi-dimensional array which is dependent on chosen bases of and and its transformation rules are based on the types of the beases (or indices) can be intrinsically related to an underlying multilineat map . By virtue of this observation, an object called a tensor is defined as the following.

Definition: A tensor of type (r,s) on a vector space over a field (or ) is a multilinear map as,

The coordinates or the (scalar) components of a tensor can then be determined once a basis for and a basis for are fixed. Therefore,

Note that r is the number of covariant indices and s is the number of contravariant indices. A tensor of type (r,s) can be imagined as an r+s-dimensional array of data containing elements. Each index corresponds to a dimention of the data array.

By this definition, a vector is a (0,1) tensor as it can be viewed as,

This implies that for each there is a (multilinear with one input) map receiving a basis covector and returning the corresponding scalar component of the vector (). This corresponding map is unique for each vector and it is called a tensor.

A covector (dual vector or a linear form) is a (1,0) tensor because a covector is a linear map

َ A linear map (or ) is a (1,1) tensor because the term in pertains to a covector’s components for a fixed and to a vector’s components for a fixed . Therefore,

in other words, a linear map is a tensor viewed as a multilinear map . Here, the multilibear form can be considered as by which gives . Note that returns a vector and being extract its j-th coordinate, and hencem the array/matrix of the linear map is retrieved.

A bilinear form is then a (2,0) tensor, where

A multilinear form is a (2,1) tensor where where can be considered as .

As an example, the cross product of two vectors defined as is a multilinear map is a (2,1) tensor.

By convension, scalars are (0,0) tensors.

Remark: for a tensor we can write,

Example: Stress tensor. The Cauchy stress tensor in mechanics is a linear map and hence a (1,1) tensor.

#### Rank of a tensor

The rank of a (r,s)-type tensor is defined as r+s. In this regard, tensors of different types can have the same rank. For example tensors of types (1,1), (2,0), (0,2) have the same rank being 2. Here we compare these tensors with eachother.

A (1,1) tensor representing a linear map is where with .

A (2,0) tensor representing a bilinear form is where with .

A (0,2) tensor is where with .

The coeffieints of the above tensor are collected in 2-dimensional array/matrix; however, they follow different transformation rules based on their types.

# Linear Algebra Cheat Sheet (1)

#### 0- Notations and convention

A variable/quantity/element can be scalar, vector, or a matrix; its type should be understood from the context if not explicitly declared.

0-1- A vector is considered as a column vector, i.e. a single-column matrix. Therefore:

z\in \mathbb{F}^{n}\equiv z\in \mathbb{F}^{n\times 1}

A row vector is therefore defined as .

0-2- Dot product of column vectors:

\text{For }x,y\in \mathbb{R}^{n\times 1}, \ x\cdot y:=x^\text Ty

0-3- The ij-th element of a matrix is denoted by . The i-th column and j-th row of the matrix are respectively denoted by and .

0-4- Columns and rows of a matrix are column and row vectors. A -matrix can then be represented as:

\begin{bmatrix} v_1& v_2& \dots & v_n\end{bmatrix}\ \text{ s.t } v_i=A_{,i}

or

\begin{bmatrix}u_1^\text T\\ u_2^\text T\\ \vdots\\ u_n^\text T\end{bmatrix}\ \text{ s.t } u_i^\text T=A_{i,}

#### 1- Orthogonal matrix

a square matrix is said (definition) to be orthogonal iff ; provided that , inverse of exists. As a result, .

The following are equivalent for :
a) is orthogonal.
b) the column vectors of are ortho-normal.
c) the row vectors of are ortho-normal.
d) is size preserving: , being the Euclidean norm, and .
e) is dot product preserving: .

#### 2- Some matrix multiplication identities

A\in \mathbb{F}^{m\times n},\ x\in \mathbb{F}^{n\times 1}\ \text{ then } Ax=\sum_{i=1}^n x_iA_{,i} A\in \mathbb{F}^{m\times n}\ , D\in \mathbb{F}^{n\times n}\ \text{ and diagonal }, B\in \mathbb{F}^{n\times k},\ \text{ s.t.}\\ \ \\ A=\begin{bmatrix} v_1& v_2& \dots & v_n\end{bmatrix}, \ B=\begin{bmatrix}u_1^\text T\\ \\ u_2^\text T\\ \vdots\\ u_n^\text T\end{bmatrix},\ D_{ij}= \begin{cases} \lambda_i &\text{, } i=j \\ 0 &\text{, } i\ne j \end{cases} \\ \ \\ \text{then }ADB=\sum_{i=1}^n \lambda_iv_i u_i^\text T

#### 3- Change of basis

let and be two basis of . Then the following holds for the coordinates of a vector with respect to the two bases:

[v]_{\beta’}=P_{\beta\to\beta’}[v]_{\beta}\quad\text{s.t} \\ \ \\ P_{\beta\to\beta’}=\begin{bmatrix} [b_1]_{\beta’}& [b_2]_{\beta’}& \dots & [b_n]_{\beta’}\end{bmatrix}

It can be proved that:

# Rank of a Matrix

Let (the results also holds for ). Then, the column rank/row rank of A is defined as the dimension of the column/row space of A, i.e. the dimension of the vector space spanned by the columns/rows of A; this is then equivalent to the number of linearly independent columns/rows (column/rows vector) of A.

Theorem: column rank of A = row rank of A.

Definition: the rank of a matrix, rank(A), is the dimension of either the column or row space of A; simply the number of linearly independent columns or rows of A.

Definition: for a linear map , the rank of the linear map is defined as the dimension of the image of . This definition is equivalent to the definition of the matrix rank as every linear map has a matrix by which it can be written as .

Proposition: . This leads to these definitions: A matrix is said to be full rank iff , i.e. the largest possible rank, and it is said to be rank deficient iff , i.e. not having full rank.

## Properties of rank

For :

1- only a zero matrix has rank zero.

2- If , then

3-

4-

5-

6- If , then for , . In addition, for , , i.e. has at most rank n.

7- A square matrix can be decomposed as where is a diagonal matrix containing the eigenvalues of . Then, , i.e. the number of non-zero eigenvalues of .

8- For a square matrix , then equivalently is full rank, is invertible, has non-zero determinant, and has n non-zero eigenvalues.

## Proofs

P6:

V=\begin{bmatrix}v_1\begin{bmatrix}v_1\\v_2\\ \vdots\\v_r \end{bmatrix}&v_2\begin{bmatrix}v_1\\v_2\\ \vdots\\v_r \end{bmatrix}&\dots &v_r\begin{bmatrix}v_1\\v_2\\ \vdots\\v_r \end{bmatrix}\end{bmatrix}

where are coordinates of the vector . This indicates that each column of is a scalar multiple of any other columns of ; therefore, the column space is one dimensional. Hence, .

For the second part, property 3 proves the statement.