PolarSPARC

Introduction to Linear Algebra - Part 2


Bhaskar S 02/06/2022


In Part 1 of this series, we covered the topic of Vectors. In this part, we will cover the topic on Matrices.

Matrices

The following are some of the characteristics of Matrices:

The following is an example of a 3 x 5 matrix:

$\begin{bmatrix} 3 & 9 & -1 & 2 & -5 \\ 8 & 2 & 0 & -7 & 9 \\ -4 & 7 & 9 & 2 & -3 \end{bmatrix}$

Matrix Transpose

The transpose of a matrix $\textbf{A}$, represented as $\textbf{A}^T$, is a new matrix where the rows and columns get swapped. In other words, the rows become the columns in the new matrix. Mathematically, every element $x_{i,j}$ in the given matrix $\textbf{A}$ becomes the element $x_{j,i}$ in the transposed matrix $\textbf{A}^T$.

The following is a 3 x 4 matrix $\textbf{A}$:

$\textbf{A} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$

The following is the transpose of the matrix $\textbf{A}$, which results in a 4 x 3 matrix $\textbf{A}^T$:

$\textbf{A}^T = \begin{bmatrix} 3 & 8 & -4 \\ 9 & 2 & 7 \\ -1 & 0 & 9 \\ 2 & -7 & 2 \end{bmatrix}$

Zero Matrix

A matrix that is represented as $\textbf{O}$ in which all the elements are zeros.

The following is an example of a 3 x 3 zero matrix:

$\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}$

Square Matrix

When the number of rows and columns are equal, it is a Square matrix.

The following is an example of a 3 x 3 square matrix:

$\begin{bmatrix} 3 & 9 & -1 \\ 8 & 2 & 0 \\ -4 & 7 & 9 \end{bmatrix}$

Symmetric Matrix

A sqaure matrix such that the matrix $\textbf{A}$ equals its transpose $\textbf{A}^T$.

The following is an example of a 3 x 3 symmetric matrix:

$\begin{bmatrix} 3 & 2 & 4 \\ 2 & 5 & 6 \\ 4 & 6 & 7 \end{bmatrix}$

Identity Matrix

A square matrix (of size N x N) that is represented as $\mathbf{I_N}$ in which all the diagonal elements are ones, while all the others elements are zeros.

The following is an example of a 3 x 3 identity matrix:

$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$

Diagonal Matrix

A matrix in which all the elements outside the elements along the diagonal are zeros.

The following is an example of a 3 x 3 diagonal matrix:

$\begin{bmatrix} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 3 \end{bmatrix}$

The following is an example of a 4 x 3 diagonal matrix:

$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & -5 \\ 0 & 0 & 0 \end{bmatrix}$

Equality of Matrices

For two matrices $\textbf{A}$ and $\textbf{B}$ to be equal, they first need to be of the same size M x N, and second each of the corresponding elements in the two matrices must be equal, that is, $a_{i,j} = b_{i,j}$.

The following is the 3 x 4 matrix $\textbf{A}$:

$\textbf{A} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$

The following is the 3 x 4 matrix $\textbf{B}$:

$\textbf{B} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$

The following is the 3 x 4 matrix $\textbf{C}$:

$\textbf{C} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & \textcolor{red}{0} & 9 & 2 \end{bmatrix}$

The two matrices $\textbf{A}$ and $\textbf{B}$ are EQUAL, while the matrices $\textbf{B}$ and $\textbf{C}$ are NOT equal (because of the element in $\textcolor{red}{RED}$ in matrix $\textbf{C}$).

Matrix Addition and Subtraction

To add or subtract two matrices $\textbf{A}$ and $\textbf{B}$ of the same size M x N, just add or subtract each of the corresponding elements in the two matrices, that is, $a_{i,j} \pm b_{i,j}$.

The following is the 3 x 4 matrix $\textbf{A}$:

$\textbf{A} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$

The following is the 3 x 4 matrix $\textbf{B}$:

$\textbf{B} = \begin{bmatrix} -1 & 0 & 2 & 4 \\ -3 & 1 & 0 & 6 \\ 7 & 0 & -3 & 4 \end{bmatrix}$

The following is the result of adding matrices $\textbf{A}$ and $\textbf{B}$:

$\textbf{A} + \textbf{B} = \begin{bmatrix} 2 & 9 & 1 & 6 \\ 5 & 3 & 0 & -1 \\ 3 & 7 & 6 & 6 \end{bmatrix}$

The following is the result of subtracting matrices $\textbf{A}$ and $\textbf{B}$:

$\textbf{A} - \textbf{B} = \begin{bmatrix} 4 & 9 & -3 & -2 \\ 11 & 1 & 0 & -13 \\ -11 & 7 & 12 & -2 \end{bmatrix}$

Matrix Scalar Multiplication

To multiple a scalar $\lambda$ with a matrix $\textbf{A}$, just multiple the scalar with each of the elements in the matrix, that is, $\lambda.a_{i,j}$.

The following is the 3 x 4 matrix $\textbf{A}$:

$\textbf{A} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$

The following is the result of multiplying $\lambda = 2$ with the matrix $\textbf{A}$:

$2 . \textbf{A} = \begin{bmatrix} 6 & 18 & -2 & 4 \\ 16 & 4 & 0 & -14 \\ -8 & 14 & 18 & 4 \end{bmatrix}$

Matrix Multiplication

To multiply two matrices $\textbf{A}$ and $\textbf{B}$, the first matrix $\textbf{A}$ must be of size M x N and the second matrix $\textbf{B}$ must be of size N x P. In other words, the number of columns in the first matrix $\textbf{A}$ MUST match the number of rows in the second matrix $\textbf{B}$. Each element in the new matrix is a linear combination represented as $\sum\limits_{k=1}^N a_{i,k}.b_{k,j}$, where $i$ represents a row and $j$ represents a column. The resulting new matrix will be of size M x P.

The following is the 3 x 2 matrix $\textbf{A}$:

$\textbf{A} = \begin{bmatrix} a & b \\ c & d \\ e & f \end{bmatrix}$

The following is the 2 x 2 matrix $\textbf{B}$:

$\textbf{B} = \begin{bmatrix} p & q \\ r & s \end{bmatrix}$

The following is the result of multiplying the two matrices $\textbf{A}$ and $\textbf{B}$:

$\textbf{A}\textbf{B} = \begin{bmatrix} a.p+b.r & a.q+b.s \\ c.p+d.r & c.q+d.s \\ e.p+f.r & e.q+f.s \end{bmatrix}$

The following is a pictorial illustration of the matrix multiplication:

Matrix Multiplication
Fig.1

Note that the order of matrices involved in the multiplication matter, meaning, $\textbf{A}\textbf{B} \ne \textbf{B}\textbf{A}$.

Let us look at an example for the matrix multiplication.

Example-1 Multiple the following two matrices: $\textbf{A} = \begin{bmatrix} 3 & 9 \\ -1 & 2 \\ 8 & 2 \\ 0 & -7 \end{bmatrix}$ and $\textbf{B} = \begin{bmatrix} 2 & 3 \\ -1 & 4 \end{bmatrix}$

Two multiply the two matrices $\textbf{A}$ and $\textbf{B}$, each element in the new matrix is a linear combination represented as $\sum\limits_{k=1}^N a_{i,k}.b_{k,j}$, where $i$ represents a row and $j$ represents a column.

That is, $\textbf{A}\textbf{B} = \begin{bmatrix} 3.2+9.(-1) & 3.3+9.4 \\ (-1).2+2.(-1) & (-1).3+2.4 \\ 8.2+2.(-1) & 8.3+2.4 \\ 0.2+(-7).(-1) & 0.3+(-7).4 \end{bmatrix} = \begin{bmatrix} -3 & 45 \\ -4 & 5 \\ 14 & 32 \\ 7 & -28 \end{bmatrix}$


Let us look at another example for the matrix multiplication.

Example-2 Multiple the following two matrices: $\textbf{A} = \begin{bmatrix} 2 & 1 & 3 \\ 5 & 4 & 6 \end{bmatrix}$ and $\textbf{B} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix}$

$\textbf{A}\textbf{B} = \begin{bmatrix} 2.2+1.3+3.4 \\ 5.2+4.3+6.4 \end{bmatrix} = \begin{bmatrix} 19 \\ 46 \end{bmatrix}$


Let us look at another interesting example of matrix multiplication involving a diagonal matrix.

Example-3 Multiple the following two matrices: $\textbf{A} = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix}$ and $\textbf{B} = \begin{bmatrix} 2 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & 6 \end{bmatrix}$

$\textbf{A}\textbf{B} = \begin{bmatrix} 2.a+0.b+0.c & 0.a+4.b+0.c & 0.a+0.b+6.c \\ 2.d+0.e+0.f & 0.d+4.e+0.f & 0.d+0.e+6.f \\ 2.g+0.h+0.i & 0.g+4.h+0.i & 0.g+0.h+6.i \end{bmatrix} = \begin{bmatrix} 2a & 4b & 6c \\ 2d & 4e & 6f \\ 2g & 4h & 6i \end{bmatrix}$

Notice how each column in the matrix $\textbf{A}$ is scaled by the corresponding column element along the diagonal of the matrix $\textbf{B}$.


Properties of Matrices

The following are some of the properties of matrix addition and matrix scalar multiplication:

System of Linear Equations

A System of Linear Equations is a set of M linear equations that consist of the same set of N variables.

The following is a generic system of linear:

$\begin{align*} & a_{11}.x_1 + a_{12}.x_2 + ... + a_{1N}.x_N = b_1 \\ & a_{21}.x_1 + a_{22}.x_2 + ... + a_{2N}.x_N = b_2 \\ & ... \\ & a_{M1}.x_1 + a_{M2}.x_2 + ... + a_{MN}.x_N = b_M \end{align*}$

The scalars $a_{11}, ..., a_{MN}$ are the coefficients and the scalars $b_1, ..., b_M$ are the constants of the system of linear equations.

The solution for the set of linear equations is a sequence of scalars $n_1, n_2, ..., n_N$, such that, $x_1 = n_1, x_2 = n_2, ..., x_N = n_N$ that satisfies the system of linear equations.

Let us look at an example on how to solve a set of linear equations using Algebraic arithmetic.

Example-4 Solve for the following three linear equations: $\begin{align*} & 2x + y - 2z = -3 \\ & x - 3y + z = 8 \\ & 4x - y - 2z = 3 \end{align*}$

Let us first label the three linear equations so we can refer to them in the algebraic arithmetic operations.

$\begin{flalign} & 2x + y - 2z = -3 \longrightarrow (1) \\ & x - 3y + z = 8 \longrightarrow (2) \\ & 4x - y - 2z = 3 \longrightarrow (3) \end{flalign}$

Performing the algebraic subtraction $(3) - (1)$, we get the following equation:

$\begin{flalign} & 2x - 2y = 6 \longrightarrow (3) \end{flalign}$

Simplifying equation $(3)$, we get:

$\begin{flalign} & x - y = 3 \longrightarrow (3) \end{flalign}$

Multiplying equation $(2)$ with scalar 2, we get:

$\begin{flalign} & 2x - 6y + 2z = 16 \longrightarrow (2) \end{flalign}$

Performing the algebraic addition $(2) + (1)$, we get the following equation:

$\begin{flalign} & 4x - 5y = 13 \longrightarrow (2) \end{flalign}$

Multiplying equation $(3)$ with scalar 4, we get:

$\begin{flalign} & 4x - 4y = 12 \longrightarrow (3) \end{flalign}$

Performing the algebraic subtraction $(2) - (3)$, we can solve for $y$:

$\begin{flalign} & y = -1 \end{flalign}$

Substituting for $y$ in equation $(3)$, we can solve for $x$:

$\begin{flalign} & x = 2 \end{flalign}$

Substituting for $x$ and $y$ in equation $(1)$, we can solve for $z$:

$\begin{flalign} & z = 3 \end{flalign}$


Things get complicated when the number of variables increases. This is where matrices come to the rescue. One could use matrices to represent the system of linear equations and find a solution using matrix operations. The process of finding the solution using the matrix operations is called Gaussain Elimination.

Using the system of three equations from the Example-4 above, the following is the matrix representation of the set of equations:

$\begin{bmatrix} 2 & 1 & -2 \\ 1 & -3 & 1 \\ 4 & -1 & -2 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} -3 \\ 8 \\ 3 \end{bmatrix}$

The above matrix equation is of the form: $\textbf{A}\vec{x} = \vec{b}$, where the matrix $\textbf{A}$ represents all the coefficients of the system of linear equations (referred to as the coefficient matrix), the column vector $\vec{x}$ represents all the variables in the system of linear equations, and the column vector $\vec{b}$ represents all the constants in the system of linear equations.


!! CAUTION !!

There will be situations when one (or more) of the variables may be missing from one or more equations - in those cases it implies that the corresponding coefficients are 0 (Zero).

Consider the following system of equations:

$\begin{align*} & x - 2y + 3z = 6 \\ & -x + y -z = -4 \\ & 2x - 3z = 3 \end{align*}$

Notice the $y$ term missing in the third equation. This simply means the coefficient of the $y$ term is 0.

The following is the matrix equivalent of the above set of linear equations:

$\begin{bmatrix} 1 & -2 & 3 \\ -1 & 1 & -1 \\ 2 & 0 & -3 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} 6 \\ -4 \\ 3 \end{bmatrix}$

In order to solve a set of linear equations using Gaussian elimination, a more compact form is used in practice, called the Augmented Matrix.

The following is the augmented matrix equivalent of the above set of linear equations:

$\begin{bmatrix} 1 & -2 & 3 &\bigm| & 6 \\ -1 & 1 & -1 &\bigm| & -4 \\ 2 & 0 & -3 &\bigm| & 3 \end{bmatrix}$

The process involved in the Gaussian elimination is to reduce the given augmented matrix to the following upper triangular matrix form (with 1s along the diagonal) through a series of row-wise matrix operations:

$\begin{bmatrix} 1 & a & b &\bigm| & p \\ 0 & 1 & c &\bigm| & q \\ 0 & 0 & 1 &\bigm| & r \end{bmatrix}$

In other words, the elements of the diagonal must be all ONEs and below the diagonal the elements must be all ZEROs.

This form of the upper triangular augmented matrix is called the Row Echelon form.

The following are the some of the properties of a matrix in row echelon form:

In order to reduce a given matrix to a row echelon form, the following are the valid row-level matrix operations that one can perform on the given augmented matrix:

If we look at the operations we performed on the set of equations in the Example-4 above, it was exactly the same as the row-level matrix operations indicated above.

Let us look at an example on how to solve a set of linear equations using Gaussain Elimination.

Example-5 Solve for the following three linear equations: $\begin{align*} & x - 2y + 3z = 9 \\ & -x + 3y = -4 \\ & 2x - 5y + 5z = 17 \end{align*}$

Let us convert the three linear equations to an augmented matrix as follows:

$\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ -1 & 3 & 0 &\bigm| & -4 \\ 2 & -5 & 5 &\bigm| & 17 \end{bmatrix}$

Performing the operation $R_2 + R_1 \rightarrow R_2$, we get the following:

$\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ 0 & 1 & 3 &\bigm| & 5 \\ 2 & -5 & 5 &\bigm| & 17 \end{bmatrix}$

Performing the operation $R_3 + (-2)R_1 \rightarrow R_3$, we get the following:

$\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ 0 & 1 & 3 &\bigm| & 5 \\ 0 & -1 & -1 &\bigm| & -1 \end{bmatrix}$

Performing the operation $R_3 + R_2 \rightarrow R_3$, we get the following:

$\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ 0 & 1 & 3 &\bigm| & 5 \\ 0 & 0 & 2 &\bigm| & 4 \end{bmatrix}$

Performing the operation $\frac{1}{2}R_3 \rightarrow R_3$, we get the following:

$\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ 0 & 1 & 3 &\bigm| & 5 \\ 0 & 0 & 1 &\bigm| & 2 \end{bmatrix}$

From the augmented matrix just above, we can derive the solution for $z$, which equals $2$.

From $R_2$, we know $y + 3z = 5$. Substituting $z = 2$, we can derive the solution for $y$, which equals $-1$.

Finally, from $R_1$, we know $x - 2y + 3z = 9$. Substituting $y = -1$ and $z = 2$, we can derive the solution for $x$, which equals $1$.


References

Introduction to Linear Algebra - Part 1


© PolarSPARC