Introduction to Linear Algebra

The following are some of the basic definitions that will be referred in to following paragraphs:

Name	Description
Scalar	is just a number (or a constant) and is represented using the lowercase greek alphabets - $\alpha, \beta, \gamma, \lambda$, etc.
Vector	a quantity that has both magnitude and direction, and is represented as a lowercase letter with an arrow on the top - $\vec{a}$
Matrix (plural Matrices)	a rectangular table of mathematical elements that is arranged in rows and columns, and is represented with an appercase letter. Example: $\begin{bmatrix} 3 & 5 & 8 \\ 2 & 4 & 9 \end{bmatrix}$

The following are some of the characteristics of Vectors:

is an ordered list of numbers (integer, real, rational, irrational, etc.), that are represented inside square brackets [] or parentheses (), and can be column-oriented $\begin{bmatrix} 3 \\ 5 \end{bmatrix}$ or row-oriented $\begin{bmatrix} 3 & 5 \end{bmatrix}$. The convention is to use the column-oriented representation
the number of elements in the vector is called the Dimensionality of the vector
to perform any mathematical operations on two (or more) vectors, they must have the same dimensionality
to perform addition of two vectors, just add the corresponding elements of the vector. Example: $\begin{bmatrix} -1 \\ 3 \end{bmatrix} + \begin{bmatrix} 3 \\ 5 \end{bmatrix} = \begin{bmatrix} -1+3 \\ 3+5 \end{bmatrix} = \begin{bmatrix} 2 \\ 8 \end{bmatrix}$
to perform subtraction of two vectors, just subtract the corresponding elements of the vector. Example: $\begin{bmatrix} -1 \\ 3 \end{bmatrix} - \begin{bmatrix} 3 \\ 5 \end{bmatrix} = \begin{bmatrix} -1-3 \\ 3-5 \end{bmatrix} = \begin{bmatrix} -4 \\ -2 \end{bmatrix}$
to perform multiplication of a vector with a scalar, just multiple the scalar with each of the elements in the vector. Example: $2 * \begin{bmatrix} 2 \\ 3 \end{bmatrix} = \begin{bmatrix} 2*2 \\ 2*3 \end{bmatrix} = \begin{bmatrix} 4 \\ 6 \end{bmatrix}$
the transpose of a column vector $\vec{a}$, represented as $\vec{a}^T$ is a row vector and vice versa.
Example: if $\vec{a} = \begin{bmatrix} 3 \\ 5 \\ 7 \end{bmatrix}$, then $\vec{a}^T = \begin{bmatrix} 3 & 5 & 7 \end{bmatrix}$

The Dot (also known as the Scalar or Inner) product of two vectors (of same dimensionality) is a single scalar. To perform the dot product of the two vectors, multiply the corresponding elements of the two vectors and then sum each of the individual products. In other words, if $\vec{a}$ and $\vec{b}$ are the two vectors, then $\vec{a}^T.\vec{b} = \sum\limits_{i=1}^n\vec{a_i}\vec{b_i}$.

Let us look at an example for the dot product.

Example-1	Find the dot product of the vectors $\begin{bmatrix} 2 \\ -1 \\ 6 \end{bmatrix}$ and $\begin{bmatrix} 3 \\ 2 \\ 1 \end{bmatrix}$
The dot product is computed using the formula $\vec{a}^T.\vec{b} = \sum\limits_{i=1}^n\vec{a_i}\vec{b_i}$. That is, $(2 * 3) + (-1 * 2) + (6 * 1) = 6 -2 + 6 = 10$

Example-1

Find the dot product of the vectors $\begin{bmatrix} 2 \\ -1 \\ 6 \end{bmatrix}$ and $\begin{bmatrix} 3 \\ 2 \\ 1 \end{bmatrix}$

The dot product is computed using the formula $\vec{a}^T.\vec{b} = \sum\limits_{i=1}^n\vec{a_i}\vec{b_i}$.

That is, $(2 * 3) + (-1 * 2) + (6 * 1) = 6 -2 + 6 = 10$

The following are some of the properties of Dot Product (of vectors):

Associative :: true with scalar - $\beta(\vec{a}^T\vec{b}) = (\beta\vec{a}^T)\vec{b} = \vec{a}^T(\beta\vec{b}) = (\vec{a}^T\vec{b})\beta$. However, NOT true with vector $\vec{a}^T(\vec{b}^T\vec{c}) \ne (\vec{a}^T\vec{b})^T\vec{c}$
Commutative :: $\vec{a}^T\vec{b} = \vec{b}^T\vec{a}$
Distributive :: $\vec{a}^T(\vec{b} + \vec{c}) = \vec{a}^T\vec{b} + \vec{a}^T\vec{c}$

We will use a geometric perspective to find the magnitude of a vector for a better intuition. A vector $\vec{a}$ in a two dimensional space (since it is easy to visualize) can be represented using a line from the origin $(0, 0)$ to some coordinate point $(x, y)$.

The following graph is the illustration of the vector:

From a geometric perspective, the dot product of two vectors $\vec{a}$ and $\vec{b}$ in a two dimensional space is the cosine of the angle between the two vectors multiplied with the lengths of the two vectors.

In other words, $a^Tb = \sum\limits_{i=1}^n\vec{a_i}\vec{b_i} = \lVert \vec{a} \rVert \lVert \vec{b} \rVert \cos \theta$, where $\theta$ is the angle between the two vectors $\vec{a}$ and $\vec{b}$.

Rearranging the equation, we get: $\cos \theta = \Large{\frac{a^Tb}{\lVert \vec{a} \rVert \lVert \vec{b} \rVert}}$.

Or, $\theta = \arccos\Large{(\frac{a^Tb}{\lVert \vec{a} \rVert \lVert \vec{b} \rVert})}$, where $\arccos = \cos^{-1}$.

The following graph is the illustration of the two vectors:

From the triangle above, using the Pythagorean Theorem, we can infer the following:

$c^2 = p^2 + q^2$ ... (1)

$a^2 = p^2 + (b-q)^2$ ... (2)

Using Trigonometry, we can infer the following:

$\sin \theta = \Large{\frac{p}{a}}$ OR $p = a \sin \theta$ ... (3)

$\cos \theta = \Large{\frac{b-q}{a}}$ OR $q = b - a \cos \theta$ ... (4)

Expanding the equation (2) from above, $p^2 + (b^2 + q^2 - 2bq) = a^2$

That is, $(p^2 + q^2) + b^2 -2bq = a^2$

Replacing the first term with $c^2$ from equation (1) above, we get $c^2 + b^2 -2bq = a^2$

Replacing q from equation (4) above, we get $c^2 + b^2 -2b(b - a \cos \theta) = a^2$

Simplifying, we get $c^2 = a^2 + b^2 - 2ab \cos \theta = {\lVert \vec{a} \rVert}^2 + {\lVert \vec{b} \rVert}^2 - 2\lVert \vec{a} \rVert \lVert \vec{b} \rVert \cos \theta$

This is the equation for the Law of Cosines.

NOTICE that when $\theta = 90^{\circ}$, the two vectors $\vec{a}$ and $\vec{b}$ are said to be orthogonal to each other and $\cos \theta = 0$. The equation for the Law of Cosines morphs into the equation for the Pythagorean Theorem.

From geometry, we can infer $\vec{c} = \vec{a} - \vec{b}$

In other words, $\lVert \vec{c} \rVert = \lVert \vec{a} \rVert - \lVert \vec{b} \rVert$

We know $\lVert \vec{a} \rVert = a^Ta$. Therefore, rewriting the above equation as:

$\lVert \vec{c} \rVert = (a - b)^T(a - b) = a^Ta - 2a^Tb + b^Tb = {\lVert \vec{a} \rVert}^2 + {\lVert \vec{b} \rVert}^2 - 2a^Tb$

Substituting in the equation for the law of cosines, we get:

${\lVert \vec{a} \rVert}^2 + {\lVert \vec{b} \rVert}^2 - 2a^Tb = {\lVert \vec{a} \rVert}^2 + {\lVert \vec{b} \rVert}^2 - 2ab \cos \theta$

Simplifying the equation, we get: $a^Tb = \lVert \vec{a} \rVert \lVert \vec{b} \rVert \cos \theta$, thus proving the geometric interpretation of the vector dot product.

The Outer product of two vectors (of same or different dimensionality) results in a matrix. In other words, if $\vec{a}$ has a dimensionality of N and $\vec{b}$ has a dimensionality of M, then the Outer Product of the two vectors is a matrix of N x M elements.

An example will make it clear

Let $\vec{a} = \begin{bmatrix} 4 \\ 1 \\ 3 \\ 2 \end{bmatrix}$

Let $\vec{b} = \begin{bmatrix} a & b & c \end{bmatrix}$

$\vec{a}\vec{b}^T = \begin{bmatrix} 4 \\ 1 \\ 3 \\ 2 \end{bmatrix} \begin{bmatrix} a & b & c \end{bmatrix} = \begin{bmatrix} 4.a & 4.b & 4.c \\ 1.a & 1.b & 1.c \\ 3.a & 3.b & 3.c \\ 2.a & 2.b & 2.c \end{bmatrix} = \begin{bmatrix} 4a & 4b & 4c \\ a & b & c \\ 3a & 3b & 3c \\ 2a & 2b & 2c \end{bmatrix}$

For each element $a_i$ in the column vector $\vec{a}$, scale the row vector $\vec{b}$ using $a_i.\vec{b}$ to form the new row of the output matrix.

A Unit Vector is a vector whose magnitude (or length) equals one (1), that is, $\lVert \vec{a} \rVert = 1$.

Given a vector $\vec{a}$ that is of a certain magnitude and in a certain direction, the idea is to find a scalar $\mu$ such that the magnitude of the scaled vector $\lVert \mu \vec{a} \rVert = 1$. A unit vector of a vector $\vec{a}$ is represented as $\hat{a}$ (pronouned a-HAT).

Given $\lVert \mu \vec{a} \rVert = 1$, we can simplify it to derive $\mu = \Large{\frac{1}{\lVert \vec{a} \rVert}}$.

Therefore, a Unit Vector for $\vec{a}$ is given as $\hat{a} = \Large{\frac{1}{\lVert \vec{a} \rVert}}$ $\vec{a} = \Large{\frac{1}{\sqrt{\sum\limits_{i=1}^N\vec{a_i}}}}$ $\vec{a}$.

Let us look at an example now.

Example-2	Find the Unit Vector for the vector $\begin{bmatrix} 3 \\ 4 \end{bmatrix}$
We know a Unit Vector of a given vector $\vec{a}$ is $\hat{a} = \Large{\frac{1}{\lVert \vec{a} \rVert}}$ $\vec{a}$. We also know $\lVert\vec{a}\rVert = \sqrt{\sum\limits_{i=1}^N\vec{a_i}}$ $= \sqrt{3^2 + 4^2} = \sqrt{25} = 5$. RESULT: the Unit Vector for $\begin{bmatrix} 3 \\ 4 \end{bmatrix}$ equals $\Large{\frac{1}{5}}$ $\begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 3/5 \\ 4/5 \end{bmatrix}$

Example-2

Find the Unit Vector for the vector $\begin{bmatrix} 3 \\ 4 \end{bmatrix}$

We know a Unit Vector of a given vector $\vec{a}$ is $\hat{a} = \Large{\frac{1}{\lVert \vec{a} \rVert}}$ $\vec{a}$.

We also know $\lVert\vec{a}\rVert = \sqrt{\sum\limits_{i=1}^N\vec{a_i}}$ $= \sqrt{3^2 + 4^2} = \sqrt{25} = 5$.

RESULT: the Unit Vector for $\begin{bmatrix} 3 \\ 4 \end{bmatrix}$ equals $\Large{\frac{1}{5}}$ $\begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 3/5 \\ 4/5 \end{bmatrix}$

We will use a geometric perspective to find the direction of a vector for a better intuition. A vector $\vec{a}$ in a two dimensional space (since it is easy to visualize) can be represented using a line from the origin $(0, 0)$ to some coordinate point $(x, y)$ and $\theta$ is the angle between the horizontal x-axis and the vector $\vec{a}$.

The following graph is the illustration of the vector:

From Trigonometry, we know $cos(\theta) = \Large{\frac{adjacent}{hypotenuse}}$.

That is, $cos(\theta) = \Large{\frac{x}{\lVert\vec{a}\rVert}}$.

Similarly, $cos(90-\theta) = \Large{\frac{y}{\lVert\vec{a}\rVert}}$.

Therefore, $\hat{a} =$ $($ $\Large{\frac{x}{\lVert\vec{a}\rVert}}$ $,$ $\Large{\frac{y}{\lVert\vec{a}\rVert}}$ $)$.

Extending this concept to a vector $\vec{a}$ of dimensionality N, $\hat{a} = $ $($ $\Large{\frac{a_1}{\lVert\vec{a}\rVert}}$ $,$ $\Large{\frac{a_2}{\lVert\vec{a}\rVert}}$ $, ..., $ $\Large{\frac{a_N}{\lVert\vec{a}\rVert}}$ $)$.

As indicated above, a Dimension is the number of elements in a vector. To get an intuitive understanding, one can look at it from a geometric perspective - a one-dimensional vector represents a line, a 2-dimensional vector represents a plane, a 3-dimensional vector represents a 3-dimensional space (also known as the Euclidean Space), and so on.

A Field is a set of numbers in a particular Dimension, on which, the four basic mathematical operations, namely, addition, subtraction, multiplication, and division are valid. In mathematics, a Field is reprseneted as a hollow capital letter, such as $\mathbb{R}$, which represents a Field of real numbers, etc.

One can associate Dimensionality with a Field by raising the Field letter with the Dimension. For example, the Field $\mathbb{R}^4$ represents a Field of all four-dimensional vectors of the form $\begin{bmatrix} a & b & c &d \end{bmatrix}$. In general, an N-dimensional Field of real number vectors is represented as $\mathbb{R}^N$.

A Subspace is the set of all vectors that can be created by all possible linear combinations of vector-scalar multiplication for a given set of vectors in $\mathbb{R}^N$ and all scalars in $\mathbb{R}$. Subspaces are often indicated using italicized upper-case letters, such as, $\textit{V}$.

In other words, a vector subspace must satisfy the following criteria:

Closure Property - must be closed under addition and scalar multiplication
Origin Property - must contain the Zero vector (a vector with all zero elements, which is equivalent to the origin in a 2-dimensional case)

The simplest linear combination is a scalar multiplication operation (scaling) of a vector. That is, for a given vector $\vec{a}$, one can create the other vectors in the same Subspace by performing a scalar multiplication $\lambda \vec{a}$, where the scalar $\lambda \in \mathbb{R}$. The set of new vectors created from $\lambda \vec{a}$ is said to be in the same Subspace as the original vector $\vec{a}$. Another linear combination could be the addition operation on two or more vectors (from the same Subspace) such as $\lambda\vec{a} + \mu\vec{b}$ or $\lambda\vec{a} - \mu\vec{b}$.

In mathematical terms, if $\vec{a}, \vec{b} \in \textit{V}$ and $\lambda, \mu \in \mathbb{R}$, then $\lambda\vec{a} + \mu\vec{b} \in V$, where $\textit{V}$ is a Vector Subspace.

A set of vectors are said to be Linearly Dependent if some scalar combination of the vectors can form the zero vector (a vector with all zero elements).

Mathematically, $\lambda_1\vec{a_1} + \lambda_2\vec{a_2} + ... + \lambda_n\vec{a_n} = \textbf{0}_n$, where $\lambda_1, \lambda_2, .., \lambda_n \in \mathbb{R}$, where atleast one $\lambda_n \ne 0$.

The following set of vectors from the 3-dimensional space are linearly dependant:

$\vec{a} = \begin{bmatrix} 0 \\ 7 \\ 2 \end{bmatrix}$

$\vec{b} = \begin{bmatrix} 1 \\ 2 \\ 6 \end{bmatrix}$

$\vec{c} = \begin{bmatrix} -2 \\ 17 \\ -6 \end{bmatrix}$

We can show $(-3)\vec{a} + 2\vec{b} + \vec{c} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}$

A Basis is a set of vectors that are linearly independent of each other and can be linearly combined to generate other vectors in the vector space they belong to.

The unit vectors $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$ and $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$ from the 2-dimensional vector space are referred to as the Standard Orthogonal basis vectors.

*** ATTENTION ***

* ATTENTION *