Vector Spaces
Definition: Vector Space
A real vector space is a nonempty set \(V\) of vectors on which are
defined addition and scalar multiplication, defined to
satisfy the following eight axioms:
\(\forall u, v, w \in V\) and scalars \(c, d \in \mathbb{R}\),
- Commutativity of vector addition: \(u + v = v + u\)
- Associativity of vector addition: \((u + v) + w = u + (v + w)\)
- Existence of an additive identity: \(\exists 0 \in V\) s.t. \(u + 0 = u\)
- Existence of an additive inverse: \(\forall u \in V, \exists -u\) s.t. \(u + (-u) = 0\)
- Existence of a multiplicative identity: \(1u = u\)
- Distributivity of scalar multiplication with respect to vector addition : \(c(u + v) = cu + cv\)
- Distributivity of scalar multiplication with respect to scalar additon: \((c + d)u = cu + du\)
- Compatibility of scalar multiplication: \(c(du)=(cd)u\)
These axioms abstract the essential algebraic properties of \(\mathbb{R}^n\). By identifying which properties are fundamental,
we can recognize vector space structure in many mathematical objects beyond \(\mathbb{R}^n\) including spaces of functions, polynomials,
matrices, and solutions to differential equations. Any set satisfying these eight axioms will behave algebraically like
\(\mathbb{R}^n\), allowing us to extend linear algebra techniques to these more general settings.
From now on, the term "vector space" refers to a "real" vector space. If the context requires a complex vector space, where
scalars \(c, d \in \mathbb{C}\), it will be explicitly mentioned. To study the general case, you will need concepts from Abstract Algebra,
but for now, this definition suffices. (See: Part 15: Intro to Abstract Algebra.)
A straightforward example of a vector space is \(\mathbb{R}^n\), where \(n \geq 1\). This is why the concept is introduced here.
By the definition, \(\forall u \in V\) and scalar \(c\),
\[0u =0\]
\[c0 =0\]
\[-u =(-1)u\]
\[cu =0 \Longrightarrow c=0 \text{ or } u =0\]
Definition: Vector Subspace
A subspace of a vector space \(V\) is a subset \(H \subseteq V\) that satisfies the
following properties:
- The zero vector of \(V\) is in \(H\)
- \(H\) is closed under vector addition: \(\forall u, v \in H, \quad u+v \in H\)
- \(H\) is closed under scalar multiplication: \(\forall u \in H\) and any scalar \(c \,\), \(cu \in H\)
Warning:
For example, the vector space \(\mathbb{R}^2\) is NOT subspace of \(\mathbb{R}^3\) because \(\mathbb{R}^2\)
is NOT a subset of \(\mathbb{R}^3\).
Recall that a linear combination of vectors
is any sum of scalar multiples of vectors, and \(\text{Span }\{v_1, v_2, \cdots, v_p\}\) represents
the set of all linear combinations of \(v_1, v_2, \cdots, v_p\). Since linear combinations involve
only vector addition and scalar multiplication, and subspaces are closed under these operations,
any span is automatically a subspace. Thus, we obtain the following result:
Theorem 1:
If \(v_1, \cdots, v_p \in V\), then \(\text{Span }\{v_1, \cdots, v_p \}\) is a subspace of \(V\).
Null Space
Having established the general concept of vector spaces and subspaces, we now examine two fundamental subspaces
associated with any matrix \(A\): the null space and the column space.
These subspaces encode essential information about the solutions to \(Ax = b\) and the linear
transformation \(x \mapsto Ax\).
Definition: Null Space
The null space of \(A \in \mathbb{R}^{m \times n} \) is the set of all solutions
to the homogeneous equation \(Ax =0\).
\[
\text{Nul } A = \{x \in \mathbb{R}^n | Ax=0 \in \mathbb{R}^m \}
\]
In other words, \(\forall x \in \text{Nul } A\) are mapped to the zero vector \(0 \in \mathbb{R}^m\)
by the linear transformation \(x \mapsto Ax\). This implies that \(\text{Nul } A\) is a subspace
of \(\mathbb{R}^n\).
When we discuss a general linear transformation \(T: V \mapsto W\) (not necessarily a matrix transformation),
the null space is also referred to as the kernel of \(T\) denoted by \(\text{Ker } T\).
It is defined as:
\[
\text{Ker } T = \{u \in V \mid T(u) = 0 \in W\}.
\]
Example:
Consider the augmented matrix for \(Ax = 0\) and its reduced row echelon form(rref).
\[
\begin{align*}
&\begin{bmatrix} 2 & 4 & 6 & 8 & 0 \\ 1 & 3 & 5 & 7 & 0 \end{bmatrix} \\\\
&\xrightarrow{\text{rref}}
\begin{bmatrix} 1 & 0 & -1 & -2 & 0 \\ 0 & 1 & 2 & 3 & 0 \end{bmatrix}
\end{align*}
\]
\(x_3 \text{ and } x_4\) are free variable, and we can express the solution as:
\[
\begin{bmatrix} x_1\\ x_2\\ x_3 \\ x_4\\ \end{bmatrix}
=
x_3\begin{bmatrix} 1\\ -2\\ 1 \\ 0\\ \end{bmatrix}
+
x_4\begin{bmatrix} 2\\ -3\\ 0 \\ 1\\ \end{bmatrix}
\]
Thus, a spanning set for \(\text{Nul } A\) is
\[\left\{
\begin{bmatrix} 1\\ -2\\ 1 \\ 0\\ \end{bmatrix},
\begin{bmatrix} 2\\ -3\\ 0 \\ 1\\ \end{bmatrix}
\right\}.
\]
This means that every linear combination of these two vectors lies in \(\text{Nul } A\).
Column Space
While the null space identifies vectors that are mapped to zero, the column space identifies
all possible outputs of the linear transformation \(x \mapsto Ax\). Together, these two subspaces
provide complementary perspectives on the behavior of \(A\).
Definition: Column Space
The column space of \(A \in \mathbb{R}^{m \times n} \) is the set of all linear combinations of the
columns of \(A = \begin{bmatrix} a_1 & a_2 & \cdots & a_n \end{bmatrix}\).
\[
\begin{align*}
\text{Col } A &= \text{Span }\{a_1, a_2, \cdots, a_n\} \\\\
&= \{a \in \mathbb{R}^m \mid a = Ax, \, x \in \mathbb{R}^n \}
\end{align*}
\]
Note: \(\text{Col } A\) is a subspace of \(\mathbb{R}^m\) and represents the range of the
linear transformation \(x \mapsto Ax\).
Additionally, \(\text{Col } A^T\) is called the row space of \(A\) and is denoted by \(\text{Row } A\).
The row space is the set of all linear combinations of the row vectors of \(A\) and is a subspace of \(\mathbb{R}^n\).
Basis
Describing a subspace, we want a spanning set that contains no redundant vectors, that is,
a set where each vector contributes essential information. This leads to the concept of a basis,
which provides the most efficient description of a subspace.
Definition: Basis
Let \(H\) be a subspace of a vector space \(V\). An indexed set of vectors
\[
\mathcal{B} = \{b_1, \cdots, b_p\} \text{ in } V
\]
is a basis for \(H\) if \(\mathcal{B}\) is a linearly independent set
and \(H = \text{Span }\{b_1, \cdots, b_p\}\).
For example, the columns of the identity matrix \(I_n\) form the standard basis for \(\mathbb{R}^n\).
Another example is the set of columns of an \(n \times n\) invertible matrix, which forms a basis for \(\mathbb{R}^n\)
because, by the invertible matrix theorem, these columns are linearly independent and span \(\mathbb{R}^n\).
Previously, we found a "spanning" set for \(\text{Nul } A\):
\[
\left\{
\begin{bmatrix} 1\\ -2\\ 1 \\ 0\\ \end{bmatrix},
\begin{bmatrix} 2\\ -3\\ 0 \\ 1\\ \end{bmatrix}
\right\}
\]
This set is actually a basis for \(\text{Nul } A\).
On the other hand, a basis for \(\text{Col } A\) must be a linearly independent spanning set with
no redundant vectors. Consider:
\[
\begin{align*}
A &= \begin{bmatrix} 2 & 4 & 6 & 8 \\ 1 & 3 & 5 & 7 \end{bmatrix} \\\\
&\xrightarrow{\text{ref}}
\begin{bmatrix} 1 & 2 & 3 & 4 \\ 0 & 1 & 2 & 3 \end{bmatrix} = B
\end{align*}
\]
Here, columns \(a_3\) and \(a_4\) are linear combinations of earlier columns:
\[
a_3 = -a_1 + 2a_2, \quad a_4 = -2a_1 + 3a_2.
\]
The pivot columns of \(B\) are linearly independent, and because \(A\) is row equivalent
to \(B\), the pivot columns of \(A\) (namely \(a_1\) and \(a_2\)) form a basis for \(\text{Col } A\).
Thus, the basis for \(\text{Col } A\) is:
\[
\left\{
\begin{bmatrix} 2 \\ 1 \end{bmatrix}, \,
\begin{bmatrix} 4 \\ 3 \end{bmatrix}
\right\}.
\]
Also, the representation of \(a_3\) and \(a_4\) in terms of \(a_1\) and \(a_2\) is unique.
This fact is formalized in:
Theorem 2:
Let \(\mathcal{B} = \{b_1, \cdots, b_n\}\) be a basis for a vector space \(V\).
Then, there exists a unique set of scalars \(c_1, \cdots, c_n\) such that:
\[
x = c_1b_1 + \cdots + c_nb_n.
\]
Proof:
Since \(\mathcal{B}\) spans \(V\), there exist scalars such that
\[
x = c_1b_1 + \cdots + c_nb_n \tag{1}
\]
Assume that \(x\) also has another representation with different scalars \(d_1, \cdots, d_n\):
\[
x = d_1b_1 + \cdots + d_nb_n \tag{2}
\]
Subtracting (2) from (1):
\[
\begin{align*}
x - x &= (c_1 - d_1)b_1 + \cdots \\\\
&\quad + (c_n-d_n)b_n
\end{align*}
\]
To hold this equation, since \(\mathcal{B}\) is linearly independent, each coefficient
must satisfy \(c_i - d_i = 0\) \((1 \leq i \leq n)\).
Thus, \(c_i = d_i\). In other words, the representation is unique.
Coordinate Systems
The uniqueness guaranteed by Theorem 2 means that once we fix a basis \(\mathcal{B}\) for a vector space \(V\),
each vector \(x \in V\) corresponds to exactly one coordinate vector in \(\mathbb{R}^n\). This establishes a
coordinate system for \(V\) relative to \(\mathcal{B}\).
Definition: Coordinate Vector
Let \(\mathcal{B} = \{b_1, \cdots, b_n\}\) be a basis for a vector space \(V\) and \(x \in V\).
The \(\mathcal{B}\)-coordinates of \(x\) are weights \(c_1, \cdots, c_n \) such that:
\[
x = c_1b_1 + \cdots + c_nb_n.
\]
The vector
\[
[x]_{\mathcal{B}} = \begin{bmatrix} c_1\\ \cdots \\ c_n \\ \end{bmatrix}
\]
is called the coordinate vector of \(x\).
The mapping \(x \mapsto [x]_{\mathcal{B}}\) is the coordinate mapping, which is
a one-to-one linear transformation from \(V\) onto \(\mathbb{R}^n\).
The coordinate mapping is an isomorphism from \(V\) onto \(\mathbb{R}^n\).
It allows us to analyze an "unfamiliar" vector space \(V\) via the "familiar" vector space \(\mathbb{R}^n\),
preserving all operations and structures.
Rank
The dimension of a subspace is a fundamental invariant that measures the "size" of the subspace.
For matrix-associated subspaces, dimension has special significance and is formalized through the
concept of rank.
Definition: dimension & Rank
The dimension of a vector space \(V\) is the number of vectors in a basis for \(V\).
The rank of a matrix \(A\) is the dimension of the column space of \(A\).
\[
\text{rank }A = \dim (\text{Col } A)
\]
The rank equals the number of pivot columns of \(A\).
Note that since
\[
\text{Row } A = \text{Col } A^T
\],
we get
\[
\dim \text{Row } A = \text{rank }A^T.
\]
Additionally, \(\dim \text{Nul } A \) is the number of free variables in \(Ax = 0\). Equivalently,
it is the number of non-pivot columns of \(A\). Thus, the following relationship holds:
Theorem 3:
For an \(m \times n\) matrix \(A\),
\[
\text{rank }A + \dim (\text{Nul } A )= n.
\]