Part 2 — A, Aᵀ and AᵀA

1) Geometric view of transformations by A, Aᵀ & AᵀA

Geometric view of transformations by A, Aᵀ & AᵀA

We are looking at the same 3×2 matrix A = [ a⃗₁ | a⃗₂ ]
with linearly independent a⃗₁ and a⃗₂ in ℝ³ and rank(A) = 2

Transformations by A, followed by Aᵀ

Direct transformation by AᵀA

Note changing relationship between a⃗₁ & a⃗₂

They start as orthogonal
After transformation by A the grid loses squareness
AᵀA brings this non-square grid back to ℝ² and the final AᵀA grid looks more distorted than the grid seen in the embedded plane in ℝ³

This is because AᵀA applies the same distortion twice: once to embed the grid via A and once pull that geometry back via Aᵀ

Suppose a⃗₁ & a⃗₂ are orthogonal with length = 1
In this case AᵀA is identity and the grid returns to original state

This is how to measure “grid distortion” by A

Suppose we have vectors x⃗ & y⃗ in the original plane, so their dot product = x⃗ᵀ · y⃗
After transformation by A, dot product becomes
(A x⃗)ᵀ · (A y⃗) = x⃗ᵀ AᵀA · y⃗ = x⃗ᵀ · (AᵀA y⃗)

2) The four subspaces of Aᵀ

The four subspaces of Aᵀ

We now apply the same construction to the transpose matrix Aᵀ:
rows of Aᵀ are columns of A and columns of Aᵀ are rows of A

Space	Lives in	Dim	Spanned by	⊥ Complement
row space(Aᵀ)	ℝᵐ	r	Rows of Aᵀ (Cols of A)	null space(Aᵀ)
null space(Aᵀ)	ℝᵐ	m−r	all x⃗ satisfying Aᵀ x⃗ = 0	row space(Aᵀ)
col space(Aᵀ)	ℝⁿ	r	Cols of Aᵀ (Rows of A)	ℓ-null space(Aᵀ)
ℓ-null space(Aᵀ)	ℝⁿ	n−r	all b⃗ satisfying A b⃗ = 0	col space(Aᵀ)

Summary table and proofs connecting null spaces of A and Aᵀ

Connecting the null spaces of A and Aᵀ

① null(A) = ℓ-null(Aᵀ)

Proof
Let x⃗ be a vector in ℝⁿ

Recall the transpose identity for matrix-vector multiplication:
(A x⃗)ᵀ = x⃗ᵀ Aᵀ

Now prove set inclusion in both directions

If x⃗ ∈ null(A), then A x⃗ = 0
Taking transpose gives (A x⃗)ᵀ = 0ᵀ = 0
Using the identity above, x⃗ᵀ Aᵀ = 0
So x⃗ ∈ ℓ-null(Aᵀ)
If x⃗ ∈ ℓ-null(Aᵀ), then x⃗ᵀ Aᵀ = 0
Taking transpose gives (x⃗ᵀ Aᵀ)ᵀ = 0
But (x⃗ᵀ Aᵀ)ᵀ = A x⃗
So A x⃗ = 0, hence x⃗ ∈ null(A)

Since each set is contained in the other, the two sets are equal

② ℓ-null(A) = null(Aᵀ)

Proof
Let y⃗ be a vector in ℝᵐ

Recall the transpose identity:
(y⃗ᵀ A)ᵀ = Aᵀ y⃗

If y⃗ ∈ ℓ-null(A), then y⃗ᵀ A = 0
Taking transpose gives Aᵀ y⃗ = 0
So y⃗ ∈ null(Aᵀ)
If y⃗ ∈ null(Aᵀ), then Aᵀ y⃗ = 0
Taking transpose gives y⃗ᵀ A = 0
So y⃗ ∈ ℓ-null(A)

Thus the two spaces are also equal

3) The four subspaces of AᵀA

The four subspaces of AᵀA

Consider any m×n matrix A

AᵀA is n×n matrix, which maps ℝⁿ to ℝⁿ
↔ Domain of AᵀA is identical to its codomain
AᵀA is symmetric: (AᵀA)ᵀ = Aᵀ(Aᵀ)ᵀ = AᵀA

① null(AᵀA) = null(A)

Suppose vector x⃗ ∈ null space(A), or A x⃗ = 0⃗
Then (AᵀA) x⃗ = Aᵀ(A x⃗) = 0⃗
↓
null space(A) ⊆ null space(AᵀA)

To prove the reverse inclusion, suppose vector y⃗ ∈ null space(AᵀA), or (AᵀA) y⃗ = 0⃗

Multiply on the left by y⃗ᵀ:
y⃗ᵀ(AᵀA)y⃗ = 0
y⃗ᵀ(AᵀA)y⃗ = (A y⃗)ᵀ(A y⃗) = ‖A y⃗‖²
So ‖A y⃗‖² = 0 ⇒ A y⃗ = 0⃗
↓
null space(AᵀA) ⊆ null(A)

Together:
null space(AᵀA) = null space(A)

② AᵀA is symmetric
↓
• row space(AᵀA) = col space(AᵀA)
• ℓ-null space(AᵀA) = null space(AᵀA)
↓
four subspaces of A reduce to two

Space	Lives in	Dim	Spanned by	⊥ Complement
row space(AᵀA) = col space(AᵀA)	ℝⁿ	r	rows (same as cols) of AᵀA	null space(AᵀA) = ℓ-null space(AᵀA)
null space(AᵀA) = ℓ-null space(AᵀA) = null space(A)	ℝⁿ	n−r	all x⃗ satisfying (AᵀA) x⃗ = 0⃗ or x⃗ᵀ (AᵀA) = 0⃗ᵀ	row space(AᵀA) = col space(AᵀA)

Summary table for the two distinct subspaces of AᵀA

Suppose A is a “tall” matrix with m > n and full column rank
↓
AᵀA has full rank = n
↓
(AᵀA) x⃗ = Aᵀb⃗ has a single solution
Importance of this concept will be illustrated on the next page