Projection Formula Derived from the Definition

Deriving orthogonal projection by requiring the residual to be perpendicular to the projection subspace

Orthogonal projection is defined by a decomposition v⃗ = v⃗⊥ + v⃗∥. The projected part v⃗∥ lies in the chosen direction or subspace, while the residual v⃗⊥ is perpendicular to it.

Those two conditions determine the familiar formulas. The one-vector case and the column-space case are the same derivation with a scalar coefficient replaced by a coordinate vector.

Explore more on GraphMath Linear Algebra

Projection onto the direction of one vector

Derivation of the orthogonal projection of vector v onto the direction of vector u from the condition that the residual is perpendicular to u.
The projection has the form v⃗∥ = s u⃗. Requiring u⃗ · (v⃗ − v⃗∥) = 0 determines s and therefore determines both the projected vector and the residual. Click the image to open the full-size version.

Projection onto the column space of U

Derivation of the orthogonal projection of vector v onto the column space of U from the condition that the residual is perpendicular to every column of U.
For a full-column-rank matrix U, the projected vector has the form v⃗∥ = Uc⃗. The residual condition becomes Uᵀ(v⃗ − Uc⃗) = 0⃗, which yields the normal equations and the projection matrix. Click the image to open the full-size version.

Concept

Projection onto one vector is the one-dimensional case. The unknown coefficient is a scalar s, and the projected vector is a multiple of u⃗.

Projection onto col(U) uses several independent spanning directions. The unknown coefficients form c⃗, and Uc⃗ is the vector in the subspace selected by the orthogonality condition.

Structure

In both derivations, the projected vector is first written in the required subspace. Then the residual is formed and required to be orthogonal to that subspace. Solving the resulting equation determines the projection.

This is why the formulas are structurally parallel: a dot product with u⃗ in the one-dimensional case becomes left multiplication by Uᵀ in the general case.

Key equations

v⃗ = v⃗⊥ + v⃗∥
v⃗∥ = s u⃗
u⃗ · (v⃗ − s u⃗) = 0
s = (u⃗ · v⃗) / (u⃗ · u⃗)
v⃗∥ = ((u⃗ · v⃗) / (u⃗ · u⃗)) u⃗
v⃗∥ = U c⃗
Uᵀ(v⃗ − U c⃗) = 0⃗
(UᵀU)c⃗ = Uᵀv⃗
c⃗ = (UᵀU)⁻¹Uᵀv⃗
v⃗∥ = U(UᵀU)⁻¹Uᵀv⃗
P = U(UᵀU)⁻¹Uᵀ

The matrix formula assumes that the columns of U are linearly independent, so UᵀU is invertible.

Query phrases

  • projection formula derived from the definition
  • derive orthogonal projection formula
  • why projection formula works
  • projection onto a vector derivation
  • projection onto a subspace derivation
  • derive P = U(U transpose U) inverse U transpose
  • orthogonal residual projection formula
  • normal equations from projection