Least squares in this setting means finding the vector ŷ = Xβ̂ in the column space of a 3 × 2 design matrix X that is closest to the data vector y ∈ ℝ³. Geometrically, col(X) is a plane in ℝ³, ŷ is the orthogonal projection of y onto that plane and the residual r = y − ŷ is perpendicular to the plane.
Projection onto V = col(A) can be written as a sum of outer products when the columns of A are orthonormal. The left diagram shows the component projections of y onto the basis directions, and the right diagram shows their head-to-tail sum, which constructs proj⟨V⟩ y.
With three data values and two parameters, the design matrix has the form X ∈ ℝ3×2, the parameter vector is β ∈ ℝ² and the data vector is y ∈ ℝ³. Every candidate fit has the form Xβ, so all fitted vectors lie in col(X), a subspace of ℝ³ of dimension at most 2.
If the two columns of X are independent, then col(X) is a plane. Least squares chooses the point of that plane that minimizes the Euclidean distance to y.
The ambient space is determined by the number of observations, not by the number of parameters. Three observed values produce a data vector with three coordinates, so y lives in ℝ³. Two parameters produce a two-dimensional model subspace inside that ambient space, which is why the fit can be drawn as projection onto a plane in ℝ³.
External reference: Wikipedia — Ordinary least squares, Projection section
Related chapter: Projection
Related work: GraphMath Linear Algebra
Contact: info@graphmath.com