Least Squares Line Fitting · Fundamentals of AI/ML

Given n points (x_i, y_i), the ordinary least squares (OLS) line minimises the sum of squared vertical distances from each point to the line. The minimiser has a closed form: β̂ = (X^⊤X)⁻¹X^⊤y. Click to add data points and watch the line — and the matrix arithmetic — update live.

data points OLS fit residuals mean (x̄, ȳ)

Click anywhere to add a point. Click on an existing point to remove it.

Display options

Residuals Mean (line passes through) Squared residuals (boxes) Confidence band (±1σ)

Fitted line

n = 0 points

—

Slope m = —

Intercept c = —

Error metrics

SSR = Σr_i² = —

RMSE = √(SSR/n) = —

R² = 1 − SSR/SST = —

Normal equations

X^⊤X =

X^⊤y =

β̂ = (X^⊤X)⁻¹X^⊤y =

What to notice during class

The OLS line passes through the centroid (x̄, ȳ). Always. This is a one-line consequence of the normal equation for the intercept.
Residuals are vertical, not perpendicular. OLS minimises vertical distance — that's why it changes if you swap x and y. (Total least squares would minimise perpendicular distance — different problem.)
One outlier destroys the fit. Try the "Linear + outlier" preset. The single far-away point swings the entire line because squared error penalises it enormously.
R² doesn't mean the model is correct. Try the "Quadratic" preset: R² may still be high, but the line is wrong — the residuals show a U-shape, not random scatter. This is the visual diagnostic.
The matrix formulation generalises. The same β̂ = (X^⊤X)⁻¹X^⊤y holds for multivariate regression, polynomial fitting, and beyond. Only the design matrix X changes.
Connection to projection. X(X^⊤X)⁻¹X^⊤ is the hat matrix H — the projection onto the column space of X. This is exactly the MTE Q7 idempotent matrix.