Linear regression

Gist

The most basic technique and the most powerful. Learn it once, and keep learning because you learn something new each time.

Mathematics

Standard format

For the simplest linear regression:

Y_{i} = b_{0} + b_{1} X_{i} + ϵ

where $b_{0}$ is the intercept and $b_{1}$ is the slope parameter that shows how much a unit increase in X would then lead to an increase in Y. The $ϵ$ is the Error term where it is expected to be drawn from a Distribution - Normal.

Matrix notation

We can also present the regression in the matrix form:

[\begin{matrix} y_{1} \\ ⋮ \\ y_{n} \end{matrix}] = [\begin{matrix} 1 & x_{1} \\ ⋮ & ⋮ \\ 1 & x_{n} \end{matrix}] [\begin{matrix} β_{0} \\ β_{1} \end{matrix}] + [\begin{matrix} ϵ_{1} \\ ϵ_{2} \end{matrix}]

Assumptions

Linearity in terms of the coefficients: so the addition of Polynomial (linear regression) such as $X^{2}$ is valid for a linear regression.
The Error term at each value of the predictor is normally distributed- though this assumption can be broken if model is used more for prediction than inference (see: )
1. Many people erroneously assume that the response/predictor values must be normal though simulation studies showed that ultimately it doesn't matter if you did it:
  "Although this study strongly recommends the appropriate use of normality tests in linear modelling—which is to evaluate the residuals and not the raw data for normality—our simulations also show that if a normality test is applied to raw data, the subsequent choice of a parametric or non-parametric test has little difference in power." (Midway and White, 2025)
Variance is constant, if not than we consider the model to have Heterodeskity
Errors are independent (i.e there's no Autocorrelation)

Parameterization

We use Least squares estimates where you can derive it from:

the matrix form $(X^{T} X)^{- 1} X^{^{'}} Y$

Interpretation

I realized that even the r

With Interactions

Note that it is not necessary to “hold constant” all other variables to be able to interpret the effect of one predictor. It is sufficient to hold constant the weighted sum of all the variables other than $X_{j}$ . And in many cases it is not physically possible to hold other variables constant while varying one, e.g., when a model contains X and X2

Coding

R

lm()

Python

Resources

Midway, S., & White, J. W. (2025). Testing for normality in regression models: mistakes abound (but may not matter). Royal Society Open Science, 12(4), 241904.