Linear regression

Gist

A model to describe the linear relationship between an independent variable and a predictor. The bread butter of statistical analysis.

Mathematics

Standard

For the simplest linear regression:

Y_{i} = b_{0} + b_{1} X_{i} + ϵ

where $b_{0}$ is the intercept and $b_{1}$ is the slope parameter that describes how much a unit increase in X would then lead to an increase in Y. Finally, $ϵ$ is the Error term where the errors are expected to be drawn from a Distribution - Normal. If errors are not normally distributed, one can use Generalized linear models (GLM).

Matrix notation

We can also present the regression in the matrix form:

[\begin{matrix} y_{1} \\ ⋮ \\ y_{n} \end{matrix}] = [\begin{matrix} 1 & x_{1} \\ ⋮ & ⋮ \\ 1 & x_{n} \end{matrix}] [\begin{matrix} β_{0} \\ β_{1} \end{matrix}] + [\begin{matrix} ϵ_{1} \\ ϵ_{2} \end{matrix}]

Key Assumptions

Linearity in terms of the coefficients: so the addition of Polynomial (linear regression) terms such as $X^{2}$ is valid for a linear regression. The effect on Y of a one unit change in X does not depend on the level of X.
The Error term at each value of the predictor is normally distributed: though this assumption can be broken if the model is used more for prediction than inference. Check Advice - Regresion
The variance of the error is constant, if not than we consider the model to have Homoscedasticity-Heteroscedasticity. Though Gelman suggests it's not that big of a deal and one can also use weighted regression ^[1]
Errors are independent (i.e there's no Autocorrelation)

Parameterization

We can use the Least squares using the calculus method^[2] or the linear algebra method (see the page for derivation)

Interpretation

Diagnostics

Check the Residuals

References

Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge university press. ↩︎
3 ↩︎