Linear regression

Gist

A model to describe the linear relationship between an independent variable and a predictor.

Mathematics

Standard

For the simplest linear regression:

where $b_0$ is the intercept and $b_1$ is the slope parameter that describes how much a unit increase in X would then lead to an increase in Y. The $ϵ$ is the Error term where it is expected to be drawn from a Distribution - Normal. If errors are not normally distributed, one can use Generalized linear models.

Matrix notation

We can also present the regression in the matrix form:

Key Assumptions

1. Linearity in terms of the coefficients: so the addition of Polynomial (linear regression) terms such as $X^2$ is valid for a linear regression.

2. The Error term at each value of the predictor is normally distributed: though this assumption can be broken if model is used more for prediction than inference:^[1]

   "Although this study strongly recommends the appropriate use of normality tests in linear modelling—which is to evaluate the residuals and not the raw data for normality—our simulations also show that if a normality test is applied to raw data, the subsequent choice of a parametric or non-parametric test has little difference in power." ^[2]

3. The variance of the error is constant, if not than we consider the model to have Heterodeskity. Though Gelman suggests it's not that big of a deal and one can also use weighted regression ^[3]

4. Errors are independent (i.e there's no Autocorrelation)

Parameterization

We can use the Least squares using the calculus method^[4] or the linear algebra method (see the page for derivation)

Interpretation

Diagnostics

Check the Residuals

###References