Least squares

Gist

The method for finding the best fit line in Linear regression (LM)

Imagine the simplest case:

Y_{i} = b_{0} + b_{1} X_{i},

and let us define the residuals ( $r_{i}$ ) as:

r_{i} = Y_{i} - (b_{0} + b_{1} X_{i}) .

The sum of squared errors is then:

S S E = \sum_{i = 1}^{N} (r_{i})^{2}

If we want to minimize the SSE, we should then find the derivative in respect to $b_{0}$ and $b_{1}$ :

\frac{\partial S S E}{\partial b_{0}} = 2 (\sum_{i = 1}^{N} Y_{i} - \sum_{i = 1}^{N} b_{0} - \sum_{i = 1}^{N} b_{1} X_{i})

\frac{\partial S S E}{\partial b_{1}} = 2 (\sum_{i = 1}^{N} Y_{i} X_{i} - \sum_{i = 1}^{N} b_{0} X_{i} + \sum_{i = 1}^{N} b_{1} X_{i}^{2})

And if you set it to 0 and solve for $b_{0}$ :

b_{0} = \frac{\sum_{i = 1}^{N} Y_{i}}{N} - \frac{\sum_{i = 1}^{N} b_{1} X_{i})}{N} = \bar{y} - b_{1} \bar{X}

If we look at Multiple regression regression in matrix form

In Maximum likelihood, if one assumes the observation is drawn from a Distribution - Normal, then it is ultimately Least Squares.