Hessian Matrix

Gist

Looking to understand the curvature of a function? Useful for optimization because it gives us the local behavior

Definition

For a scalar-valued function, the Hessian matrix calculates the second partial derivative. It is a matrix of functions that should be evaluated at some points:

Example

Let's say we have the equation below:

Let us calculate the first partial derivative with respect to $x$ and $y$:

and

Then we can take the derivative of $\frac{\partial f}{\partial x}$ with respect to x and y:

as well as with $\frac{\partial f}{\partial y}$ with respect to $x$ and $y$:

and

Therefore the Hessian matrix of $f(x,y)$ is:

We then calculate the Determinant

How to interepret

• Positive definite Hessian → local minimum
• Negative definite Hessian → local maximum
• Indefinite Hessian → saddle point
• Zero determinant → test is inconclusive

Properties

1. Symmetric

Usage

You can figure out if you’re at a maximum/minimum point by calculating the Hessian.

For Maximum-likelihood

As we're trying to maximum the log-likelihood when estimating parameters wtih Maximum likelihood, it makes sense we use the Hessian matrix.

Additionally, the asymptomatic covariance matrix of the parameters is given by the inverse of the negative of the Hessian matrix.

The negative Hessian evaluated at the MLE is the same as the observed Fisher information matrix evaluated at the MLE.

Association with the Taylor Series

The Hessian matrix arises naturally in the second-order Taylor Series of a scalar function.

References