Covariance

Definition

The covariance is a measure of the joint variability between two variables. When the two variables show similar behaviors, then the covariance is positive. When the two variables show opposite behaviors, then the covariance is negative.

Mathematically, the covariance between variables $X$ and $Y$ can be described as:

This can also be expressed (through the linearity property of Expectation):

Fun fact, you can also do it for Imaginary numbers.

Properties

1. Location invariance: $Cov(x,y)=Cov(x-a,y)$
2. Multilinearity: $\lambda Cov(x,y)=Cov(\lambda x,y)$
3. Symmetry: $Cov(x,y)=Cov(y,x)$

Notes:

• Covariance is sensitive to the scale of the variables. Large values can inflate the result.
• Covariance does not indicate the strength of the relationship in a standardized way.
• Outliers can significantly affect the covariance calculation.
• A correlation is just standardized covariance. For the Pearson correlation, for example, it is a ratio of the covariance matrix to the square root of the variables variances: