2 Correlation
To see if life expectancy grows when the GDP increases I will use a statistical measure known as the Spearman rank correlation coefficient.
It’s a number between -1 and 1 that describes how well two indicators correlate, in the following sense.
- A value of 1 means that if I rank (sort) the data from smallest to largest value in one indicator, it will also be in ascending order according to the other indicator. In other words, if one indicator grows, so does the other.
- A value of -1 means a perfect inverse rank relation: if I sort the data from smallest to largest according to one indicator, I will see it is sorted from largest to smallest in the other indicator. When one indicator goes up, the other goes down.
- A value of 0 means there is no rank relation between the two indicators.
A positive value smaller than 1 (or a negative value larger than -1) means there is some direct (or inverse) correlation, but it is not systematic across the whole dataset.
The p-value indicates how significant the result is, in a particular technical sense. To say a correlation is statistically significant doesn’t necessarily mean it is important or strong in the real world, but only that there is reasonable statistical evidence that there is some kind of relationship. Typically, the obtained correlation coefficient is considered statistically significant if the p-value is below 0.05.
The pandas module doesn’t calculate complex statistics. There are other modules in the Anaconda distribution for that. In particular, scipy (Scientific Python) has a stats module that provides the spearmanr() function. The function takes as arguments the two columns of data to correlate. Contrary to the functions you’ve seen so far, it returns two values instead of one: the correlation and the p-value. To store both values, simply use a pair of variables, written in parenthesis.
To show the results in a nicer way, I will use the Python print() function, which displays its arguments in a single line.
In []:
from scipy.stats import spearmanr
gdpColumn = gdpVsLife[GDP]
lifeColumn = gdpVsLife[LIFE]
(correlation, pValue) = spearmanr(gdpColumn, lifeColumn)
print('The correlation is', correlation)
if pValue
print('It is statistically significant.')
else:
print('It is not statistically significant.')
Out[]:
The correlation is 0.493179132478.
It is statistically significant.
Although there is a statistically significant direct correlation (life expectancy grows as GDP grows), it isn’t strong.
A perfect (direct or inverse) correlation doesn’t mean there is any cause-effect between the two indicators. A perfect direct correlation between life expectancy and GDP would only state that the higher the GDP, the higher the life expectancy. It would not state that the higher expectancy is due to the GDP. Correlation is not causation.
Exercise 10 Correlation
Calculate the correlation between GDP and population in Exercise 10 in the Exercise notebook 3.
Remember to run the existing code in the notebook before you start the exercise. When you’ve completed the exercise, save the notebook.