4.3.2 Statistical errors, confidence and power

Type I error (whose probability is commonly noted as α) is where our findings reject the null hypothesis (no difference) when in fact it is true. In other words, a false positive. By convention, the threshold for α is usually set at 5%, implying it is acceptable to have a 5% chance of rejecting the null hypothesis when it is true. Type II error (whose probability is commonly noted as β) is where our findings uphold the null hypothesis when it is not true. In other words, a false negative. By convention, the threshold for β is usually set at about 20%. The difference in probabilities of a false negative and a false positive relates to the burden of evidence required to reject the null hypothesis. For example, it is more important to exclude false positives when making claims such as one treatment being better than another (if that is not true), than to exclude false negatives such as there is no difference between two treatments (even when there is).

By contrast, the two other cells in Table 5 illustrate when correct inferences are made: accepting the null hypothesis when it is true or rejecting it when false. Confidence (1-α) is the probability of retaining the null hypothesis when it is true. In this case, we are accepting that the observed difference in the data is due to chance only. Statistical power (1-β) is the probability of finding an effect or association where one exists. Typically, as statistical power increases, confidence decreases. The only way to increase both power and confidence is to increase the sample size. Small sample sizes are often the reason for failing to detect an association when it is there (lack of statistical power).

Hint: Does this sound familiar? If you’ve completed one of the Sampling modules, you were introduced to the concepts of confidence and power. Now, you can see that confidence level, which is usually set as 95%, is 1 – α, and power, which is usually set at 80%, is 1 – β.