2.1 Statistical significance

We need to introduce the concept of ‘statistical significance’ in order to answer this question. The level of statistical significance is the threshold at which we decide whether to reject the null hypothesis (to make a decision). A result is statistically significant when the situation described in the null hypothesis is highly unlikely to have occurred.

By utilising the concept of statistical significance, we can have a concrete way of examining the claim concerning the null hypothesis, using the data we have collected, to make a clear decision on when to reject the null hypothesis and when not to. In this way, we do not have to guess whether the test statistic is too high or too low.

How can we determine the appropriate level of statistical significance to use? Determining the appropriate level of statistical significance is a crucial step in hypothesis testing. To understand this process, we need to introduce the concept of “level of confidence” or “confidence level”. These terms are used interchangeably in statistics and research methodology. The level of confidence represents how certain we are in our decision to reject the null hypothesis.

The level of confidence is closely related to confidence intervals. Typically, decision-makers set confidence level at 95% when establishing a confidence interval. Once we have determined confidence level, we can calculate the level of statistical significance as 1 – level of confidence.

For example, if we choose a 95% level of confidence, the level of statistical significance would be:

1 – 95% = 5% or 0.05

We refer to this level of statistical significance as “alpha” or “α”.

Let us apply this concept to our coffee example. Decision-maker 2 found that the average foam height was 1.5 cm, which differs slightly from the expected value of 1 cm. The question is whether this difference is significant enough to reject or fail to reject the null hypothesis (H₀).

To make this decision, decision-makers must consider the alpha level. The alpha level serves as a threshold for statistical significance. It helps determine whether the observed results are likely due to chance or represent a genuine difference between the expected and observed values. Once decision-makers determine the alpha level, they can use it to decide whether to reject or fail to reject the null hypothesis based on Decision-maker 2’s findings. The choice of alpha level is critical, as it directly influences the conclusion drawn from the study.

For instance, if the decision-makers choose an alpha level of 0.05 (corresponding to a 95% confidence level), they would reject the null hypothesis if the probability of obtaining their results by chance is less than 5%. If the probability is greater than 5%, they would fail to reject the null hypothesis.

Choosing the appropriate alpha level requires careful consideration. A lower alpha level (e.g., 0.01) makes it harder to reject the null hypothesis, reducing the risk of false positives but potentially missing real effects. A higher alpha level (e.g., 0.10) makes it easier to reject the null hypothesis, potentially detecting more real effects but also increasing the risk of false positives. The significance level helps us decide whether to reject or fail to reject the null hypothesis when the results are not clearly for or against it. By setting a specific threshold for statistical significance, we can make more objective decisions about our hypotheses based on the collected data.

This approach provides a systematic method for evaluating hypotheses, allowing decision-makers to draw meaningful conclusions from their data. It also helps to standardise the decision-making process across different studies and fields of research, ensuring consistency and comparability in scientific findings. However, it is crucial to remember that statistical significance does not always imply practical significance, and decision-makers should consider the context and real-world implications of their findings alongside the statistical results.