# 2.1 Statistical significance

We need to introduce the concept of ‘statistical significance’ in order to answer this question. The level of statistical significance is the threshold at which we decide whether to reject the null hypothesis (to make a decision). A result is statistically significant when the situation described in the null hypothesis is highly unlikely to have occurred.

By utilising the concept of statistical significance, we can have a concrete way of examining the claim concerning the null hypothesis, using the data we have collected, to make a clear decision on when to reject the null hypothesis and when not to. In this way, we do not have to guess whether the test statistic is too high or too low.

How can we determine the appropriate level of statistical significance to use? Determining the appropriate level of statistical significance is a crucial step in hypothesis testing. To understand this process, we need to introduce the concept of “level of confidence” or “confidence level”. These terms are used interchangeably in statistics and research methodology. The level of confidence represents how certain we are in our decision to reject the null hypothesis.

The level of confidence is closely related to confidence intervals. Typically, decision-makers set confidence level at 95% when establishing a confidence interval. Once we have determined confidence level, we can calculate the level of statistical significance as 1 – level of confidence.

For example, if we choose a 95% level of confidence, the level of statistical significance would be:

1 – 95% = 5% or 0.05

We refer to this level of statistical significance as “alpha” or “α”.

Let us apply this concept to our coffee example. Decision-maker 2 found that the average foam height was 1.5 cm, which differs slightly from the expected value of 1 cm. The question is whether this difference is significant enough to reject or fail to reject the null hypothesis (H0).

To make this decision, decision-makers must consider the alpha level. The alpha level serves as a threshold for statistical significance. It helps determine whether the observed results are likely due to chance or represent a genuine difference between the expected and observed values. Once decision-makers determine the alpha level, they can use it to decide whether to reject or fail to reject the null hypothesis based on Decision-maker 2’s findings. The choice of alpha level is critical, as it directly influences the conclusion drawn from the study.

For instance, if the decision-makers choose an alpha level of 0.05 (corresponding to a 95% confidence level), they would reject the null hypothesis if the probability of obtaining their results by chance is less than 5%. If the probability is greater than 5%, they would fail to reject the null hypothesis.

Choosing the appropriate alpha level requires careful consideration. A lower alpha level (e.g., 0.01) makes it harder to reject the null hypothesis, reducing the risk of false positives but potentially missing real effects. A higher alpha level (e.g., 0.10) makes it easier to reject the null hypothesis, potentially detecting more real effects but also increasing the risk of false positives. The significance level helps us decide whether to reject or fail to reject the null hypothesis when the results are not clearly for or against it. By setting a specific threshold for statistical significance, we can make more objective decisions about our hypotheses based on the collected data.

This approach provides a systematic method for evaluating hypotheses, allowing decision-makers to draw meaningful conclusions from their data. It also helps to standardise the decision-making process across different studies and fields of research, ensuring consistency and comparability in scientific findings. However, it is crucial to remember that statistical significance does not always imply practical significance, and decision-makers should consider the context and real-world implications of their findings alongside the statistical results.

## Activity 2: Level of alpha

Allow around 10 minutes for this activity

Imagine you are working as a marketing analyst for a new product launch, and you want to make sure that your market research is accurate and reliable. You need to determine the appropriate alpha level for your survey results, based on the desired confidence level.

Can you determine α?

## Table 1 Level of alpha

Level of Confidence (C) | α
--- | ---
90% | (A text entry box would appear here, but your browser does not support it.)
95% | (A text entry box would appear here, but your browser does not support it.)
99% | (A text entry box would appear here, but your browser does not support it.)

Interactive feature not available in single page view.

### Discussion

## Table 1 Level of alpha (completed)

Level of Confidence (C) | α
--- | ---
90% | 10%
95% | 5%
99% | 1%

α is calculated as 1 − C.

1 − 90% = 10%

1 − 95% = 5%

1 − 99% = 1%

There is an inverse relationship between confidence levels and alpha levels, as increasing the confidence level leads to a decrease in the alpha level, and vice versa.

OpenLearn - Data analysis: hypothesis testing
Except for third party materials and otherwise, this content is made available under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 Licence, full copyright detail can be found in the acknowledgements section. Please see full copyright statement for details.