Multiverse analysis

One way to take robustness to its most extreme is to perform ‘multiverse analysis’. Although this sounds like something out of Dr Who - a science fiction TV show featuring intrepid explorers of time and space - it’s actually a lot less out-of-this-world (however, still very cool). Multiverse analysis is where researchers try to perform all possible reasonable analyses on the data, in order to explore which analyses show the effect they’re interested in and which don’t.

The image shows a distant galaxy, surrounded by stars.

Activity 3:

Allow 15 minutes

Using the hack your way to scientific glory activity from Week 3, try to write down as many different combinations of analysis parameters and the results you get from them (perform and write down as many as you can in fifteen minutes). In effect, you will be doing a mini-multiverse analysis. What does it mean that the results are so variable, depending on the analysis performed?

To use this interactive functionality a free OU account is required. Sign in or register.
Interactive feature not available in single page view (see it in standard view).

When you are ready, press 'reveal' to see our comments.

Discussion

There are actually NINE HUNDRED different analysis combinations you could use in this activity! Conducting a full multiverse analysis on data like these would take a very long time, and go beyond the scope of the usual research project. But perhaps thinking about some of the most important robustness checks you could do with your research could be a first step towards enhanced robustness, without exploring the entire multiverse.

Robustness

The open research decision tree