Multiverse analysis
One way to take robustness to its most extreme is to perform ‘multiverse analysis’. Although this sounds like something out of Dr Who - a science fiction TV show featuring intrepid explorers of time and space - it’s actually a lot less out-of-this-world (however, still very cool). Multiverse analysis is where researchers try to perform all possible reasonable analyses on the data, in order to explore which analyses show the effect they’re interested in and which don’t.
Activity 3:
Allow 15 minutes
Using the hack your way to scientific glory [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)] activity from Week 3, try to write down as many different combinations of analysis parameters and the results you get from them (perform and write down as many as you can in fifteen minutes). In effect, you will be doing a mini-multiverse analysis. What does it mean that the results are so variable, depending on the analysis performed?
When you are ready, press 'reveal' to see our comments.
Discussion
There are actually NINE HUNDRED different analysis combinations you could use in this activity! Conducting a full multiverse analysis on data like these would take a very long time, and go beyond the scope of the usual research project. But perhaps thinking about some of the most important robustness checks you could do with your research could be a first step towards enhanced robustness, without exploring the entire multiverse.
Robustness