Robustness

Robustness refers to the strength and reliability of results. Results can be considered more robust (and therefore have more integrity) if they hold up under various conditions, e.g. different data analyses. When results are robust to different data analyses, this indicates that the conclusions drawn from the research are not overly dependent on specific features of one type of analysis, and are therefore likely more widely applicable.

The two images show a ruined wooden bridge on the left and a cast iron bridge on the right. The bridge on the left is likely to be less robust to extreme weather events than the one on the right
The bridge on the left is likely to be less robust to extreme weather events than the bridge on the right.

In some fields, it’s common to run many robustness analyses. For example, in economics, it is typical to have dozens of pages of any paper showing that a particular result holds up no matter how you measure the variables, which participants you include, which statistical model you use, and even when you control for a variety of factors that could be an alternative explanation for the effect.

However, in other fields it’s less common. For example, in psychology, papers are often published where only one key analysis is performed to examine the results. If data and materials aren’t shared openly, it means others outside of the original research team cannot even choose to run these analyses themselves to check the robustness of the results. This is a good example of how integrity is difficult to check without transparency.

Avoiding the pitfalls

Multiverse analysis