Avoiding the pitfalls

Sometimes academic journals have specific word counts, and it can be difficult to fit a lot of nuance and several additional analyses into the body of the paper. Where this is the case, it can be helpful to include all this information as supplementary information accompanying the paper (e.g. as a document uploaded to the Open Science Framework) rather than in the paper itself.

Preregistration, which you learned about in Week 4, can also help you to avoid the pitfalls described on the previous page. If you plan to use p-values to make conclusions about whether your statistical tests are significant or not, you will need to outline a significance threshold in your preregistration. You will also need to outline how you will interpret different results, including whether they will support a specific theory.

Preregistration can also protect you from burying results that don’t support your conclusions: if you preregister that you will run certain analyses, you will need to report the results of these, regardless of what they were.

Importantly, without anyone checking your preregistration, you could still make decisions in the preregistration that make your results less credible. For example, you could pick a very high significance threshold (e.g. p < 0.1), which would make false positive results more likely. You could say that a result supports a theory that doesn’t make sense, or you could ‘bury’ additional analyses that negate the results you’d prefer to draw attention to, if they weren’t included in your preregistration. Preregistration doesn’t automatically give your work more integrity, but it can help you to think through your research decisions more clearly before you start, and stop you from tricking yourself later.

Activity 2:

Allow about 30 minutes