Limits to replication

There are fields and methodologies where the value of replication is hotly debated. For instance:

  • Some argue that replication should be encouraged in qualitative research, whereas others argue that there are still open questions about whether replication is possible, desirable, or even aligned with the fundamental principles of qualitative research.
  • Economics has had a long history with replication studies, but not under this name. In economics, replication often takes place as ‘robustness checks’, where researchers test if their results hold when they use different datasets.
  • Research in the humanities is primarily interpretive and context-specific, focusing on understanding human experiences, cultures, texts, and historical events. This interpretive nature makes exact replication more challenging.

  

It is important to think carefully about whether replication makes sense for your field and methodology.

If you are working in a field where replication is important, and if your study replicates the one you are trying to replicate, you can be pretty confident about the result.

But what does it mean if, like Priya’s first attempts, your study does not replicate? One explanation could be that the original result was a ‘false positive’, and so the failed replication is a ‘true negative’. Another explanation is that the replication result was a ‘false negative’, and that the original study was a ‘true positive’. It’s also possible that differences between the two studies are responsible for the different results.

  

Activity 1:

Allow about 20 minutes

This activity relates to our examples of typical direct and conceptual replication studies. By way of reminder:

  • A researcher finds a surprising finding in their research. To test whether they should rely on this result, they conduct a replication immediately after, using all the same materials and the same participant pool. This is a direct replication.
  • A researcher wants to replicate a study they’ve read about. They don’t think the original study was well-designed, but they think the hypothesis is interesting, so they design a new study testing the same hypothesis but in a different way. This is a conceptual replication.

Now imagine these two researchers both carry out their studies. List the reasons why each of these two researchers may not replicate the original result.

To use this interactive functionality a free OU account is required. Sign in or register.
Interactive feature not available in single page view (see it in standard view).

When you are ready, press 'reveal' to see our comments.

Discussion

You might have listed:

  • The original result was a false positive
  • The replication result is a false negative
  • There are important differences between the original study and the replication study:
    • a.These could be small changes that researchers didn’t think should be important but that turned out to be (e.g.: which brand of a specific chemical was used).
    • b.It could be that the replication researcher didn’t realise these were differences because there wasn’t enough detail in the original paper to be able to work out how everything had been done.
    • c.The replication researcher might know they’re making a change from the original protocol, but approve this change because theoretically it shouldn’t make a difference to the result.

Replication studies

The replication crisis