Replication studies

Hi everyone, my name’s Priya Silverstein and I'm a post-doctoral researcher for the Psychological Science Accelerator, and I'm also the author for this course. My pronouns are ‘they, them’.

As part of my PhD, I ran my first replication study. It wasn't meant to be a big part of my PhD, but it ended up being one of the biggest parts!

I thought that before starting any of my own original research, it would make more sense to start with a replication study. However it wasn’t that simple, so when I ran the replication study, surprisingly, we got a null result, and I was a bit confused about why this might be. So the first thing that I did was I contacted the original researchers to ask them what they thought might be the problem.

They got back to me and they said that they thought it was because of some differences between the stimuli, so the things that I'd shown in my study versus the things that they showed in the original. And some of these differences were things I couldn't have known, because they didn't outline the specifics of that in their original paper.

I made some edits to the protocol to the way that I was going to run the study.

And then I thought, okay, now that I've had approval from the original authors this new version should be able to replicate the original study. So I ran it again and surprisingly I still wasn't able to replicate the result.

Erm and so this was quite disappointing, both for me and the original authors, because it meant that I wasn't able to find the same thing that they did.

So... This was my first experience of replications. And you might think that that was enough to put me off doing any more, but instead, quite the opposite. I ended up realising how important replications are.

So yeah, ever since starting with that first replication study as part of my PhD, I've now kind of made that my specialty.

My advice for anyone who would be conducting their own replication study comes from some of the mistakes that I made as part of that first replication study that I did.

So my first piece of advice would be to always contact the authors before you begin your replication study.

I think I was a bit naïve, and thought if I just follow what's written in the paper then how can I go wrong? But papers don't have enough space to include everything about a study that you would need to know in order to conduct a good replication.

So I'd recommend talking to the original authors, coming to an agreement with them, making sure that they agree that the protocol that you've proposed, they would agree that's a good faith replication attempt of their study.

Another thing that I did wrong is that I only collected the same number of participants as in the original study, for my replication, because I thought that was more ‘replication-y’, because it was the same amount of participants as the original. But now, after learning more about both replication studies, but also sample size more generally, I would really recommend to go with a much larger sample than the original study that you're replicating.

And this is just so that you can be a little bit more sure about what your findings mean. So, in my study, I wasn't able to replicate the same result as the original authors, but this could just be because the true effect size that's in the world for that effect that I was looking at might be smaller than what they kind of measured in the original study.

If I had used a much larger number of participants, if I still wasn't able to replicate the study, we could be a bit more sure that it wasn't just because of low sample size.

I ran my study just as a normal study where we finished the entire study and then submitted it to a journal for publication. And I was lucky that it was successful in getting published.

But it could have been a lot harder for me to publish, which would have been a bit disappointing and taken a lot of time. So what I would recommend instead is submitting any replication study as a registered report.

A registered report is essentially where your paper gets peer-reviewed before you have collected data. So the peer reviewers say whether your protocol makes sense, recommend any suggested changes, and then once they've accepted it, the journal agrees to accept your study, regardless of what the outcome is. So that would be my third piece of advice.