Documenting decisions transparently

Week 4 is about ways to make your research more transparent. As you discovered earlier, transparency means being clear about exactly what you did at every stage of your research. However, over the course of an entire research project (typically months to years), it’s highly likely that you will forget certain aspects of how the study was conducted, and when and why decisions were made.

One of the most important ways you can make sure that others will be able to replicate your research is by keeping detailed records of all aspects of the project, and updating them as you go. It’s a bit like keeping a record of the twists and turns you made while going through a maze. Without detailed notes, remembering all the decisions you made at every stage of your research project can be difficult. You will build up a lot of information, which can be hard to compile once you get to the writing stage.

The most important distinction is whether decisions were made before or after data collection begins. This is because when researchers are able to look at the data they can be swayed by what they see. Preregistration (or Registration in some fields) is the practice of publishing the plan for a study, including research questions, hypotheses, research design, and data analysis plans before the data has been collected or examined.

A preregistration document is time-stamped and typically registered with an independent party (e.g., an open access repository) so that it can be publicly shared with others. Preregistration provides a transparent documentation of what was planned at a certain time and allows third parties to assess what changes may have occurred afterwards. Importantly, it’s fine for changes to occur, it’s just important to know when and what these were, and why these changes were made.

Having a more detailed preregistration leaves fewer research degrees of freedom. In other words, the more detailed a preregistration is, the better third parties can assess any possible changes and how they may affect confidence in the results.

One platform for preregistering research is the Open Science Framework (OSF). On the OSF there are support videos and documentation to guide you through the process. You can use a variety of templates depending on your discipline and methodology to preregister your study in varying levels of detail. These templates include:

• Standard OSF template (good for most science disciplines)
• Systematic Reviews
• Social Psychology
• Qualitative
• Secondary Data

If none of the available templates suit you, you can write your own document and preregister this in an open-ended preregistration!

Confirmatory analyses refer to analyses that were set before data collection or examination, and that test whether a hypothesis is supported by the data. Exploratory analyses are carried out when some data have already been collected. They are useful for discovering patterns in that data or extending to new topics or subjects. They foster hypothesis development and refinement.

Preregistration often aims to clearly distinguish confirmatory from exploratory analyses. This is helpful because you won’t be able to convince yourself (or others) that you had hypotheses before you saw your data, when actually you added these ‘post-hoc’, after seeing the results.

If you are thinking of preregistering either type of research, here are some things to consider:

• Both quantitative and qualitative research can be confirmatory, and so preregistration for confirmatory research can be used for both.
• For exploratory research, preregistration can be a great way to document initial study plans, even if those later change through an iterative process.
• Preregistering exploratory studies can be useful for both quantitative and qualitative research, for all disciplines.
• If your discipline has another good way of keeping track of how study plans change over the course of the research, then this could also work well instead of preregistration.

It is important to distinguish between confirmatory and exploratory analysis so that results can be interpreted accordingly.

Whether your work is confirmatory or exploratory, preregistering keeps a permanent record of your ideas at the design stage, before you start the analysis.

The process of preregistering involves answering a series of questions about your research. There are many templates of preregistration forms available. In the next activity, we will walk you through some typical questions.

When answering the questions, you should aim to be as precise and detailed as possible. By doing this, you are being transparent about your research plans from the outset. The benefits include establishing a clear and detailed plan for your research, that you can revisit and update as you make your way through your research project.

A detailed and comprehensive preregistration demonstrates that you haven’t engaged in the questionable research practices that you learned about last week, and can be useful for reviewers and readers when they assess the integrity of your research.

In the activity below, you will gain experience of the type of information you will need to provide when preregistering a research project. Please answer the questions based on one of your existing research projects, or a project you would like to do in the future.

Allow about 30 minutes

Activity 1.1:

Activity 1.2:

Activity 1.3:

This question concerns whether you have already collected data. There are three possible options: simply select the one that best describes the stage you are at with your research, provide further details, then click ‘reveal’ to see our comments.

Have any data been collected for this study already?

  a) Yes, at least some data has been collected for the study already.

  b) No, no data have been collected for the study already.

  c) It’s complicated because we have already collected some data or are using secondary data.

Now let’s continue our walk through preregistration. The next question gets to the heart of the matter: what your research is about. You need to be clear and concise in your responses, so that when you return to your preregistration document, it will clearly encapsulate what your plans were at this point in time.

Activity 1.4:

Next, you’ll be asked questions about the design of your study. The first of these questions relates to quantitative research. If your research is not quantitative, you may wish to skip to Activity 1.6.

These are some questions you might want to ask yourself when answering:

1. What are your independent variables and your dependent variables?
2. How do these variables relate to each other?
3. How will they be measured (a self-scale report, a behavioural task)?
4. What is your sample size and criteria?
5. How did you determine your sample size?
6. Do you have a ‘between’, ‘within’ or ‘mixed’ study design?
7. Are you using counterbalancing?

Activity 1.5:

If your research is qualitative, you will also be asked to specify exactly how you plan to conduct your research, although your answers are likely to be a little different.

For instance, you might be interested in the experience of parenting a child prodigy. How will you approach the task? With a questionnaire? An interview? A focus group? Open or closed questions? Or supposing you are interested in changes in depictions of families through the twentieth century. What evidence will you use? Newspapers and magazines? Archive photographs? How will you analyse them? Discourse analysis? Visual analysis?

Activity 1.6:

We hope this activity helps you feel confident about your next preregistration. These questions have concentrated on your methodology plan, but you will also need to provide information about your analysis plans. We will explore the analysis stage in more detail next week.

We’ve talked about preregistration as a way to be transparent before you collect your data. How about once you’ve collected your data and are writing up your research? You should be honest about how you conducted your study, and anything that has changed since you planned (and perhaps preregistered) it. One way of being transparent when writing up your research is to use reporting guidelines.

Reporting guidelines are sets of rules or standards that help researchers present their findings clearly and transparently. They're like a checklist that ensures all-important information about a study is included in a research paper. These guidelines vary depending on the type of study or field of research, but they generally help researchers communicate their methods, results and conclusions effectively, making it easier for others to understand and evaluate their work.

Here are some reporting guidelines for different fields:

STROBE (STrengthening the Reporting of OBservational studies in Epidemiology)

• STROBE provides a checklist to enhance the reporting of observational studies in epidemiology, encompassing key aspects such as study design, participant selection, data collection methods, and statistical analysis.

COREQ (COnsolidated criteria for REporting Qualitative research)

• COREQ provides a checklist of items that researchers should address when reporting qualitative research, covering aspects such as study design, data collection, analysis, and interpretation.

EQUATOR (Enhancing the QUAlity and Transparency Of health Research)

• EQUATOR provides a variety of reporting guideline templates for various branches of health research, including reporting guidelines for randomised trials, observational studies, systematic reviews, qualitative research, animal studies, economic evaluations, and more.

There are many benefits to using reporting guidelines. Most obviously, they help researchers to clearly and comprehensively communicate all the important information about their study. This is helpful for the researcher themselves, and for anyone else who wants to read, understand, and potentially build upon their work. However, if you’re unable to find reporting guidelines for your particular field, being as transparent as possible and including as much detail as possible is your best bet!

Activity 2:

Allow about 30 minutes

In the activity below, you get the chance to practise writing your own simple set of guidelines.

So far, you have learned a lot about the principles of open research and how they are applied. You may be planning to begin incorporating open research practices into your own research immediately, or perhaps you will want to do so in the future. To help you navigate more quickly to the information you need, the course team have developed an open research decision tree. This companion to the course helps you remind yourself of the principles of open research, and how to take open research actions. You can continue to use it after you have finished the course. There is also an open access version on the course description page, so you can share the link with colleagues.

Scroll down to the decision tree (below), and click on it: you need to wait a few moments for it to load. Then take some time to familiarise yourself with the open research decision tree.

View interactive version

 

1. Use the open research decision tree to remind yourself about principles in open research. Go to the welcome page and click the 'principles' button. What do you find? Now click 'Transparency'. Then click 'Manuscript writing and preregistration'. This takes you to some concise summaries. Clicking the second and third of these will bring you to the explanations you’ve just read! You’ll find this handy after you’ve finished the course and want to look something up.
2. Use the open research decision tree to learn about actions. Go back to the welcome page and click on the 'Actions' pathway, then select the stage you are at with your research. Let’s assume you’re at the planning stage – click ‘I’m planning’. Now click ‘Planning to collect and analyse data’ to remind yourself how you can plan for replicability (something you studied in Week 3). Clicking that will bring you back to the explanations in Week 3.
3. Now follow your own trail through the interactive.

This week’s quiz will help you consolidate your understanding of transparency, preregistration and reporting guidelines. The questions will help you revise the key points. It is more important to read the feedback we have written than to get the questions right first time.

Answer the following questions about key terms:

This week, you have dug deeper into transparency in research – documenting how your study was conducted and when and why decisions were made. You learned about preregistration and reporting guidelines: how these can increase transparency, and potentially help you avoid questionable practices.

You were also introduced to the open research interactive decision tree, which you can continue to use throughout the course. You can return to the decision tree after you have finished the course as needed.

In Week 5, you’ll move on to consider the trustworthiness or believability of research findings.

Click here to move on to the next week