A closer look at preregistration

This blog has been written by Sabine Vreeburg and Chantal Hukkelhoven

Preregistration is an Open Science practice. In this Open Science Blogpost we’ll explain what preregistration is, why you would preregister your research, and how you can get started.

What is preregistration?

When you preregister your research, you submit your research plan to a registry before you start your investigation. Preregistration creates a timestamped document in which you register research questions and hypotheses, data collection methods and proposed ways of analysing the data.

By preregistering you separate planned from unplanned parts of your research. Preregistration is useful for both exploratory and confirmatory research.

Below you can find answers to Frequently Asked Questions (FAQ) on preregistration.

Why would you preregister your research?

There are several reasons to preregister your research projects.

• Preregistration contributes to the transparency of science.
• By preregistering you separate hypothesis-generating (exploratory) from hypothesis-testing (confirmatory) research^1,2.
• Preregistration helps to prevent practices that are often unintended³ but still reduce the credibility of science, such as p-hacking⁴; HARKing (hypothesising after results are known)⁵; non-publication of non-significant results, resulting in publication bias⁶; and selective reporting of outcomes^2,7.
• Preregistration can give you more confidence in your work. You know that you are not fooling yourself with your unintended biases.
• Preregistration may improve your study design. When forced to explicitly think about the data you’re gathering and the subsequent data analysis, you may discover design flaws sooner⁸.
• When you preregister a study, you can show that you (first) thought of the study even when you are scooped.
• Preregistration is recommended by Horizon Europe and can thus yield a higher evaluation score in the Excellence section of a Horizon Europe application. In fact, it is likely that increasingly more funding agencies will ask for preregistration in the future. Finally, the Dutch Royal Netherlands Academy of Arts and Sciences (KNAW) identifies preregistration as a strategy to improve reproducibility⁹.
• Scientific journals award badges to publications that report preregistered research.

For an overview of possible practical challenges in preregistration and their solutions, please read the following paper by Nosek et al.: The preregistration revolution².

Which webinars on preregistration can you recommend? 

An informative  and recent (2022) webinar concerns “Preregistration: Why, What and How” by Prof. dr. Harm Veling. This webinar has been organised by the Open Science Community Wageningen.

Other interesting webinars are for example:  “Preregistration: Improve Research Rigor, Reduce Bias”, “Preregistration on OSF” and “Registered Reports for Early Career Researchers”.

Is preregistration mandatory?

WUR does not currently have a policy on preregistering research. So, the short answer is “no”, in general preregistration is not mandatory.

However, since 2014 preregistration has been mandatory for clinical trials in medical research in the EU. In other fields preregistration is becoming more common. See, for example, the transparency checklist that was developed for social and behavioural research¹⁰. Preregistration is also recommended by Horizon Europe and will be evaluated in the Excellence section of a Horizon Europe application.

How to get started on preregistering my research?

A website that contains lots of valuable information is the website of the non-profit Center for Open Science. Here you can find the following:

• Templates of many preregistration forms
• Contacts of researchers who have previously preregistered their work and who have said they would be happy to help you
• Help docs and instructions to register any project on OSF
• Checklist of items to include when developing an analysis plan for some common statistical models
• Checklist of items to include when describing the results of preregistered research
• Example preregistrations

Where to preregister your research plan?

Preregistrations can be deposited in many repositories, such as the following:

• Open Science Framework, a platform from the non-profit Center for Open Science (COS) that offers multiple options for preregistration for many kinds of research and provides standardised preregistration templates
• AsPredicted: domain-general registry service for to-be-executed empirical studies. To pre-register a study on AsPredicted, researchers answer nine simple questions about their research design and analyses. The platform then generates a time-stamped, single-page PDF document that includes a unique URL for verification.
• preclinicaltrails.eu: preclinical animal study protocols
• PROSPERO: prospective register for systematic review
• RIDIE: prospective registry of impact evaluations of development policies and programmes in low- and middle-income countries

The COS also contains a continuously updated overview of journals that offer peer reviewed forms of preregistration: registered reports.

I would like to start preregistering my research, but I am not sure whether my research is suitable

For an overview of possible practical challenges in preregistration and their solutions, we recommend reading the following paper by Nosek et al.: The preregistration revolution². Nosek et al. describe the ideal preregistration and 9 possible challenges (e.g. you happen to perform many small experiments, or you plan to use a pre-existing dataset).

Is a preregistration peer reviewed?

Preregistration does not have to be peer reviewed. However, preregistration can be peer reviewed and an increasing number of journals offer this possibility. Peer reviewed preregistration comes in two different forms:

• A peer reviewed preregistration can be directly published in the form of a Study Protocol (e.g. in BMC Public Health).
• The peer review of the preregistration is, in principle, an acceptance for publication of the whole study, regardless of its outcomes. This format is called a Registered Report. You can find an overview of journals with Registered Reports here.

Are my preregistrations public?

This depends on what platform you chose. Preregistrations are often entered in a repository with an embargo period, after which they are made public. However, if you want, your preregistrations can be published immediately or remain private indefinitely. Some platforms, such as the Open Science Framework, have a limited embargo period (max. 4 years).

Can I make changes to my preregistered plan?

Although you cannot change your time-stamped preregistration, you can add a new preregistration indicating the differences to the previous. You can also describe in your manuscript how and why you deviated from the initial plan. Reviewers can then evaluate whether these changes are legitimate.

Preprint vs. preregistration

Sometimes preprints and preregistration are confused, but they are different:

• A preprint is a manuscript of a finished research project that is published before peer review.
• A preregistration is the registration of a research plan before the start of a scientific study. A preregistration can be reviewed but does not need to be. A preregistration can be published immediately or after an embargo period, or it can remain private indefinitely.

Study protocol vs. registered report

Study protocols and registered reports are both publishing formats. They both involve a peer reviewed form of preregistration.

• A Study Protocol (e.g. in BMC Public Health) is the publication of a peer-reviewed study plan before the study is carried out.
• A Registered Report is a peer-reviewed publication of a scientific study that was accepted for publication before the study was carried out based on the study plan. The peer-review of the study is carried out in two phases. First, the study design is peer-reviewed. When the design is accepted, the study takes place, the data are analysed and the report is written. The manuscript is then peer-reviewed to see if the authors carried out the research according to the registered plan, and if they did not,  reviewers investigate if the deviations from the plan are justified.

More information

For any questions about preregistration, please contact our specialists at openaccess.library@wur.nl

Further reading

1          Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. & Kievit, R. A (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science 7, 632-638, https://doi.org/10.1177/1745691612463078.

2          Nosek, B. A., Ebersole, C. R., DeHaven, A. C. & Mellor, D. T (2018). The preregistration revolution. P Natl Acad Sci USA 115, 2600-2606, https://doi.org/10.1073/pnas.1708274114.

3          Nuzzo, R. How scientists fool themselves – and how they can stop (2015). Nature 526, 182-185, https://doi.org/10.1038/526182a.

4          Head, M. L., Holman, L., Lanfear, R., Kahn, A. T. & Jennions, M. D (2015). The Extent and Consequences of P-Hacking in Science. PLOS Biology 13, e1002106, https://doi.org/10.1371/journal.pbio.1002106.

5          Rubin, M. When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress (2017). Review of General Psychology 21, 308-320, https://doi.org/10.1037/gpr0000128.

6          van der Naald, M., Wenker, S., Doevendans, P. A., Wever, K. E. & Chamuleau, S. A. J (2020). Publication rate in preclinical research: a plea for preregistration. BMJ Open Science 4, e100051, https://doi.org/10.1136/bmjos-2019-100051.

7          Kimmelman, J. & Anderson, J. A. Should preclinical studies be registered? (2012). Nature Biotechnology 30, 488-489, https://doi.org/10.1038/nbt.2261.

8          Kupferschmidt, K. More and more scientists are preregistering their studies. Should you? (2018). Science, https://doi.org/10.1126/science.aav4786.

9          Replication Studies -Improving Reproducibility in the Emperical Sciences.  (Royal Netherlands Academy of Arts and Sciences (KNAW), 2018).

10         Aczel, B. et al. A consensus-based transparency checklist (2020). Nature Human Behaviour 4, 4-6, https://doi.org/10.1038/s41562-019-0772-6.