Open Science and Reproducibility-Registered Reports to the Rescue

Did you know that you can get a study accepted for publication before you even start the study? This is called a registered report, and it holds a lot of promise when it comes to battling the reproducibility crisis and increasing motivation.

I have spent many hours in labs in the last 10 years doing work that I knew would probably never be published. There were many reasons for this. First, the experiment didn’t fit into a bigger well-designed study. It might at some point, but it didn’t from the start. Second, I wasn’t finding any difference between my groups of cells or mice. Third, writing papers is a lot of work, and I was likely to leave the lab before the writing would finish. And more often than not, once the PhD or Postdoc who did most of the work leaves, the work never gets published. This isn’t exactly the recipe for motivating a scientist.

What if we could turn this work around? What if we started with the part that often never happens: writing the paper. Without the results but with the other parts: Introduction, the hypothesis we are testing, materials and methods (if you want to see examples of what this can look like, go to this zotero group that lists registered protocols and registered reports in the journal eLIFE).

Then we do the experiments, which are now from the start part of a bigger well designed and thought through study. Finally, we fill in the results we get in the already outlined manuscript. And the real beauty is that we can get the study accepted for publication before we even do the study! So even if we don’t find a difference between our groups, we can still publish the paper!

What a way to be motivated to get it done, and do it in a structured and thought through way! No more wasting time in the lab on experiments that will never see the day of light.

From my perspective from the lab bench, registered reports can not only battle the reproducibility crisis, they can also help young scientist to thrive in their work.

Let’s go back to the reproducibility crisis and how registered reports may improve the quality of science and science reporting. In a previous post, I wrote about reproducibility and about how two threats to it are p-value bias and HARKing (hypothesizing after results are known).

Registered reports hold the promise to end these two practices, and thereby ensure that the published literature represent reality more than it does today.

When the study is evaluated for publication before the results are known, the only parameter that can be evaluated are relevance of the study question and quality of the methods and study design. Thus, regardless of the p-values, the study will be published if the quality of the work was high from the start and is kept high to the end.

When it comes to HARKing, it will not be possible for the authors to change their hypothesis without disclosing it, as the original hypothesis is already registered. There’s nothing wrong with changing a hypothesis. But if the study was designed to test a different hypothesis, this should always be disclosed by the authors. The reason for this need to disclose is that the probability for false positive findings increase, and the reader should know that.

By now, you have probably realized that I’m enthusiastic about registered reports. I’m a skeptic by nature, and I would love to have less reason to be skeptic about the papers I read. My enthusiasm grew after I attended ESOF2018 this summer, where the concept of registered reports were brought up by several speakers in sessions on reproducibility.

If you’re a skeptic like me, you probably have many questions on the weaknesses of registered reports. For example:

Won’t my ideas be scooped if I share them in such detail before I do the study?

Or:

What will happen to creativity when everything is planned from the start?

Luckily, these questions and many more have already been asked by skeptics and answered by some of the bright minds at Center for Open Science. You can find them here by going to their FAQ section or get the PDF here. At the Center of Open Science webpage, you can also get an explanation of the steps involved in producing a registered report.

If your curious about open science in general, I recommend listening to this episode of the Curie podcast where Natalie Von Der Lehr talks to John Tennant an avid open science advocate who is developing a MOOC (massive open online course) on open science.

Now, what is the status of registered reports at moment?

It’s still a growing concept, and more and more journals offer this way of publication. At ESOF2018 speakers said that 121 journals were offering publishing via registered reports. According to Center for Open Science, the number has now increased to 148 journals.

Although the journal Nature human behavior now offers publishing via registered reports, I’m skeptic as to whether the majority of glam journals will offer this opportunity any time soon. Because, if they do, they will have to publish negative findings and not only the significant novel discoveries they prefer. However, there’s no need for this to block the advancement of registered reports as the authors are not bound by the journal who accepted it initially, Center for Open Science writes in their extensive FAQ.

So if you get amazing results, and you cannot resist going for the glam, you can retract your manuscript from the original journal and submit to the journal of your choice. To me, this is a clear win-win situation for the authors.

What do you think? Is it time to turn the workflow around?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s