Clinical trials have had a pre-registration protocol for a while now. Basically, if you want to run a clinical trial, you have to specify how you will conduct your research including how you will analyze it once the data have been collected. This pre-registration attempts to avoid so-called p-hacking and the publication bias that results. P-hacking is a term for the situation where a researcher (intentionally or unintentionally) explores (mines) their data to determine what relationships result in a statistically significant result and then they publish those results. By requiring you to specify in advance how analyses will be performed, you cannot (intentionally or unintentionally) p-hack.

Unfortunately, pre-registration of research is not a common practice, but more and more options are available for pre-registration. Today, PLOS ONE announced that the will provide a mechanism for Registered Reports. Like many pre-registration options, there are two stages to the PLOS ONE Registered Report Process: 1) Registered Report Protocol and 2) Registered Report Research Article. In the Registered Report Protocol, the research writes up what they plan to do and this undergoes peer-review the way a typical article would except that there are no results. Once the Registered Report Protocol has been approved, the researcher conducts their research according to the Registered Report Protocol. Once completed, a Research Article is submitted and its adherence to the Registered Report Protocol is evaluated through peer-review. In this second step, the only thing that matters is how well the actual research followed the protocol.

Now, this is still not a fool-proof system. One way to game the system is to pre-register lots and lots of possibilities and then only follow-up on the ones that worked. I think there should be a limit to how much you can pre-register or, perhaps, you get credit for writing final reports of pre-registered studies and you can use these credits to write more pre-registrations. Also somebody could keep track of pre-registered studies with no final report.

Another way to game the system is to try lots of things (without pre-registering), registering the things that worked, and then publishing those things. We can eliminate this if only pre-registered studies would be taken seriously. Any other ideas?

Of course, most research is not a straight path. That is, things come up that you have to deal with. Ideally this is spelled out in the pre-registration, but what if it isn’t. Obviously, the adherence to the protocol will likely never be 100% but the peer-reviewers in the second step above should determine whether the researchers made a good-faith effort to abide by their registered protocol.



blog comments powered by Disqus

Published

30 January 2020

Tags