Reviewing reviewers reviews

Peer review is the cornerstone of science. It is the process that (tries to) safeguard the science in the scientific article. It is the relatively simple process of anonymous proofread (review) by a relevant scientist (peer).

Software review is one of the many cornerstones of our modern society. It is the relatively simple process by which software bugs (unintended actions) are hunted down and fixed.

Software review can be done in variety of ways. As a scientist I am partial to open source review. Open source review is a same same but different process to scientific peer review. Its not anonymous, and its not as formal. In science, the journal editor selects the peer which reviews. Open source software review is only one of the many review options available, and its a beautifully organic process! For computer “peer review” to take place, the program must be open source, meaning that its source code, its components, are visible to all. And when it is open source, people spend their free time to double check every nook and cranny of the code.

Software and peer review is something that everyone takes for granted. We (including scientists) assume that published science just works. We assume that our emails are read only by the recipient (and NSA). We assume that using our credit card online is safe.

And there is another interesting phenomenon of our modern times. We rely on immensely complex things that we cannot possibly be expected to really know. We rely on the predictive power of pharmaceutical science for ever more efficient treatments, and we rely on the security of software to do banking, and gossip private secrets. Different fields in which individuals cannot be simultaneously proficient.

I like to think that independent review is another major advance in the organization of our modern society. To steal a quote from computer scientists: “given enough eyeballs, all bugs are shallow”.

So how do these two processes fail?
Scientific peer review has been under heavy fire. As has computer and network security. Remember the recent heartbleed vulnerability? It was a software bug, discovered in the source code, that made secure communications insecure. Even more worrisome, it had existed for over 2 years!
Science has had its fair share of heartbleeds. From strings of well publicized retractions of published articles, to studies showing that many high profile publications cannot be reproduced (implying that they were false positives).

So we are all doomed, nothing is to be trusted! Thankfully not.
Peer review has its clear issues: bad science, and computer bugs can sneak through, and good science can get lost on the way.
Please notice two things though: first, the reason I can write about this, is because we found out about these issues! So something does work. And more importantly, this flawed system is the best we have! Like in democracy, where terrible, racist and populist parties can gain power.

And that’s the thing, I can’t think of a better alternative. Can you?

One thought on “Reviewing reviewers reviews

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s