Almost every human activity in the first world has been impacted by technology. Our means of production have been fundamentally altered, and while creating enormous wealth, the changes have often been disruptive, painfully so at times. As this ongoing transformation accelerates, “business as usual” has become an oxymoron.

Paradoxically, while science is at the foundation of all our technological progress, it is like the eye at the center of the storm – the academic mode of operation has hardly changed over the last two centuries. And why not? An argument could be made not to fix what isn’t broken. For instance, sometimes you hear the scientific process compared to the Open Source movement, arguing that both strive for a transparent meritocracy where openness ensure that mistakes will not survive for long. Unfortunately, this idealized view is a fallacy on more than one count.

There are lots of good reasons for the Open Source coding paradigm, but it does not give license to forgo quality control and code review, as for instance the heartbleed bug, or the most recent widespread (and ancient!) bash vulnerability, illustrated.

On the other hand, the scientific peer review process is not anything like the open communication that goes on in public forums and email lists of Open Source software like the Linux kernel. Peer review is completely closed off from public scrutiny, yet determines what enters the scientific discourse in the first place.

The main medium of communicating scientific results remains the publication of papers in scientific journals, some of which charge outrageous subscription fees that shut out poorer institutions and most individuals. But this isn’t by far the worst impediment to open scientific exchange. Rather, it is the anonymous peer review process itself, which is by design not public. Authors are often given opportunities to correct a paper by re-submitting if the original one is rejected, but ultimately the peer reviewers serve as gatekeepers.

For a discipline that is the foundation of all our technology, the knowledge generating process of science has been surprisingly untouched, and due to the build in anonymity, it has also managed to escape any scrutiny. That would be all fine and good if it was actually working. But it is not. We know little about stellar papers that may have been rejected and now linger forever in obscurity in some pre-print archive, but we know all the trash that passed final approval for publications. For instance, we get sensational headlines like this, promising an entirely new take on dark energy, but if you actually read up on this you realize that the author of the underlying paper, who is just referred to as a University of Georgia professor, is actually not a physicist but a biologist. Now far be it from me to discourage a biologist from wanting to do physics, but if you bother to read his paper on the matter you will quickly realize that the man apparently doesn’t grasp the basics of general and special relativity. Yet, this was published in PLOS One which supposedly follows a rigorous peer review process. They even bothered to issue a correction to an article that is just pseudo science. Mind boggling.

Now you may think, well, this is PLOS One, although a leading Open-Access journal, the underlying business model must surely imply that they cannot pay decent money for peer review. Surely more prestigious journals, published by an organization that is known for its ludicrous journal subscription prices, such as Elsevier, will have a much more rigorous peer review process. Unfortunately you would be wrong. May I present you this Rainbow and Unicorns Gravity paper. It has, of course, caused the predictable splash in the mainstream media. The paper should never have been published in this form. You don’t have to take my word for it, you can read up on it in detail on the blog of Sabine Hossenfelder, whose 2004 paper on black hole production the authors listed as a reference. When challenged to write up a criticism to submit to the same journal, Sabine didn’t mince words:

This would be an entire waste of time. See, this paper is one of hundreds of papers that have been published on this and similar nonsense, and it is admittedly not even a particularly bad one. Most of the papers on the topic are far worse that that. I have already tried to address these problems by publishing this paper which explicitly rules out all models that give rise to a modified dispersion relation of the same type that the authors use. But look, it doesn’t make any difference. The model is ruled out – you’d think that’s the end of the story. But that isn’t how science works these days. People continue to publish papers on this by just ignoring it. They don’t even claim there is something wrong with my argument, they just ignore it and continue with their nonsense.

I have wasted enough time on this. There is something really, really going wrong with theoretical physics and this is only one indication for it.

Later in the comment thread she also had this to say:

I have had to talk to students who work on related things (not exactly the same thing) and who were never told that there are any problems with this idea. Even though I know for sure their supervisor knows. Even though I have published half a dozen of comments and papers explicitly explaining what the problems are. Honestly, it’s things like this that make me want to leave academia. This just isn’t research any more this is only sales strategies and networking.

The problem goes beyond peer review and comes down to accountability, but because the peer review is anonymous by design, it is especially easily corrupted, and I know of cases that resulted in exactly what Sabine spelled out: Top talent leaving academia and theoretical physics. The more I look into this the more I believe that this process at the heart of the scientific endeavour is fundamentally broken, and urgently needs fixing.