Richard Horton, the editor of The Lancet, in an editorial for the Medical Journal of Australia (MJA 2000:172: 148-9) once wrote:

“The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”

In issue 45 of the Real-World Economics Review, Grazia Ietto-Gillies published an article, “A XXI-century Alternative to XX-century Peer Review” (issue no. 45, 15 March 2008, pp. 10-22), in which she argued that traditional peer review should be replaced by ex-post bottom-up peer comments.

This would take “advantage of the new communication technologies. In her system, papers for publication are submitted to an appropriate open-access site where they would be screened to weed out “crankish” papers, for example, and then published by being posted on the site. Those who wish to comment on them are free to do so, subject to similar screening. Thus, papers would be published quickly, and those who have comments can also gain credit as they would be posted on a linked site. Taken in isolation, her proposals have the merit of formalizing peer review at its informal best.”

In issue 47, 6 scientists, including Steven Harnad reply to her proposals.

The 6 comments are worth reading in full, here we only cite Harnad’s cautionary remarks:

“(1) There is nothing wrong with classical Peer Review (PR) that a supplementary Open Access (OA) system will not fix — but OA is an “ex post” supplement to PR publication, not an “ex ante” substitute for it.

(2) OA means immediate free webwide access to post-PR journal articles (“postprints”) immediately upon acceptance for publication, plus, in cases where the authors desire it, free access also to their pre-PR “preprints” even earlier, for pre-PR commentary.

(3) This solves most of the problems cited by Grazia Ietto-Gillies in “A XXI-century alternative to XX-century peer review”: access, speed, scope, corrective feedback.

(4a) Classical PR is also (a) an answerable mechanism, with the referees, optionally anonymous, privately answerable to the editor, as is the author, for producing a paper that, once accepted, has been revised to meet the known and trusted quality standards of the journal in question; the editor is in turn publicly answerable to the journal’s usership with its reputation for quality.

(4b) PR is also a (necessarily “ex ante”) filter for users, so that they need not waste their limited reading time trying to peer review raw drafts for themselves, nor risk their scarce and precious research time trying to build on unsound results that have not met a known and trusted quality standard.

(5) “Ex-post” open commentary is neither answerable to an editor who answers for maintaining the journal’s quality standards, nor is the author of an unrefereed draft answerable, having the option of revising or not revising to meet arbitrary self-appointed commentators’ recommendations.

(6) Most serious referees and users do not have the time or the desire to work their way through raw unrefereed drafts, neither to referee them, nor, worse, to risk using them, unrefereed.

Systems like the one proposed by Ietto-Gillies have been proposed many times. What is needed is to test them, to demonstrate that they are capable of generating at least the same standards of quality and useability that we have now in each field — and also that they are sustainable and scalable. (Everything new works at first, for a while.)

Until they are thus tested and proven, these are just evidence-free conjectures — and conjectures that go against the actual experience of editors, which is:

(i) that qualified referees (who often want and need the option of anonymity) are a scarce, overused resource that is already hard to mobilize when referees know that authors are answerable to editors to ensure that they take their referee reports seriously, hence even less likely to donate their time and attention to unfiltered and unanswerable raw drafts;

(ii) that authors, (who often want and need the option of not making their unrefereed drafts public) seek qualified feedback from referees anwerable to a qualified editor of a journal with known quality standards, so that their own article too can be certified as having met those known quality standards;

(iii) that users need research that is filtered and certified to have met known and trustworthy quality standards in advance (i.e., “ex ante”).”