14 thoughts on “Peer Review”

  1. The real problem is the deprioritization of reproduction of results, a cornerstone of science. Which is a problem that Feynman called out decades ago. Too many people these days, even scientists, think that the gauge of truth comes down to being printed in a reputable, peer reviewed journal. That’s not where scientific debate ends, that’s where it begins.

  2. In the area of computer science it is not very good either. You would expect people to publish software source code and datasets. But most often than not it does not happen.

      1. I’ve banged my head against many a mathematical description of an algorithm trying to get an implementation that actually works. Back when I was working for a research institution I would look forward to the conferences so I could corner an author and demand they explain their notation, as they’d often reply with incoherent terseness when you emailed.

      2. When people publish an algorithm and then give execution times on actual hardware it is clearly obvious they have actually coded it in a real programming language. So why not bother release the code of their sample implementation?

  3. I have commented on this site, on Instapundit, and other sites about this problem. The whole peer review system is broken. It might have made sense at a time when publishing required a printing press and distribution was by snail mail, and communications between peers was more limited.

    However, today we have the Internet. Publishing and distribution and inter-peer communication have never been easier or less expensive.

    What is needed is a cross between Wikipedia and arXiv. Authors would publish their paper on the site, an associated comments page would handle peer review, and citation would be by hyperlink and trackback.

    In fact, this is precisely why the internet and hypertext were invented in the first place! That this is not already the norm for scientific publishing is amazing, considering the internet has been around for more than forty years and in widespread use for nearly 20 years.

    1. It’s only surprising if you don’t consider that science is also a social process. Social processes have inertia. One used to be able to find a lot of authors who deliberately eschewed use of the early dedicated word processors and, later, of PC word processing software, in favor of typewriters or even of writing longhand. Most of these people are now dead. Every field has it’s irrational stick-in-the-muds and science is no exception. The Web has only been around long enough for maybe half the scientists set in their peer-reviewed journal ways to have died off or retired. The dominant paradigm will shift, but it’s a process, not an event.

      1. That is part of the problem, but an even larger problem is that university tenure and promotion decisions are almost universally based on the faculty members publication in recognized peer-reviewed journals. The higher the journal’s rejection rate and perceived reputation as “exclusive” the more “points” a faculty member earns. Self-published articles, and even books, usually count as zero. The same tends to be true for research institutes, although earning grants also become a factor, the number of publications in recognized peer-review journals has a role in hiring, promotion and salary.

        Until the reward process recognizes other forms of scholarship and treats it with the same level of respect there will be no core changes and only second rate work rejected by journals will be dumped online.

  4. I’d go as far as to say the early peer review process was crowdsourcing. The referees in the process had to know whether feedback was coming from an old spat, or an actually valid critique of the work.

    1. I think a commenter reputation function would be an easy addition – I’ve seen it in use on some forums.

      The more I think about this, the more I think changing the scientific publishing process is a good idea. Blogs and Twitter have radically changed the news business, nobody actually prints encyclopedias anymore, Salman Khan is changing education. This is no different. The code required is all pretty much robust and readily available.

      So how does one change from the old paradigm?

      1. It is bound to happen. There are websites such as academia.edu already. There is also researchgate.net but it seems more like a spam site than a useful website to me.

  5. I understand that a lot of math is done this way now: forums of people discussing a problem, and that they have solved some serious problems that way. Anyone know anything about it?

    1. Sort of like Sam Jaffe leaving his hard problems on a window-facing blackboard in ‘The Day the Earth Stood Still’ and leaving the door open so passers-by, like Klaatu, can drop in and lend a hand. 🙂

Comments are closed.