PubMed now allows comments on abstracts — but only by a select few

Written by Retraction Watch on October 25, 2013. Posted in Current News

PubMed today launches a pilot version of PubMed Commons,

“a system that enables researchers to share their opinions about scientific publications. Researchers can comment on any publication indexed by PubMed, and read the comments of others.”

In general, we’re big fans of post-publication peer review, as Retraction Watch readers know. Once it’s out of its pilot phase — and we hope that’s quite soon — PubMed Commons comments will be publicly available. So this is a step forward — but only a tentative one. That’s because of the first bullet point in the terms of service commenters agree to:

So PubMed Commons isn’t exactly a town commons, unless you happen to live in a town whose residents are all scientists.

Stanford’s Rob Tibshirani, who was involved in organizing PubMed Commons, wrote in a blog post prepared in advance of the launch:

“One would like the system to be inclusive as possible but many scientists would not be interested in posting comments in a system with a high proportion of irrelevant or uninformed comments. NIH also needed a rule for who could post that would be pretty clear cut and not based on e.g. some judgment of the experience or knowledge of the participants. The decision was made that comments could only be posted by authors of papers in PubMed. This would make the situation symmetric in that all people who comment can have their own work commented on. It would also include a large number of potential participants and would meet NIH’s need for something unambiguous. Unfortunately it would leave out many people who could add valuable input, including many graduate students, patient advocates, and science journalists. I’m a little worried about this restriction, as I want to make the system open to as many users as possible. But hopefully that is a pretty wide net, and it may be widened further in the future.”

We’re a bit worried about this restriction, too.

David Lipman, director of the National Center for Biotechnology Information, which runs PubMed, tells Retraction Watch:

“We wanted to have it be inclusive enough that it would be a wide range of people commenting. On the other hand, if we open it up to anybody initially, we were afraid we’d see a lot of the the kind of comments you see in Popular Science and on newspaper sites, and scientists don’t want to deal with that. This felt inclusive but filtered enough, to people with a stake in it.”

Popular Science, of course, recently shut down their comments after tiring of “trolls and spambots.” The magazine cited research of comments’ so-called “nasty effect” by University of Wisconsin professors Dominique Brossard and Dietram Scheufele. We asked them for their take. Scheufele told us:

“If the same effects hold that we found for online news articles, we could see a similar effect where just the tone of (unmoderated) discourse reshapes how we interpret the scientific articles immediately preceding them.  In this sense, separating the outlet for publication from the place where we might engage in – sometimes heated – debates about what these articles mean or how much we agree or disagree with conclusions, might lead to a better debate overall.”

We’re actually among those who can comment on PubMed Commons, because we’ve published a paper indexed in PubMed — all about post-publication peer review, as it turns out. But we really hope this net widens, and soon. Of course many scientists find it beneath them to read or respond to comments from the great unwashed masses — you know, the great unwashed masses whose taxes pay for their grants. What a terrible inconvenience! Sure cuts down on the number of people who might criticize your paper, though.

We think this is a particular problem because PubMed won’t allow anonymous commenters, either. Tibshirani writes:

“After much discussion, the group remained deeply split on this issue. Those wanting anonymous posts were concerned that many scientists, especially junior researchers, would be reluctant to make critical comments. But those opposed to anonymous comments believed that the quality of interchange would be higher if commenters were required to identify themselves. In the end, these differences weren’t really resolved and the decision was to start without anonymous comments and reevaluate after the system had been fully public for a while. While debating this issue various proposals were put on the table for ways to allow participants to review and essentially sponsor the anonymous post of another participant.”

That last bit is interesting, and could be a bit of a stopgap measure. But it’s not enough. By one measure, those opposed to anonymity are correct to say that “the quality of interchange would be higher if commenters were required to identify themselves.” Comments such as “This was a great contribution to the literature” are more likely. But the kind of respectful but critical questioning that helps correct the scientific record will be much less likely, because of realistic fears that senior scientists won’t like it very much when you criticize them. We’re once again struck that many journals and scientists seem to believe fervently in anonymous peer review to prevent this sort of revenge, but won’t allow anonymous comments because they make them uncomfortable.

Lipman:

“I absolutely think that is an issue, and that is the argument that went on internally among people who were involved. There are some people who won’t be afraid to make a collegial, constructive, but highly negative critique. But I do think that the advantage of having your name show is, you’re more careful about how you write it.

We already have the web as a whole where you can say whatever you want. Then we have letter to the editor. I’m imagining that this is something in between, more efficient, more fluid, and more unrestricted than a letter to the editor.”

Fair enough, but we really hope the net is widened to include anonymous comments, too. Or PubMed could leave that space to PubPeer, which has been quite successful so far and has managed to moderate effectively. But that would be a shame. PubMed has a great opportunity to be inclusive, and its traffic would mean high visibility for post-publication peer review, along with more pressure on journals and others to correct the scientific record.

Scheufele has a different take on that, though. He worries about “the crowdsourcing of academic peer-review:”

“…We spent decades or centuries coming up with a system that would allow us to fund the best and most promising research.  The system is far from perfect, but it is far superior to putting scientific expertise on the same level as public opinion when it comes to judging the scientific merits of an article.  I could very much see an instance, for example, where public discourse questions if we need research on girls’ body image.  The scientific importance of this research is obvious to me, but I think it would just be a matter of time before we’re seeing a variant of campaign ads like this on this topic.”

He continues:

“As I said elsewhere, more conversation about issues, including science, is always a good thing.  What the best format is for those conversations and how we can structure it in order to make sure we get the best outcomes, however, are empirical questions.  and I am always wary of efforts – no matter how well meaning they might be – that rely on someone’s intuition about what might work rather than empirical data about what will work and what will not.  And I am not seeing a lot of empirical data being used to justify this new mode of online commenting by PubMed.  Until we have that research available, it may make sense to maintain at least some of the weight-bearing walls of the ivory tower.”

What PubMed Commons will end up looking like will be informed by this sort of discussion. Lipman seems open to change:

“This is not an NIH project, except that NIH is allowing them to do it. We need to get the drive and direction from people who are going to be active in it.”

In a FAQ, the NLM calls PubMed Commons

“…a forum for open and constructive criticism and discussion of scientific issues. It will thrive with high quality interchange from the scientific community.”

Let’s make it truly open, and end that second sentence with “high quality interchange,” full stop.

Read more from Retraction Watch.

Trackback from your site.