Publishing Fluff

We all know that the number of papers published every year is growing rapidly, but does that mean we are more productive and more creative? Some of the old timers (you know the type) will tell you, no new research is being done, we are just rehashing old science and incrementally improving on it. Is the quality of papers decreasing ? Some data seems to point that way:

"While brilliant and progressive research continues apace here and there, the amount of redundant, inconsequential, and outright poor research has swelled in recent decades, filling countless pages in journals and monographs. Consider this tally from Science two decades ago: Only 45 percent of the articles published in the 4,500 top scientific journals were cited within the first five years after publication. In recent years, the figure seems to have dropped further. In a 2009 article in Online Information Review, Péter Jacsó found that 40.6 percent of the articles published in the top science and social-science journals (the figures do not include the humanities) were cited in the period 2002 to 2006."

The problem is that as the quantity of fluff paper increases, peer review suffers because the system is overwhelmed, and the bar lowers.

Newsflash: scientific publishing is controlled by cliques

14 leading stem cell scientists have written an open letter criticizing the undue influence of small groups of scientists on what gets published in journals. Of course this is not news to anyone in the field, and a pervasive problem in science.

Here are some excerpts and opinions of the letter:

" Stem cell experts say they believe a small group of scientists is effectively vetoing high quality science from publication in journals.In some cases they say it might be done to deliberately stifle research that is in competition with their own. "

" The journal editor decides to publish the research paper usually when the majority of reviewers are satisfied. But professors Lovell-Badge and Smith believe that increasingly some reviewers are sending back negative comments or asking for unnecessary experiments to be carried out for spurious reasons. In some cases they say it is being done simply to delay or stop the publication of the research so that the reviewers or their close colleagues can be the first to have their own research published. "

" "Editors should be able to see when reviewers are asking for unnecessary experiments to be carried out and if it's the difference between an opinion of the referee and a factual problem. But what tends to happen is that the editor takes the opinion of an editor rather than the factual substance," he said. One of the main reasons for this, according to Professor Smith, is that journals are in competition. Editors have become dependent on favoured experts who both review other people's stem cell research and submit their own papers to the journal. If the editor offends these experts, they may lose future papers to a rival. This is leading to the journals publishing mediocre science, according to Professor Lovell-Badge. "

How to publish a comment

We've all read papers that didn't pass the smell test. I'm willing to bet none of you ever tried to publish a comment in the journal debunking the conclusions of said paper. This PI tried and as you can see, it's nearly impossible to get it published perhaps because it would involve the editors acknowledging that they shouldn't have published the paper in the first place. I highly recommend you read this saga, it is both funny and tragic. Here is how he proposes to fix the system:

1. All data and parameters associated with any open publication should be available to anyone interested in it. The NIH has mandated this for its grant recipients, but sharing data and parameters should also be a required condition for a publication in any journal. Refusing to do so after a paper is published should be considered scientific misconduct.

2. Anyone knowingly publishing a paper that clearly contradicts the work of another group should be required, also as a condition for publication, to discuss the matter with that group well before publication. In the past, this was considered good
scientific etiquette, but gone, apparently, are those days, so a rule is in order.

3. Journal editors should be more aware of referee conflicts of interest. Reviewers should be required to stipulate any conflict of interest in reviewing a paper, even if it’s simply that they don’t like the authors.

4. No journal editor should be allowed to edit a Comment on a paper he allowed to be published. This is an obvious, unacceptable conflict of interest.

5. Comments should not be required to be so short as to prevent them from making sense. I suggest two journal pages, or, better, three. Or how about this radical idea: they should be as long as it takes to make the point.

6. Crazy rules that allow logically offensive situations, like the one that called for rejecting a Comment because the Reply is unpublishable, should be deleted immediately. And Comments and replies need not, and should not, be published together. Indeed, a Comment on a Reply is a good idea, yielding an interesting ongoing dialog that would benefit the community.

7. The reviewers who review a Comment should also review the Reply. They’re the best qualified, as they’re already familiar with the work. This would prevent the insane situation that occurred here, in which the highest quality review of the Comment was simply lost.

8. Reviews should themselves be reviewable. Currently, reviewers can say whatever they like, and there is no check on them. Authors should be allowed to nominate
irresponsible reviewers, such as Reviewer #2 in the above scenario. Confirmed irresponsible reviewers should then be identified and removed from reviewer databases, which would be shared with other journals. Writing an irresponsible review
should be considered a form of scientific misconduct.

9. While removing unethical reviewers would help, improving reviews of ethical ones is also important. Currently there is no compensation of any sort for reviewers and hence no encouragement to do a good job. I believe that reviewers should be paid for their services. People take paid jobs much more seriously than volunteer efforts. Knowing this, social psychologists pay their subjects simply to fill out questionnaires because it yields much higher-quality results. And what could be more important than the accuracy of the archival scientific literature?

10. Require scientific ethics courses in grad school. Problems like those that I encountered are a proverbial ticking time bomb for science. What if those opposed to taking action against global warming were to make the claim that science shouldn’t be believed in this matter because its process is so rife with poor ethics that it can’t be trusted?

Should copyrights be abolished in academia?

We've all been exposed to strange situations where we had to ask for permission to use our own figures or data from a paper into another paper, or a thesis etc... And the fact that one can retain copyrights is in my opinion one of the best benefits of some open-access journals. But Steven Shavell thinks we should go even further in this paper:

"The conventional rationale for copyright of written works, that copyright is needed to foster their creation, is seemingly of limited applicability to the academic domain. For in a world without copyright of academic writing, academics would still benefit from publishing in the major way that they do now, namely, from gaining scholarly esteem. Yet publishers would presumably have to impose fees on authors, because publishers would not be able to profit from reader charges. If these publication fees would be borne by academics, their incentives to publish would be reduced. But if the publication fees would usually be paid by universities or grantors, the motive of academics to publish would be unlikely to decrease (and could actually increase) – suggesting that ending academic copyright would be socially desirable in view of the broad benefits of a copyright-free world. If so, the demise of academic copyright should be achieved by a change in law, for the ‘open access’ movement that effectively seeks this objective without modification of the law faces fundamental difficulties."

Getting Rid of Scientific Noise: Lifetime Publication Limits?

Here's an interesting thought experiment proposed in a Nature Medicine editorial: what if each scientist could only publish 20 papers during his/her entire career?

"you get 20 tickets. Every time you publish a paper, you hand over one of them. Once you run out of tickets, your publishing days are over. As simple as that.

If we adopted this model, many articles reporting incremental advances would no longer be written, and many specialized journals would disappear. And with far fewer papers to read, each one reporting a much more complete piece of research, search committees or funding bodies could directly evaluate the work of a given scientist, instead of (as is often the case) leaning on surrogate indicators such as a journal's impact factor or number of citations.

At the extreme, we might not even need journals (and editors) anymore; everything would be published in preprint servers like those used by physicists, and the community would simply evaluate and rank the different contributions as they become available. This way, the whole community could act as reviewers, doing away with the existing peer-review process. This is somewhat reminiscent of what some websites are already trying to do, so far with limited success. But if everybody agreed to publish just 20 papers to keep the size of the literature manageable, then the journal of the future might conceivably be a preprint server."

It's a thought-provoking proposal, and I would certainly love to see a renewed focus on quality, rather than quantity in scientific research. Good science requires time and commitment to a specific problem, not a mad rush to produce meaningless data and minimum publishable units. You know there's a problem with the system when even at the undergraduate level, many students don't even seem to feel they have the time to learn how to pH solutions or perform basic assays because they're so busy calculating the shortest possible path to getting their names on their lab's next paper.

However, imposing an arbitrary lifetime publication limit is not a practical solution whatsoever. First of all, it's impossible to say what the appropriate number of lifetime "quality pubs" is going to be, and that number is going to vary from one researcher to the next, one field to another, etc, etc. Second, such a proposal does not address the issue of career advancement through the academic hierarchy. Currently, the total number and impact factor of pubs is what gets nascent scientists recognition, respect, jobs, and funding. What we need if we want to get back to a focus on quality over quantity is a shift in scientific culture, not a voucher system.

On open-access and peer review

We've covered this topic often on both this blog and on the podcast but it may be time to revisit it one more time. Open access (OA) publishing has won the heart of most scientists. By definition our work only gains value by sharing, and sharing is easier when it's free. As an author of papers, I can also retain copyrights over my own work so that I am free to do whatever I please with it, including reproducing it in my thesis or posting it on my blog. I mean what's not to love. Well some people actually oppose it, obviously people who have a vested interest in the multi-million dollar publishing business. PLOS, one of the pioneers in open-access, has basically proven that not only can the idea work, but it can make for a high profile journal. But with the introduction of PLOS-ONE it may have shown its Achilles' heel. See the opponents of open-access always pointed out that openness would destroy peer review. Obviously a ridiculous assertion, since peer review is already open and free. But with PLOS ONE peer review was changed from outside "random" assigned expert to "in-house" academic editors. In fact I strongly suspect that Nature's precedings, was built to fail, an exercise in futility, to show how openness equals lack of peer review and craptacular creationist papers published. And while the opponents of open access like PRISM might have an axe to grind, they also make a good point, what is peer review, and who oversees peer reviewers?

First lets check out what they say one their site: what's at risk?

• undermining the peer review process by compromising the viability of non-profit and commercial journals that manage and fund it;

• opening the door to scientific censorship in the form of selective additions to or omissions from the scientific record;

• subjecting the scientific record to the uncertainty that comes with changing federal budget priorities and bureaucratic meddling with definitive versions; and

• introducing duplication and inefficiencies that will divert resources that would otherwise be dedicated to research.

The first point could have been better formulated. But what I think they are trying to say is "hold-on you guys can't police yourself, you need publishers to manage peer review". I mean they don't actually fund peer review since it's free but they do manage it. I don't see why editors of OA journals can't do the same, but it does raise the need for a peer-review watchdog (more on that later)

The second point is even more asinine. They're saying "Hold-on you can't choose what can and cannot be published, that's censorship, let corporations do that job". This is ironic since the purpose of journals like PLOS ONE is to let people publish based on the quality of the science not the importance of the findings. Importance is subjective, and often Nobel winning discoveries like PCR or antibodies were rejected from publication.

The third argument is also flawed because funding shouldn't affect OA journals, they are run by the publication costs defrayed by the authors. In fact one could argue that traditional private publishing is at the mercy of investors, stock market crashes and economic conditions.

Finally duplications could actually be a problem, especially for journals like PLOS ONE. If papers are not discriminated upon the basis of importance, very small incremental findings could be published which do not really lead the field forward. That being said, I would rather have more information than less. In fact if information is freely available, perhaps you would be less likely to unknowingly duplicate someone's work that was published in an obscure journal to which your institution doesn't have access.

Then what's the answer?:

Well one big step in the right direction was the creation of BPR3 an acronym coined by none other than Kevin Z (of bayblab podcast fame). Bloggers for peer review reporting aims to overview the peer review process, and check out their take on PLOS ONE. Ultimately we need some kind of watchdog, a "peer-review watch", and that it could be born in a blog just shows you the power of web 2.0 and openness.

In the mean time, why not check out this spoof of PRISM (the axis of evil publishers), called PISD (I think it is pronounced [piss]-t):
"We don't believe it's a publisher's place to make the value judgment that dissemination of scientific information is a greater social good than the increase in capital associated with charging higher subscription rates. That judgment should be made by politicians, ethicists, economists, and other valuable members of society. If you would like to know more, we suggest you consult review articles on the theory of utility or the laws of supply and demand. If your institution doesn’t provide access to such articles, we'll be happy to sell them to you at a reasonable price."

Finally, a Digg for Real Scientists

I got a email today inviting me to try out DiscoveR8, a site that is now in beta testing. For those who know Digg, it's basically that but for peer-reviewed scientific papers. For those who have been hiding under a rock for the last year or so and don't know Digg, the concept is that anyone and everyone can submit their own papers or other papers they'd like to share with others who frequent the website. Anyone who comes across your post then clicks the button next to the story to give it the thumbs down or thumbs up and comments for discussion can also be posted. Based on user voting the website keeps track of the most popular papers. How long until your Discover8 ranking is more important than impact factor?