Friday, May 18, 2007

Bugs in PLOS ONE

PLoS ONE is the sujet du jour in the bay today. The quality of the submissions in PLoS has been increasing, but how long will that last? We had a journal club concerning a new report in PLoS about P53 mutations in ovarian cancer suggesting that patients with mutated p53 actually respond better to chemotherapy, probably because they have a harder time repairing damage, but can still undergo apoptosis in a p53-independent manner. The findings are not as novel as they might sound since it's been known for a while that in some instances, the oh-so-famous p53 tumour supressor can misbehave. P53 loss of function renders cells suceptible to DNA dammage and p53-mediated senescence of stromal cells may be required for the initiation of certain types of tumour. At the end of the talk, some of the PIs expressed their mistrust of this free-for-all that is PLoS ONE. Is this a generation gap in science? Are we just so used to the wikipedia, facebook, youtube, digg, blog sharing networks that we think science should be freed, liberated. I think if we've learned anything from the internet, it is that the masses are generally less intelligent than the individuals that they are comprised of.

Take the latest hot paper in PLoS ONE, "Order in spontaneous behavior". In that paper they hooked up fruit flies to a flight simulator, and because the fly can generate erratic patterns of flight that are endogenous to their neuronal circuitry at that instance, and not merely pre-wired, the authors concluded that they have a form of free will. Free will as the author points out, is an oxymoron: "the term ‘will’ would not apply if our actions were completely random and it would not be ‘free’ if they were entirely determined. So if there is free will, it must be somewhere between chance and necessity - which is exactly where fly behavior comes to lie."

This echoes what Einstein believed: "I don’t believe in the freedom of the will. Schopenhauer’s saying, that a human can very well do what he wants, but can not will what he wants, accompanies me in all of life’s circumstances and reconciles me with the actions of humans, even when they are truly distressing. This knowledge of the non-freedom of the will protects me from losing my good humor and taking much too seriously myself and my fellow humans as acting and judging individuals".

Which leads me to the quality of the reviewers on PLoS. Most PI's are obviously too busy to give free time to peer-review stuff on the internet that wont get them any type of recognition. Furthermore it blurs the line between "expert in the field" peer review and "random degenerate grad student" review. And really, when the best young minds are free to write anything on the internet, what comes out is probably "I for one welcome our new cyborg fruit fly overlords". What we lack is accountability and a positive reward in your career from contributing to peer-review, or publishing in non-traditional journals (is it even a journal?).

On the one hand some of the reviews appear quite adequate, yet some seem to be overly philosophical, probably because they were written from out-of-field scientists and not experts: "The findings actually have nothing to do with free will. Free will is a feeling I have (when I do something deliberately) that I am doing what I am doing because I feel like it: a feeling that my willing it is the cause of my doing it. It is undeniably true that that is what it feels like to do something deliberately. But whether what feels like the cause -- feeling -- is indeed the cause of my doing is an entirely different matter. The real cause might, for example, be a fractal order mechanism of the kind reported by Maye et al. But that mechanism is the causal mechanism it is irrespective of whether it happens to be accompanied by (or generates) feelings. And it certainly does not explain how or why we (let alone the fruit fly) feel anything at all. And without feeling there is no free will, just mechanisms, whether deterministic or nondeterministic -- unless we are ready to believe in telekinesis."

But what really took me over the edge, is this blog spamming on that paper's annotations. It's one thing for the bayblab to spam digg, but this is a taste of what's to come to the scientific discussion and peer review process if it remains open...


Bayman said...

I agree science will always need the expert peer-review process, so I don't think PLOSone will be a mainstream alternative for scientific publishing anytime soon. From the looks of it, it probably won't be long before it's stigmatized with a reputation for publishing interesting, but sketchy theories with weak experimental evidence (ie Journal of Theoretical Biology), because those who have strong data simply cannot afford to choose PLOSone over a prestigious classical journal. Still, it could fulfill a useful role in stimulating and provoking counter-mainstream ideas.

But back to classical must keep it I think, but the power and SPEED of the system could be greatly enhanced by taking advantage of modern communications technology facilitated by the web. Those who have published their own papers will know that currently available services that journals use for uploading, editing and reviewing manuscript absolutely SUCK! They're almost slower than just mailing the damn things in.
Why don't journals have their own blog-type discussion forum (private and password protected) where a larger number of reviewers can discuss the paper? A transcript of the discussion could be provided to the authors as commentary. Just one obvious example...

Hopefully as more and more scientists are increasingly web-savvy, there will be a natural shift to using technology to discuss ideas and colaborate etc...I think a progression to open science has to be driven by scientists themselves, rather than some journal publisher. In the same way that the New York Times and classical publications will never lead the way for bloggers and newsgroups and the like, or as Microsoft or Apple will never drive open-source application development, scientists themselves will have to be the ones who usher in an OpenScience. That's how this stuff has to happen...not by someone sitting around and saying "ok, open publishing will start NOW", but by people who actually do work gradually exploring new ways to harness technology to their advantage

Bayman said...

Speaking of uploading data to huge limiting factor in using digital technology in science is the lack of standard formatting for data. Some journals want .jpg this, .tiff that, this resolution that resolution, not to mention the problems with microarray, proteomics data etc, that prevent one experiment from being compared to another etc. Now that I think of it, I will say that digital standardization is the biggest current limitation to the progress of biological science...ultimately, this comes down to restructuring the very labs we work in to make them digitally enabled...instruments that collect digital data that is compabtible with a common standard, direct uploading of that data of digital lab books. Access terminals/digital lab book stations at every work area. Only then will web-enabled science be truly possible. That's the way it's gotta be!

kamel said...

Scooped on the bayblab! I was just talking with Rob about posting the fly story yesterday.

I agree with a lot of Bayman's points about the importance of expert peer review. PLOSone is peer reviewed with focus on technical concerns, but articles aren't weeded out on the basis of subjective concerns such as importance to the field. As Bayman points out, this can lead to 'interesting but sketchy theories', but the flipside is that the openness means more access to smaller findings or negative results (speaking from experience, I can tell you that those aren't so easy to publish in a 'classical' journal).

The open comments, too, are a double-edged sword. While having them open to the public can lead to the sort of blogspamming AC alluded to, it also engages the general public and involves them in the process. It encourages scientific literacy and awareness which I think, in turn, increases research funding. Science discourse shouldn't be limited to the so-called 'ivory towers'. Regarding the drosophila story, I definitely noticed that the author(s) engaged in discussion not only at PLoS but also left comments on a number of blogs reporting the story, which I think is great.

kamel said...

Just another minor point about the blogspam. The blog that was posted in the annotations belongs to the senior author on the paper being discussed, wasn't posted by the blog owner and the link leads to a list of science blogs that are discussing the story which is a bit different from the bayblab heading over there and posting a link to ourselves saying 'check out our commentary here'

bjoern said...

Very valid points. May I add a few of my own:
- I personally think PLoS One needs to keep the open, but moderated forums attached to the papers (for the reasons mentioned already).
- As pointed out, none of the authors have any relation to the author who posted my blog address on our paper (at least the nick doesn't tell me anything).
- Two of the three reviews are now posted with our paper (they improved it greatly - it took us 6 months to implement and calculate the suggestions)
- I think it's one of the truly great things about PLoS One that it's NOT some whim of an editor which decides if your work is worth publishing! *Ideally*, all science would be published there if rationality and not vanity were guiding every scientist's submission strategy. :-)

Bayman said...

Great points! I guess you have just given the ultimate proof that PLOS is working the way it's supposed to, when you read a paper and commentary, write about it on your blog, and the original author joins the discussion! I applaud your attitude re publishing- science would be better off if more people follow your lead.

Bayman said...

By the way, Bjoern has a greatwebsite and blog. Definitely worth a visit.