Notice: load_plugin_textdomain was called with an argument that is deprecated since version 2.7 with no alternative available. in /var/www/html/wp-includes/functions.php on line 3320
Peer Review Is Dead! Long Live Peer Review! « The Central Line

Peer Review Is Dead! Long Live Peer Review!


CC from Flickr user wenzday01

CC from Flickr user wenzday01

The internet has fundamentally changed the way we understand and interact with the world: not just as physicians, but in our daily lives; however medicine (especially academic medicine) still lives in the dark ages. The ways of old are starting to show signs of wear, that this is beginning to change. And as things in the age of information move at an ever more-rapid pace, I think the changes will be here before we know it. I, for one, welcome our new data overlords.

The medical journal was initially created as a forum: a way to publicly share information with your colleagues, and get credit for the discovery. Say you wanted to tell the world of a new surgical technique. Or a new drug that you’ve discovered to help your patients. You could discuss it with a few colleagues in the hospital. But if you think you’re really onto something — something that really might be great and really might help not just your patients, but everyone’s — you’ve got to spread the word. And that’s how the journals started. Not with research, but with physician opinions and approaches and case reports and “Hey look what I found out”s. If you go back to the early publications of the New England Journal of Medicine — which now allows you to search from their archives from 1812 on – you can see some pretty cool stuff. Punch in your favorite subject and you’re transported back in time to when physicians like you were still trying to figure out what the hell was going on with this patient, instead of the biochemical cytokine pathway of today. It’s pretty incredible.

So here’s my first point: look how we share information today (and honestly, we’re just getting started): Twitter, Facebook, emails, blogs, text messages, Google, Wikipedia. Sure sure, we still share some very important information through medical journals, but they simply can’t keep up. Hundreds of new medical journals are launched every year, for everyone’s own sub-sub-specialty out there. Yet the hunger for publication and knowledge continues to grow. Let’s just consider the case report, for example. Imagine you’re staffing a hospital in the late 70s/early 80s in New York, or San Francisco, or Los Angeles, and you find these small crops of patients with really, really weird infections. You scratch your head, dig in a little deeper, and publish what you’re finding in the New England Journal of Medicine in the December 10, 1981 edition. Four months later, several replies are published: it’s marijuana use; no no, it’s the amyl nitrates that the gay men are using; of course not, it’s the CMV they’ve been exposed to; no, you’re wrong, this is something entirely new we’ve never seen before. It’s an absolutely fascinating read of the natural course of HIV’s research pattern, but one that I imagine would be very different today (and will be different when the next HIV/AIDS-like disease hits):

(Sorry for sticking words in your mouths, gentlemen.)

(Sorry for putting hypothetical words in your mouths, gentlemen.)

Okay okay, so fine, that’s just case reports. And medicine and science and the scientific method evolved, and It Was Good, and then medical journals became the place to publish research. Big trials. Lots of money. Which brings me to my second, unforunate point: peer review is not all it’s cracked up to be. Some concerning data (ironically, yes, published in the journals):

Now, I’m not saying that peer review should be discarded, or that journals should cease to exist, or that we should throw the baby out with the bathwater. I am, saying, however, that I think there’s room for another option, using the internet, social networks, and crowdsourcing. (NB: In this topic I am building on existing ideas from Chris Nickson/LITFL’s Time to Publish Then Filter? and The Wisdom of Crown Review which also references these BMJ and Annals of EM opinion pieces.) I agree with Chris: I don’t know exactly what form this should take, but something like an academic Twitter (Trip Database’s TILT?) might not be a bad start. I hate to make this all a popularity contest (mostly because I lost those so vigorously in high school), but the cream typically rises to the top when something is put to the crowds.  (But sadly, not always. Okay, at least, the academic crowds.)

Or perhaps it’s meta-reviews of the data. It’s online Critical Care Journal Clubs, or it’s a rating system to articles with ratings from colleagues you like and trust (and who know the literature better than you) like Leon Gussow’s 5/5 Skull and Crossbones at his Toxicology blog. Or podcasts reviewing a single topic. I’m not sure if it’s centralized. Who knows. Someone will build it and get it right (maybe me?) and we’ll go from there.

And all these great online links and resources lead me to my final point: “academic” works cannot and should not be limited to the length of one’s search in Pubmed as author. Yes yes, I’m suggesting the beginning of an academic new world order, and should be burned at the stake for such heresy (especially since I’m going into academics). But “publish or perish” should not simply mean “get your name in a journal.” Academics is the pursuit of knowledge, the pursuit of teaching and education. Case in point: Rob Reardon, narrator of so many of those fantastic ultrasound videos that I’m forever loving, is a well-published article in the journal world as well. But I guarantee you this: the amount of education that Rob has produced on his website — and that people have learned from — already exceeds the amount of whole-world educational impact of his Pubmed career. It’s simply exposure from the internet versus exposure through one journal.

Like-minded people (frequently education-minded, tech-oriented like myself) are doing this all over the web. They’re frequently (but not always) affiliated with some sort of academic place — be it an official medical school or simply an area where residents rotate — and do it because they enjoy it. And none of it would make it into a journal article. It’s too short, or too fast, or too digital, or simply too practical — but yet clearly useful. And it should be valid and appropriate academic work, recognized by our peers. (Let the crowds do the peer-reviewing of these publications if they like. Don’t like one of Rob’s videos, or disagree with him on something he says? Leave a comment or send a message on Twitter for all the world to see.)

There is a huge, huge volume of really high-quality learning on the web, especially in Emergency Medicine (much of which I’ve documented here), and it’s only becoming better.

Journals are here to stay — and I welcome them. They provide an important resource to develop and publish research and trials, and are still the biggest forum available to spread one’s medical ideas. But at the same time, there is content and ideas and a wealth of knowledge and information-sharing going on that is occuring not in sequence — but in parallel with them. Information that is simply out of the realm and scope of the journals and old-fashioned peer review. We are starting to develop the tools to share this information, and I look forward to where the next 10 years take us. (Hopefully to at least a modicum of technologic advancement in the snail’s pace at which medicine frequently changes.)

  1. #1 by Chris Nickson - January 11th, 2011 at 10:03

    Brilliant post Graham. We live in exciting times.

    Thanks for mentioning my posts, which were really just an overflow of the excitement I felt in reading the papers in the BMJ and the Annals. I can’t begin to imagine what the future holds for peer review, but I feel the momentum building and I am certain that things will change and that social media and the web will be part of the revolution.

    The late Denis Dutton, a countryman of mine and creator of the Arts and Letters Daily (http://www.aldaily.com/) once said, “The great thing about the internet is that there is no gatekeeper, and the problem with the internet is that there is no gatekeeper”. At LITFL we’ve recently initiated a new project called ‘The LITFL Review’ (http://lifeinthefastlane.com/2011/01/the-litfl-review-001/) which provides a weekly summary of of all the great emergency medicine happening from the world of blogs, podcasts, online video, Twitter, and other social media. We don’t want it to be a gatekeeper, but it may provide a gateway for the uninitiated. We hope to foster the online and social media emergency medicine community that I believe will become an important force in determining the future of our wondrous specialty.

    All the best,
    Chris

  2. #2 by Med School Odyssey - January 12th, 2011 at 15:33

    The recent scandals involving fraudulent research suggest that peer-review is not the effective means to keep scientists honest that many seem to think that it is. As an example, I point to the fact that it took 12 years before Andrew Wakefield’s autism-vaccination scam was finally declared as fraudulent by the medical community. Similar examples of delayed-judgment are legion.

    I’d like to think that, as the internet continues to enable faster dissemination of information to more people, this sort of thing will fade into history, but I doubt it. Laypersons tend not to have the background that would enable them to sort out fact from fiction when it comes to things like medical research, which explains why anti-vaccination advocates seized upon Dr. Wakefield’s bogus findings. When you couple that with high-profile and uneducated wonks like Jenny McCarthy preaching the evils of vaccines, it’s not hard to see the damage.

    The public places an enormous amount of faith in the scientific community to police itself when it comes to the fidelity of their research. The reverence which physicians claim to hold for laying their name to a prescription needs to apply to peer-review as well. I don’t mean to indict the reviewers of Dr. Wakefield’s research as co-conspirators, but one wonders whether they should bear some of the guilt here.

  3. #3 by OD - February 12th, 2011 at 13:10

    Brilliant posts. It’s been three years since I read something from you…

    However, so much emphasis is placed on how much you published in peer reviewed journals with good impact factors. You simply cannot get a good NIH grants unless you have ample publications. Sharing information via a non-peer review format is definitely in the future but again, how do you somewhat verify the presented information. I’ll admit I don’t have the answer to that question.

    I do like writing and publishing because it forces me to read something new beyond the scope of gaining medical knowledge:I want to contribute to academic medicine. Plus, I like to write. However, not everything needs to be reviewed by some expert in her dark office. When you have “undesirable” results/outcomes, it may save another patient’s life or another hospital some money if you tweet or blog about it. Case series seriously do not need rigid-6-month reviews before they get rejected. The EBM score is low anyway so why tear it apart.

    However, I do think the randomized controlled trials, prospective cohort studies and practically all the basic science research/work should pass the peer-review test before public recognition. Call me old school…

    PS: The Andrew Wakefield thing is an exception. It was already a controversial issue that many in the medical community didn’t buy anyway…but a lot of mothers did. Sad.

  4. #4 by OD - February 12th, 2011 at 13:11

    Maybe I should use spellcheck the next time…

(will not be published)
*

  1. No trackbacks yet.