Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
How anonymous
post-publication peer review
uncovers bad science
Leonid Schneider,
science journalist
with Laborjournal
leo...
Junior scientists are often told by their advisors:
- If you can deliver this result,
you will publish a nice paper and ha...
$$$
Authors and institutions have little incentive to
produce reliable quality science
Paper-to-funding convertion
Funding...
Journals and funding agencies prefer
simplistic, but sensationalist “break-through” science
• Cancer cure!
• Stem cells /”...
Scientists occasionally help data to fit their
theoretical model for a publication
• Selective data acquisition and evalua...
Peer review weeds out bad science. Really?
• Data is submitted on trust as
being honest/reliable
• Peer Reviewers are scie...
$$$
Traditional
peer
review
Traditional peer review is anything but transparent
Years and years of research…
- Journal Editors
- Decide on Quality,
Novelty, Impact
- Appoint peer
reviewers
- Make final decisions
- Peer Reviewers
- ...
Convincing peer reviewers is by far
the most important task of a scientist
A peer-reviewed paper is a badge of honour
Things surely changed for him
since he published in Nature…
• Publications are ...
Scientists waste time, money and their careers trying to
reproduce unreliable or manipulated results
• Poor reproducibilit...
What do you do if you spot data irregularities or
irreproducibility in a published paper?
1. Write to authors
2. Write to ...
What happens if a published paper is reported
to be wrong or even contain manipulated data?
1. Correction (rare)
2. Retrac...
Your paper is wrong,
professor!
See you at the
exam…
Individual criticisms are unwelcome and dangerous
• Financial interes...
Solution: make valid criticisms public,
but anonymously!
• Publicly available valid criticisms
are much more difficult to ...
More on this:
http://www.laborjournal.de/editorials/424_11.lasso
Adam Marcus
Ivan Oransky
Retraction Watch takes whistle-b...
PubPeer allows anonymous post-publication
peer review, including evidence
On PubPeer you could ask critical questions anonymously
Or, you can post evidence of data irregularities on PubPeer,
also anonymously
I have some issues
with your paper, Sir!
Pros and cons of anonymous commenting
(aka witch-hunts)
• Protects whistle-blower...
PubPeer protects the anonymity, even when
the criticized scientists go to court
Blog post at Laborjournal about this interesting and
useful experience:
http://www.laborjournal.de/blog/?p=8281
Autors als...
STAP: one of the biggest fraud scandals uncovered,
thanks to post-publication peer review on PubPeer
Photo credit: Maigrot/REA
The case Olivier Voinnet
• PhD with Sir David
Baulcombe at
The Sainsbury
Laboratory, Norwich
• R...
Photo credit: Maigrot/REA
The case Olivier Voinnet
• EMBO Gold Medal
(2009)
• EMBO Member
• EMBO Young
Investigator grant
...
It started with people finding irregularities in
David Baulcombe’s papers
Olivier Voinnet publications
flagged on PubPeer
Olivier Voinnet and other involved reply on PubPeer
and promise to investigate
Vicki Vance
Professor of Botany, University of South Carolina
Vicki Vance: the key Whistle-Blower in Voinnet case
To be continued?
How anonymous post-publication peer review uncovers bad science
Upcoming SlideShare
Loading in …5
×

How anonymous post-publication peer review uncovers bad science

3,768 views

Published on

Presentation at the JGU Mainz Minisymposium „Plagiate, Fälschung, Pfusch- Regeln für gutes wissenschaftliches Arbeiten“, 3.07.2015

Published in: Science

How anonymous post-publication peer review uncovers bad science

  1. 1. How anonymous post-publication peer review uncovers bad science Leonid Schneider, science journalist with Laborjournal leonid.schneider@gmail.com Twitter: @schneiderleonid
  2. 2. Junior scientists are often told by their advisors: - If you can deliver this result, you will publish a nice paper and have a job - If you don’t deliver this result, you will not publish any paper and have no job Is bad science individual or systemic failure?
  3. 3. $$$ Authors and institutions have little incentive to produce reliable quality science Paper-to-funding convertion Funding used for research…
  4. 4. Journals and funding agencies prefer simplistic, but sensationalist “break-through” science • Cancer cure! • Stem cells /”Reprogramming” • One-Gene-Phenotype models • Translational/Commercial potential Biological systems are very complicated, but in biological papers simplicity rules!
  5. 5. Scientists occasionally help data to fit their theoretical model for a publication • Selective data acquisition and evaluation (very common) • “Adjustments” or manipulation of data (less common) • Data falsification / fraud (very rare)
  6. 6. Peer review weeds out bad science. Really? • Data is submitted on trust as being honest/reliable • Peer Reviewers are scientist colleagues, not data specialists • Peer Reviewers only analyse science, not data integrity • Peer review is not always done diligently enough How did this pass peer review????
  7. 7. $$$ Traditional peer review Traditional peer review is anything but transparent Years and years of research…
  8. 8. - Journal Editors - Decide on Quality, Novelty, Impact - Appoint peer reviewers - Make final decisions - Peer Reviewers - 1-4 people - Unknown to authors or readers - Potential COI, personal animosities, lack of competence… $$$ Too many financial and personal interests involved Years and years of research…
  9. 9. Convincing peer reviewers is by far the most important task of a scientist
  10. 10. A peer-reviewed paper is a badge of honour Things surely changed for him since he published in Nature… • Publications are public evidence of success • They are to be admired and not questioned • Often not the content counts, but where it is published
  11. 11. Scientists waste time, money and their careers trying to reproduce unreliable or manipulated results • Poor reproducibility in combination with high competition undermines productivity, but also work moral, trust and motivation • It leads to even more data manipulation and fraud in science
  12. 12. What do you do if you spot data irregularities or irreproducibility in a published paper? 1. Write to authors 2. Write to journal 3. Write to authors’ institution
  13. 13. What happens if a published paper is reported to be wrong or even contain manipulated data? 1. Correction (rare) 2. Retraction (even rarer) 3. Nothing (most common)
  14. 14. Your paper is wrong, professor! See you at the exam… Individual criticisms are unwelcome and dangerous • Financial interests behind publications prevent institutional investigations • Institutions often refuse to react to anonymous hints • Whistle-blowers are often punished or dismissed as incompetent or malicious
  15. 15. Solution: make valid criticisms public, but anonymously! • Publicly available valid criticisms are much more difficult to be ignored • Whistle-blowers are protected by the anonymity under which they are free to post concerns
  16. 16. More on this: http://www.laborjournal.de/editorials/424_11.lasso Adam Marcus Ivan Oransky Retraction Watch takes whistle-blowers seriously
  17. 17. PubPeer allows anonymous post-publication peer review, including evidence
  18. 18. On PubPeer you could ask critical questions anonymously
  19. 19. Or, you can post evidence of data irregularities on PubPeer, also anonymously
  20. 20. I have some issues with your paper, Sir! Pros and cons of anonymous commenting (aka witch-hunts) • Protects whistle-blowers • Only objective evidence and arguments matter, not who has raised them or why or where • Unsubstantiated claims, personal insults • Sock-puppeting (also by authors!) Against: For:
  21. 21. PubPeer protects the anonymity, even when the criticized scientists go to court
  22. 22. Blog post at Laborjournal about this interesting and useful experience: http://www.laborjournal.de/blog/?p=8281 Autors also reply to PubPeer criticisms, often constructively
  23. 23. STAP: one of the biggest fraud scandals uncovered, thanks to post-publication peer review on PubPeer
  24. 24. Photo credit: Maigrot/REA The case Olivier Voinnet • PhD with Sir David Baulcombe at The Sainsbury Laboratory, Norwich • Research leader at CNRS institute in Strasbourg (age 33) • Professor at ETH Zürich (since 2010)
  25. 25. Photo credit: Maigrot/REA The case Olivier Voinnet • EMBO Gold Medal (2009) • EMBO Member • EMBO Young Investigator grant • ERC start-up grant • Max-Rössler-Prize (ETH Zürich, 2013)
  26. 26. It started with people finding irregularities in David Baulcombe’s papers
  27. 27. Olivier Voinnet publications flagged on PubPeer
  28. 28. Olivier Voinnet and other involved reply on PubPeer and promise to investigate
  29. 29. Vicki Vance Professor of Botany, University of South Carolina Vicki Vance: the key Whistle-Blower in Voinnet case
  30. 30. To be continued?

×