Presentation at the JGU Mainz Minisymposium „Plagiate, Fälschung, Pfusch- Regeln für gutes wissenschaftliches Arbeiten“, 3.07.2015

1. How anonymous
post-publication peer review
uncovers bad science
Leonid Schneider,
science journalist
with Laborjournal
leonid.schneider@gmail.com
Twitter: @schneiderleonid

2. Junior scientists are often told by their advisors:
- If you can deliver this result,
you will publish a nice paper and have a job
- If you don’t deliver this result,
you will not publish any paper and have no job
Is bad science
individual or systemic failure?

3. $$$
Authors and institutions have little incentive to
produce reliable quality science
Paper-to-funding convertion
Funding used for research…

4. Journals and funding agencies prefer
simplistic, but sensationalist “break-through” science
• Cancer cure!
• Stem cells /”Reprogramming”
• One-Gene-Phenotype models
• Translational/Commercial
potential
Biological systems
are very complicated,
but in biological papers
simplicity rules!

5. Scientists occasionally help data to fit their
theoretical model for a publication
• Selective data acquisition and evaluation
(very common)
• “Adjustments” or manipulation of data
(less common)
• Data falsification / fraud
(very rare)

6. Peer review weeds out bad science. Really?
• Data is submitted on trust as
being honest/reliable
• Peer Reviewers are scientist
colleagues, not data specialists
• Peer Reviewers only analyse
science, not data integrity
• Peer review is not always done
diligently enough
How did this pass
peer review????

7. $$$
Traditional
peer
review
Traditional peer review is anything but transparent
Years and years of research…

8. - Journal Editors
- Decide on Quality,
Novelty, Impact
- Appoint peer
reviewers
- Make final decisions
- Peer Reviewers
- 1-4 people
- Unknown to authors
or readers
- Potential COI,
personal animosities,
lack of competence…
$$$
Too many financial and personal interests involved
Years and years of research…

9. Convincing peer reviewers is by far
the most important task of a scientist

10. A peer-reviewed paper is a badge of honour
Things surely changed for him
since he published in Nature…
• Publications are public
evidence of success
• They are to be admired
and not questioned
• Often not the content
counts, but where it is
published

11. Scientists waste time, money and their careers trying to
reproduce unreliable or manipulated results
• Poor reproducibility in combination with high competition
undermines productivity, but also work moral, trust and
motivation
• It leads to even more data manipulation and fraud in science

12. What do you do if you spot data irregularities or
irreproducibility in a published paper?
1. Write to authors
2. Write to journal
3. Write to authors’
institution

13. What happens if a published paper is reported
to be wrong or even contain manipulated data?
1. Correction (rare)
2. Retraction (even rarer)
3. Nothing (most common)

14. Your paper is wrong,
professor!
See you at the
exam…
Individual criticisms are unwelcome and dangerous
• Financial interests behind publications
prevent institutional investigations
• Institutions often refuse to react to
anonymous hints
• Whistle-blowers are often punished or
dismissed as incompetent or malicious

15. Solution: make valid criticisms public,
but anonymously!
• Publicly available valid criticisms
are much more difficult to be
ignored
• Whistle-blowers are protected by
the anonymity under which they
are free to post concerns

16. More on this:
http://www.laborjournal.de/editorials/424_11.lasso
Adam Marcus
Ivan Oransky
Retraction Watch takes whistle-blowers seriously

17. PubPeer allows anonymous post-publication
peer review, including evidence

18. On PubPeer you could ask critical questions anonymously

19. Or, you can post evidence of data irregularities on PubPeer,
also anonymously

20. I have some issues
with your paper, Sir!
Pros and cons of anonymous commenting
(aka witch-hunts)
• Protects whistle-blowers
• Only objective evidence and
arguments matter, not who has
raised them or why or where
• Unsubstantiated claims,
personal insults
• Sock-puppeting (also by authors!)
Against:
For:

21. PubPeer protects the anonymity, even when
the criticized scientists go to court

22. Blog post at Laborjournal about this interesting and
useful experience:
http://www.laborjournal.de/blog/?p=8281
Autors also reply to
PubPeer criticisms, often
constructively

23. STAP: one of the biggest fraud scandals uncovered,
thanks to post-publication peer review on PubPeer

24. Photo credit: Maigrot/REA
The case Olivier Voinnet
• PhD with Sir David
Baulcombe at
The Sainsbury
Laboratory, Norwich
• Research leader at
CNRS institute in
Strasbourg (age 33)
• Professor at ETH
Zürich (since 2010)

25. Photo credit: Maigrot/REA
The case Olivier Voinnet
• EMBO Gold Medal
(2009)
• EMBO Member
• EMBO Young
Investigator grant
• ERC start-up grant
• Max-Rössler-Prize
(ETH Zürich, 2013)

26. It started with people finding irregularities in
David Baulcombe’s papers

27. Olivier Voinnet publications
flagged on PubPeer

28. Olivier Voinnet and other involved reply on PubPeer
and promise to investigate

29. Vicki Vance
Professor of Botany, University of South Carolina
Vicki Vance: the key Whistle-Blower in Voinnet case

30. To be continued?