Recommended
Recommended
More Related Content
Similar to Publication bias- Chris Graf 2018
Similar to Publication bias- Chris Graf 2018 (20)
Recently uploaded
Recently uploaded (20)
Publication bias- Chris Graf 2018
- 1. publicationethics.org Publication bias COPE: Promoting integrity in research and its publication Chris Graf, Co-Chair, COPE, Committee on Publication Ethics, at the first UK Research Integrity Forum, UUK, March 2018. ORCID ID: 0000-0002- 4699-4333. And with contributions from COPE Trustees Deb Poff, Geri Pearson, Charon Pierson, Michael Wise. Disclosure: CG volunteers for COPE, Committee on Publication Ethics. CG works for Wiley.
- 2. publicationethics.org ‘‘ Kay Dickersin, Department of Epidemiology, Johns Hopkins University. JAMA. 1990 Mar 9;263(10):1385-9. Publication bias is the tendency on the parts of investigators, reviewers, and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings Publication bias
- 3. publicationethics.org Problems that arise • Authors who write poorly or don't write in English do not get heard. • Null results are not published. • Replication studies are not done because they won’t advance a researcher’s career. • Gives the impression that certain things are more significant than they really are.
- 4. publicationethics.org It’s important that research publishers and editors examine their role What we do when we publish research influences how research is done, as well as how it is communicated. • That’s not what was intended by anyone, and it’s unfortunate. • It’s interwoven with how academic careers are made, how research is funded, how discoveries are rewarded and celebrated.
- 5. publicationethics.org Stepping back, what’s a research publisher’s (and editor’s) mission? • Do we mostly filter research, present the best, highest impact research back to researchers, so they know where to devote their (precious, limited) time and attention? • Do we mostly help researchers communicate all good quality research in its best possible form, and to the most relevant and widest possible audience?
- 6. publicationethics.org Consider what we aspire to On rock stars and jazz bands and new ways to think about quality
- 7. publicationethics.org 6 Rock stars Image rights: CC BY-SA 2.0 https://www.flickr.com/photos/evarinaldiphotography/6966856933 Taylor Swift Nobel prizes 821 Nature articles 40.137Nature Magazine’s Impact Factor
- 8. publicationethics.org 17million Jazz bands Image rights and credit: CC BY 2.0 https://www.flickr.com/photos/infrogmation/3042501580 Preservation Hall Jazz Band, St. Peter Street, French Quarter, New Orleans Academics and post docs 1,803,249research articles in Scopus 1 Median citations in 2017 to articles from 2016
- 9. publicationethics.org It’s all about rock stars We need to help everyone aspire to something different (as well). •A way for members of the jazz band to aspire to transparency, feel recognized, and be rewarded when they are transparent (see: TOP statements https://osf.io/sm78t/) •A way for research publishers and editors to aspire to quality standards built around integrity and ethics first, and (perhaps) then impact (see: Better Peer Review work at Wiley, to be published).
- 10. publicationethics.org Practically: New ways of working • Pre-registration (including clinical trials registration, but beyond that) https://cos.io/prereg/ • Registered Reports (an innovation in peer review) http://bit.ly/2FSKX9i • Other ways for researchers to communicate their research, and to be recognised for it (for example, data, preprints) • Transparent peer review (unsigned, or signed) http://bit.ly/2GOxDzS • Writing workshops and other support for authors who struggle to write in English
- 11. publicationethics.org ‘‘ Siddhartha Murkerjee, The Emperor of All Maladies: A Biography of Cancer ‘‘ “Murkerjee shows this brilliantly in this book about the history of the war on cancer. Science was diverted down many rabbit holes by chasing funding, fame, and publication,” says Charon Pierson, COPE Secretary. “We really need to reduce barriers to access through innovation so all voices (Taylor Swift and members of jazz bands, to carry forward the analogy) have an equal shot at discovery. As editors and publishers we need to focus first on integrity, then on impact. But this is one of those wicked problems: everyone needs to be part of the solution.” Every experiment is a conversation with a prior experiment, and every new theory a refutation of the old Conversations we have with prior work are biased
- 14. publicationethics.org 12,000+ members, 100+ countries • As an organization, COPE’s role is to assist editors of scholarly journals and publisher/owners in their endeavour to preserve and promote the integrity of the scholarly record through policies and practices that reflect the current best principles of transparency as well as integrity. • COPE is a membership organization. Our members are primarily editors of journals and publishers although we are currently exploring expanding our membership. Part of this potential expansion is being explored with a pilot project with five universities around the world. • COPE operates, manages and governs the organization with a small group of paid employees and a large group of very active volunteers who serve on the trustee board and council.
- 15. publicationethics.org 30+ Council members Lead all the work of COPE, Subcommittees, Working groups 12,000+ members 10+ Trustees Members of Council with legal responsibilities for COPE Vote Vote Appoint
- 17. publicationethics.org ‘‘We need a culture of responsibility for the integrity of the literature… it’s not just the job of editors Ginny Barbour, 2012—2017 COPE Chair
Editor's Notes
- “Publication bias is the tendency on the parts of investigators, reviewers, and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings. Much of what has been learned about publication bias comes from the social sciences, less from the field of medicine. In medicine, three studies have provided direct evidence for this bias. Prevention of publication bias is important both from the scientific perspective (complete dissemination of knowledge) and from the perspective of those who combine results from a number of similar studies (meta-analysis). If treatment decisions are based on the published literature, then the literature must include all available data that is of acceptable quality. Currently, obtaining information regarding all studies undertaken in a given field is difficult, even impossible. Registration of clinical trials, and perhaps other types of studies, is the direction in which the scientific community should move.”
- Gives the impression that certain things are more significant than they really are, due to more stuff having been published on them. We now have excellent tools to examine the panoply of bacterial species in our gut, say, ignoring all fungal and viral species that are also resident there. There are few papers on effects of treatments on women and children, leading to the sometimes (often?) false conclusion that whatever applies to white males also applies to them
- So we need to step back and ask ourselves, what’s our mission as research publishers and editors? Do we mostly filter research, present the best, highest impact research back to researchers, so they know where to devote their (precious, limited) time and attention? Do we mostly help researchers communicate all good quality research in its best possible form, and to the most relevant and widest possible audience? Or put another way, is research publishing: A service to researchers-as-readers? A service to researchers-as-authors?
- And does it have to be one thing, or the other? It’s all about rock stars right now (I’d say). So maybe we need to help everyone aspire to something different as well – perhaps a way for members of the jazz band to aspire to transparency, and feel great about that (see: TOP statements), or for research publishers and editors to aspire to quality standards build around integrity and ethics first, and perhaps then impact (see: Better Peer Review work at Wiley, to be published).
- So we need to step back and ask ourselves, what’s our mission as research publishers and editors? Do we mostly filter research, present the best, highest impact research back to researchers, so they know where to devote their (precious, limited) time and attention? Do we mostly help researchers communicate all good quality research in its best possible form, and to the most relevant and widest possible audience? Or put another way, is research publishing: A service to researchers-as-readers? A service to researchers-as-authors? Standardize processes (particularly around transparency) Integrity first, then (perhaps) impact
- Practically, this means: adopt and encourage new ways of working… pre-registration (including clinical trials registration, but beyond that) https://cos.io/prereg/ https://www.isrctn.com/ The Registered Reports innovation in peer review https://hub.wiley.com/community/exchanges/discover/blog/2018/02/21/registered-reports-try-this-innovation-in-peer-review Other ways for researchers to communicate their research, and to be recognised for it (beyond journal articles, and like preprints and data sharing and trail results registries). “I think much of this comes from my background working with very diverse Earth observation data. I think the observational sciences are culturally quite different from the experimental sciences. And one thing I note is that often a really good data set might have huge scientific and social benefit, yet have few if any papers directly associated with it. For example, a friend of mine just spent the last two years of her life assembling THE benchmark data set of passive microwave remote sensing data over the last 40 years—ensuring consistent calibration and interpolation across many satellites and sensors. These data are used in everything from operational shipping and weather forecasting to global climate assessment and local hydrological prediction. She will get one or two papers from this that will get a few citations and some recognition, but they are secondary and trivial to the impact of the data. She is a soft-funded researcher who is well respected as a top expert with these type of data and associated techniques, but she is essentially unknown to her university and would never be eligible for tenure.” Personal communication from a researcher to Chris Graf (requested permission to use this story) Transparent peer review (unsigned, or signed) http://www.embopress.org/transparent-process#Review_Process Writing workshops and other support for authors who struggle to write in English How do we do this at scale?
- “Every experiment is a conversation with a prior experiment, and every new theory a refutation of the old.” Siddhartha Murkerjee The Emperor of All Maladies: A Biography of CancerThe conversations we have with prior work are biased and Murkerjee shows that brilliantly in this book about the history of the war on cancer. Science was diverted down many rabbit holes by chasing funding, fame, and publication. We really need to reduce barriers to access through innovation so all voices (or instruments to carry forward your analogy) have an equal shot at discovery. As editors and publishers we need to focus first on integrity as you state, then on impact, but this is one of those wicked problems. Funding drives research, sometimes in perverse ways. Academia has its own perverse incentives related to publication. Authors, who are primarily researchers, want to do research and not deal with the time consuming tasks related to publishing. More collaboration such as this conference and COPE's work with institutions are a good start, but more needs to happen.
- It is that collaboration that’s key, because cultural change is hard [15]. This is a road we have to travel together: funders, institutions, researchers, editors, journals and publishers How we peer review and publish research at Wiley is built on the great work of our editors and publishing teams. This work is always a collaboration with researchers and academic communities around the world who write, peer review, and then build upon the work we publish with their own research. Looking wider, funders and institutions shape how we work, when they decide how to fund and reward research and academic work, and when they decide how to support researchers with training and mentorship. When it comes to research integrity and publishing ethics, this is a road we must travel together.