Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Zen and the Art of Research Assessment

432 views

Published on

Talk given by Prof Stephen Curry at the R2R Conference, London, Feb 2017 – a reminder to researchers, in an overly metricised world, to focus on the things that matter in their work.

Conference programme: https://r2rconf.com/programme/

Published in: Science
  • Be the first to comment

  • Be the first to like this

Zen and the Art of Research Assessment

  1. 1. Zen and the Art of Research Assessment Stephen Curry Imperial College R2R Conference | 20 Feb 2017
  2. 2. An Inquiry into Values
  3. 3. Why do people become researchers? To understand the world To change the world To earn a living To be remembered Wikimedia Commons (Dona Eidam/USGS)
  4. 4. How do people become researchers? h"p://www.nature.com/news/science-publishing-the-golden-club-1.13951 To understand the world To change the world To earn a living To be remembered Easier to achieve if publishing is open and evaluaHon more broadly based Publish in a high impact factor journal
  5. 5. Measurement has its uses… h"ps://www.premierleague.com/tables?co=1&se=42&mw=-1&ha=-1 English Premier League Table (2015-16)
  6. 6. …but where are the limits?
  7. 7. The Metric Tide (2015): a call for responsible metrics • Indicators, not metrics? • Numbers inform but cannot replace judgement • Users & providers need to be transparent The Metric Tide Report of the Independent Review of the Role of Metrics in Research Assessment and Management July 2015 h"p://www.ascb.org/dora/ h"p://www.hefce.ac.uk/pubs/rereports/Year/2015/metricMde/Title,104463,en.html
  8. 8. Swimming against the metric Hde: be careful what you wish for “Don’t aim at success […] for success, like happiness, cannot be pursued; it must ensue, and it only does so as the unintended side-effect of one’s dedicaMon to a cause greater than oneself…” Viktor Frankl On your deathbed, will your bibliography or your h-index be uppermost in your mind?
  9. 9. Swimming against the metric Hde: be humble Fickle nature of human judgement… Fallacy of What You See Is All There Is Halo effects Anchoring effects Hindsight bias Loss aversion “I have yet to meet a successful scientist who lacks the ability to exaggerate the importance of what he or she is doing.” Daniel Kahneman
  10. 10. Swimming against the metric Hde: accoun;ng for the unexpected Apollo 8 Mission Transcript (24 Dec 1968) Anders: "Oh, my God, look at that picture over there. There's the Earth coming up. Wow, is that pre`y!" Anders (to Lovell): "You got a color film, Jim? Hand me a roll of color, quick, would you?" Lovell: "Oh, man, that's great! Where is it?" Anders: "Hurry. Quick." Lovell: "Down here?" Anders: "Just grab me a color. A color exterior. Hurry up. Got one?" Lovell: "Yeah, I'm lookin' for one. C368." Anders: "Anything quick." Anders: "I think we missed it." Within seconds, Lovell sees the shot again in another window. He asks for the camera. Anders: "Wait a minute, just let me get the right seing here now, just calm down. Calm down, Lovell!" h"p://www.npr.org/secMons/thetwo-way/2013/12/23/256605845/on- anniversary-of-apollo-8-how-the-earthrise-photo-was-made
  11. 11. Focusing researcher assessment on published outputs is problema;c h"p://journals.plos.org/plosmedicine/arMcle?id=10.1371/journal.pmed.1001747 h"p://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/ h"p://dx.doi.org/10.1016/j.cub.2007.06.014 h"p://www.palgrave-journals.com/arMcles/palcomms2016105h"p://nuffieldbioethics.org/project/research-culture/
  12. 12. Can the openness of our scienHfic heritage help us to do be`er research? Is the amateur (and open) ethos sMll one of the norms of the academy?
  13. 13. Some of the work is technical (and requires collaboraHon…) Can openness help to change behaviour? Cita%on Distribu%ons Royal Society Journals EMBO Journal PLOS PNAS Nature Nature CommunicaHons Nature Chemistry ScienHfic Reports Acta Cryst. (A-F) … No promo;on of JIFs PLOS eLife ASM journals …
  14. 14. Academic leaders and insHtuHons have to do their bit… Vale, R. D. (2012) Mol Biol Cell 23, 3285–3289. h"p://www.nature.com/news/fewer-numbers-be"er-science-1.20858 1. Research, publicaHons, grants 2. Managerial & academic duHes 3. Mentoring & teaching 4. Clinical work (if applicable) 5. Entrepreneurship & community outreach Researcher assessment at UMC Utrecht h"ps://emckiernan.wordpress.com/pledge/
  15. 15. A public good: how open science can be be`er science Preprints: faster communicaHon Focus on the content, not the container Largest possible audience (sharing & scruHny) Data sharing: re-use & scruHny benefits Be`er for changing the world
  16. 16. A public good: how open science can be more inclusive CiHzen science teaches researchers about: new (non-tradiHonal) audiences & scienHsts CommunicaHon + ParHcipaHon = Public Trust
  17. 17. PoliHcal realiHes: we have to be open with the public “People in this country have had enough of experts.” Michael Gove, MP “…we have to do more to show how, and why, universiHes are one of society’s great achievements and best hopes.” Chris Husbands (VC, Sheffield-Hallam University)
  18. 18. Thank you AlternaHve Metrics? CitaHon counts are easy and not uninteresHng, but… How reliable is your research? How easy is it for other researcher to access your data? How many lives has your work saved? How many people has your research helped to heal? How many people have been lived from poverty through your published output? How much economic growth has your work generated? How much has your research addressed significant naHonal and internaHonal challenges? How much has your research reduced inequality in the world? How many young minds has your work inspired? How well has your mentoring helped to nurture the next generaHon of researchers?

×