OpenAIRE-COAR conference 2014: Next generation research evaluation: the ACUMEN Portfolio and web based information tools, by Clifford Tatum - Leiden University
Upcoming SlideShare
Loading in...5
×
 

OpenAIRE-COAR conference 2014: Next generation research evaluation: the ACUMEN Portfolio and web based information tools, by Clifford Tatum - Leiden University

on

  • 164 views

...


Presentation at the OpenAIRE-COAR Conference: "Open Access Movement to Reality: Putting the Pieces Together", Athens - May 21-22, 2014.
Session 4: The impact of openness and how to evaluate research.
Next generation research evaluation: the ACUMEN Portfolio and web based information tools, by Clifford Tatum - Centre for Science and Technology Studies (CWTS) Leiden University

Statistics

Views

Total Views
164
Views on SlideShare
164
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

 OpenAIRE-COAR conference 2014: Next generation research evaluation: the ACUMEN Portfolio and web based information tools, by Clifford Tatum - Leiden University OpenAIRE-COAR conference 2014: Next generation research evaluation: the ACUMEN Portfolio and web based information tools, by Clifford Tatum - Leiden University Presentation Transcript

  • Next generation research evaluation: ! ! ! ! ! ! ! ! ! ! ! the ACUMEN Portfolio and web based information tools Clifford Tatum and Paul Wouters Centre for Science and Technology Studies Leiden University, The Netherlands http://dx.doi.org/10.6084/m9.figshare.1033681
 OpenAIRE - Athens, Greece 22 May 2014
  • 1. ACUMEN portfolio 2. Implementation plan 3. Towards portability Presentation Outline Tatum & Wouters | 22 May 20142
  • Academic Careers Understood through Measurements and Norms ! European 7th Framework collaborative project Capacities, Science in Society 2010 Grant Agreement: 266632 ! 9 institutional partners, in 7 countries website: research-acumen.eu 3
  • Evaluation Machines [1] [1] Dahler-Larsen (2011) The Evaluation Society. Stanford University Press. ➡ researchers cannot opt out of evaluation ➡ decreased openness to question evaluation practices ➡ institutional support for evaluation is often stronger than evidence about its use ➡ pressure to do evaluation is often stronger than the pressure to do good evaluation Tatum & Wouters | 22 May 20144
  • Research Evaluation Gap ➡ discrepancy between evaluation criteria and the social and economic functions of science ➡ evaluation methods (esp. qualitative) have not adapted to increased scale of research ➡ available quantitative measures are often not applicable at the individual level ➡ lack of recognition for new types of work that researchers need to perform Tatum & Wouters | 22 May 20145
  • ACUMEN research aimed at empowering researchers (and evaluators) comparative analysis of peer review systems in Europe assessment of scientometric indicators in performance evaluation analysis of gender dimension in researcher evaluation assessment of webometric (and altmetric) indicators ethnographic study of important evaluation events Tatum & Wouters | 22 May 20146
  • ACUMEN research comparative analysis of peer review systems in Europe assessment of scientometric indicators in performance evaluation analysis of gender dimension in researcher evaluation Common Data Strategy assessment of webometric (and altmetric) indicators Bulgaria Czech Republic Denmark Estonia Finland France Germany Hungary Israel Italy Netherlands Poland Slovenia Spain United Kingdom 4 Academic Disciplines (a) astronomy and astrophysics (b) public and occupational health (c) environmental engineering (d) philosophy (including history and philosophy of science) ethnographic study of important evaluation events 15 European countries Tatum & Wouters | 22 May 20147
  • ACUMEN Portfolio Career  Narrative
 Links  expertise,  output,  and  influence  together  in  an   evidence-­‐based  argument;  included  content  is  negotiated   with  evaluator  and  tailored  to  the  particular  evaluation Output  ! -­‐  publications   -­‐  public  media   -­‐  teaching   -­‐  web/social  media   -­‐  data  sets   -­‐  software/tools   -­‐  infrastructure   -­‐  grant  proposals Expertise  
 -­‐  scientific/scholarly   -­‐  technological   -­‐  communication   -­‐  organizational   -­‐  knowledge  transfer   -­‐  educational Influence
 
 -­‐  on  science   ! -­‐  on  society   ! -­‐  on  economy   ! -­‐  on  teaching Tatum & Wouters | 22 May 20148
  • ACUMEN Portfolio Career  Narrative
 Links  expertise,  output,  and  influence  together  in  an   evidence-­‐based  argument;  included  content  is  negotiated   with  evaluator  and  tailored  to  the  particular  evaluation Output  ! -­‐  publications   -­‐  public  media   -­‐  teaching   -­‐  web/social  media   -­‐  data  sets   -­‐  software/tools   -­‐  infrastructure   -­‐  grant  proposals Expertise  
 -­‐  scientific/scholarly   -­‐  technological   -­‐  communication   -­‐  organizational   -­‐  knowledge  transfer   -­‐  educational Influence
 
 -­‐  on  science   ! -­‐  on  society   ! -­‐  on  economy   ! -­‐  on  teaching Evaluation  Guidelines  
 -­‐ aimed  at  both  researchers  and  evaluators   -­‐ development  of  evidence  based  arguments   (what  counts  as  evidence?)   -­‐ expanded  list  of  research  output   -­‐ establishing  provenance   -­‐ taxonomy  of  indicators:  bibliometric,   webometric,  altmetric   -­‐ guidance  on  use  of  indicators   -­‐ contextual  considerations,  such  as:  stage  of   career,  discipline,  and  country  of  residence Tatum & Wouters | 22 May 20149
  • Aims: Portfolio & Guidelines ➡ Instrument for empowering researchers in the processes of evaluation ! ➡ Taking into consideration all academic disciplines ! ➡ Suitable for other uses (e.g. career planning) ! ➡ Able to integrate into different evaluation systems Tatum & Wouters | 22 May 201410
  • Implementation Plan Tatum & Wouters | 22 May 201411
  • CWTS (lead partner) research - EU eval policies - national policies - indicator standards - link portfolio to DBs int’l coordination - international policies - research management - standards & practices implementation - CERIF - CASRAI - ORCID - others? translate portfolio concept to CERIF/CASRAI euroCRIS ARMA Tatum & Wouters | 22 May 2014 ACUMEN-2 12
  • Policy Alignment (ACUMEN 2) EU research evaluation protocol France eval policy UK RCUK/REF Netherlands NWO/SEP Norway eval policy Denmark eval policy Spain eval policy Finland eval policy Greece eval policy … National & EU Research Evaluation Policies common evaluation elements standard model on the basis of commonality ACUMEN Portfolio portfolio expression(s) in CERIF Tatum & Wouters | 22 May 201413
  • CERIF - interchange solution (source: Ed Simons, euroCRIS)14
  • Potential Obstacles ➡ increased quantity of evaluations and added pressure on evaluators ➡ researcher information often locked in institutional CRIS systems ➡ validation workflows can create a bottleneck (and validation not portable) ➡ provenance of research objects, especially with regard to non-standard output ➡ profile fatigue: institutional website/CRIS, grant applications, ResearchGate, Academia.edu, LinkedIn, etc. Tatum & Wouters | 22 May 201415
  • Profile Portability Tatum & Wouters | 22 May 2014 Interoperability among research object metadata benefits from persistent IDs that are both open and accessible across systems ! …this is possible today 16
  • Tatum & Wouters | 22 May 2014 Open ID Stack Author ID DOI - article - dataset - software - other - OA version Organization Funding source / / Proposal 17
  • Tatum & Wouters | 22 May 2014 / / Open ID Stack export to CERIF/XML for use elsewhere CRIS systems open to use by Web services 18
  • thank you! email: c.c.tatum@cwts.leidenuniv.nl website: research-acumen.eu Tatum & Wouters | 22 May 2014 http://orcid.org/0000-0002-2212-3197 19