The impact of evaluations on developments in academic careers
Upcoming SlideShare
Loading in...5
×
 

The impact of evaluations on developments in academic careers

on

  • 91 views

A brief overview over the state of affairs in my art of the ACUMEN project. The slide on p. 7 has been my core slide during the project. If you are interested in the outcomes of the project, then ...

A brief overview over the state of affairs in my art of the ACUMEN project. The slide on p. 7 has been my core slide during the project. If you are interested in the outcomes of the project, then download the deliverable here: http://research-acumen.eu/wp-content/uploads/D1.11-Impact%20of%20evaluations%20on%20academic%20careers_v1.0.pdf

Statistics

Views

Total Views
91
Slideshare-icon Views on SlideShare
91
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    The impact of evaluations on developments in academic careers The impact of evaluations on developments in academic careers Presentation Transcript

    • The impact of evaluations on developments in academic careers Frank van der Most DANS and the e-Humanities Group ! ACUMEN meeting, 9 - 11 January 2013 1
    • ACUMEN Academic Careers Understood through
 Measurement and Norms www.research-acumen.eu ! “Currently, there is a discrepancy between the criteria used in performance assessment and the broader social and economic function of scientific and scholarly research.” (proposal, p. 3) !2
    • ACUMEN ✦ “... broader social functions of the scientific system are often not included in its quality control mechanisms.” (proposal, p. 3) ✦ so, ACUMEN will develop •a portfolio for researchers, and •criteria and guidelines for Good Evaluation Practices !3
    • Research Question ✦ How do evaluations influence the development of individuals’ careers in academia?
 (What is the impact of evaluations?) •Use perspective for ACUMEN ✦ Concepts •Wide notion of evaluations •Careers + ‘work’ !4
    • Project partner Country People Universiteit Leiden Netherlands Paul Wouters, Clifford Tatum, Laura de Ruiter Bar Ilan University Israel Judit Bar-Ilan, Hadas Shema Consejo Superior de Investigaciones Cientificas (CSIC) Spain Isidro F. Aguillo, Mario Fernández Pérez,
 Amalia Mas Bleda University of Wolverhampton United Kingdom Mike Thelwall, Kayvan Kousha Sihtasutus Archimedes Estonia Ülle Must, Arto Mustajoki, Marcus Denton,
 Taavi Tiirik, Oskar Otsus Humboldt-Universität zu Berlin Germany Stefan Gradmann, Frank Havemann Technische Hochschule Wildau Germany Rainer Voss, Hildrun Kretschmer, Ulla Große, Martina Brandt, Frank Hartman, Dana Mietzner Danmarks Biblioteksskole Denmark Birger Larsen, Jesper W. Schneider,
 Lorna Wildgaard Koninklijke Nederlandse Akademie van Wetenschappen (KNAW) Netherlands Andrea Scharnhorst, Sally Wyatt,
 Frank van der Most
    • Who does what? Work Package Who 1 Evaluation impact KNAW, Archimedes 2 Institutional Web Presence University of Wolverhampton, CSIC 3 Researchers’ Web image Bar Ilan University 4 Gender effects of evaluation Technische Hochschule Wildau 5 New bibliometric indicators Humboldt-Universität ,Danmarks Biblioteksskole 6 ACUMEN portfolio All 7 Management Universiteit Leiden !6
    • Impact A change in a career Evaluatio F.V. van der Most, 2012 !7
    • Impact Event A Event U A change in a career Evaluatio nt D F.V. van der Most, 2012 Event B Event V Event W Micro story Event C !7
    • Method ✦ Quite structured, in depth interviews ✦ 4 Countries : UK, Germany, the Netherlands, Poland ✦ 4 Disciplines : Astronomy/astrophysics, Environmental engineering, Philosophy (+), Public Health ✦ 3 levels of seniority: PhD + 5 to 10 years, + 11 to 25 years, + 26 years or more (or equivalents) !8
    • Method: interview protocol ✦ What are the three most important developments in your career? ✦ What are the three most influential evaluations in your career? !9
    • Work in progress Number of interviews done AA EE P+ PH Total UK 4 (3+1) 4 (3+1) 3 (2+1) 4 (3+1) 15 (11+4) Germany 2 (2+0) 3 (3+0) 3 (2+1) 3 (3+0) 11 (10+1) The Netherlands (ongoing) 1 (1+0) 2 (1+1) 1 (1+0) 3 (2+1) 7 (5+2) Poland
 (feb 2013) - - - - Numbers in brackets: individuals + deans/HR managers !10
    • Indicator, what is the overlap? Don’t know exactly, yet. !11
    • Basic answers: 3 most influential evaluations Coding group Total Job applications 21 Grants 14 Job-related evaluations (not appl.s) 11 Diplomas 6 Other 5 Paper reviews 5 Informal evaluations (other) 3 Evaluations leading to invitations 3 Feedback 3 Evaluation of teaching 1 Total 72 !12
    • Basic answers: 3 most influential evaluations Coding group Total !13
    • Basic answers: 3 most influential evaluations Coding group Total Job applications 16 !13
    • Basic answers: 3 most influential evaluations Coding group Total Job applications 16 Grants 14 !13
    • Basic answers: 3 most influential evaluations Coding group Total Job applications 16 Grants 14 Informal evaluations 12 !13
    • Basic answers: 3 most influential evaluations Coding group Total Job applications 16 Grants 14 Informal evaluations 12 Diplomas 6 Other 5 Paper review 4 Job-related evaluations (not appl.s) 3 Evaluations leading to invitations 1 Evaluation of teaching 1 Total 62 !13
    • Examples: informal job applications !14
    • Examples: informal job applications ✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH) !14
    • Examples: informal job applications ✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH) ✦ An informal conversation leads from a temporary to a permanent position as lab manager (UK EE) !14
    • Examples: informal job applications ✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH) ✦ An informal conversation leads from a temporary to a permanent position as lab manager (UK EE) ✦ An earlier conversation later on appears to be an important element in a job application (UK AA) !14
    • Examples: evaluations leading to invitations !15
    • Examples: evaluations leading to invitations ✦ A HoD asks the interviewee to lead a project, a great opportunity for a relatively junior researcher (UK PH) !15
    • Examples: evaluations leading to invitations ✦ A HoD asks the interviewee to lead a project, a great opportunity for a relatively junior researcher (UK PH) ✦ An editor asks the interviewee to become the successor (UK PH) !15
    • Examples: other ✦ Course applications (BA to PhD) (2) ✦ ‘Still to be determined’ (1) ✦ Winning a prize (1) ✦ Examinations during a study (1) !16
    • Relevance for governance of science ✦ Evaluations seem to influence careers (but further analysis needed) ✦ Share of informal evaluations may be more influential than expected •The ‘system’ is not so ‘systematic’ or
 not so transparent •Sometimes systematically opaque (Academic freedom) !17
    • Thank you for your attention! ! ! Questions, suggestions? ! ! ! www.researchaffairs.nl