A brief overview over the state of affairs in my art of the ACUMEN project. The slide on p. 7 has been my core slide during the project. If you are interested in the outcomes of the project, then download the deliverable here: http://research-acumen.eu/wp-content/uploads/D1.11-Impact%20of%20evaluations%20on%20academic%20careers_v1.0.pdf

1. The impact of evaluations on
developments in academic
careers
Frank van der Most
DANS and the e-Humanities Group
!
ACUMEN meeting, 9 - 11 January 2013
1

2. ACUMEN
Academic Careers Understood through
Measurement and Norms
www.research-acumen.eu
!
“Currently, there is a discrepancy between the criteria
used in performance assessment and the broader
social and economic function of scientific and
scholarly research.” (proposal, p. 3)
!2

3. ACUMEN
✦ “... broader social functions of the scientific system
are often not included in its quality control
mechanisms.” (proposal, p. 3)
✦ so, ACUMEN will develop
•a portfolio for researchers, and
•criteria and guidelines for Good Evaluation
Practices
!3

4. Research Question
✦ How do evaluations influence the development of
individuals’ careers in academia?
(What is the impact of evaluations?)
•Use perspective for ACUMEN
✦ Concepts
•Wide notion of evaluations
•Careers + ‘work’
!4

5. Project partner
Country
People
Universiteit Leiden
Netherlands
Paul Wouters, Clifford Tatum, Laura de Ruiter
Bar Ilan University
Israel
Judit Bar-Ilan, Hadas Shema
Consejo Superior de Investigaciones
Cientificas (CSIC)
Spain
Isidro F. Aguillo, Mario Fernández Pérez,
Amalia Mas Bleda
University of Wolverhampton
United
Kingdom
Mike Thelwall, Kayvan Kousha
Sihtasutus Archimedes
Estonia
Ülle Must, Arto Mustajoki, Marcus Denton,
Taavi Tiirik, Oskar Otsus
Humboldt-Universität zu Berlin
Germany
Stefan Gradmann, Frank Havemann
Technische Hochschule Wildau
Germany
Rainer Voss, Hildrun Kretschmer, Ulla Große,
Martina Brandt, Frank Hartman, Dana Mietzner
Danmarks Biblioteksskole
Denmark
Birger Larsen, Jesper W. Schneider,
Lorna Wildgaard
Koninklijke Nederlandse Akademie van
Wetenschappen (KNAW)
Netherlands
Andrea Scharnhorst, Sally Wyatt,
Frank van der Most

6. Who does what?
Work Package
Who
1
Evaluation impact
KNAW, Archimedes
2
Institutional Web Presence
University of Wolverhampton, CSIC
3
Researchers’ Web image
Bar Ilan University
4
Gender effects of evaluation
Technische Hochschule Wildau
5
New bibliometric indicators
Humboldt-Universität ,Danmarks
Biblioteksskole
6
ACUMEN portfolio
All
7
Management
Universiteit Leiden
!6

7. Impact
A change
in a career
Evaluatio
F.V. van der Most, 2012
!7

8. Impact
Event A
Event U
A change
in a career
Evaluatio
nt D
F.V. van der Most, 2012
Event B
Event V
Event W
Micro story
Event C
!7

9. Method
✦ Quite structured, in depth interviews
✦ 4 Countries : UK, Germany, the Netherlands,
Poland
✦ 4 Disciplines : Astronomy/astrophysics,
Environmental engineering, Philosophy (+), Public
Health
✦ 3 levels of seniority: PhD + 5 to 10 years, + 11 to
25 years, + 26 years or more (or equivalents)
!8

10. Method: interview protocol
✦ What are the three most important developments
in your career?
✦ What are the three most influential evaluations in
your career?
!9

11. Work in progress
Number of
interviews done
AA
EE
P+
PH
Total
UK
4 (3+1)
4 (3+1)
3 (2+1)
4 (3+1)
15 (11+4)
Germany
2 (2+0)
3 (3+0)
3 (2+1)
3 (3+0)
11 (10+1)
The Netherlands
(ongoing)
1 (1+0)
2 (1+1)
1 (1+0)
3 (2+1)
7 (5+2)
Poland
(feb 2013)
-
-
-
-
Numbers in brackets: individuals + deans/HR managers
!10

12. Indicator, what is the overlap?
Don’t know exactly, yet.
!11

13. Basic answers:
3 most influential evaluations
Coding group
Total
Job applications
21
Grants
14
Job-related evaluations (not appl.s)
11
Diplomas
6
Other
5
Paper reviews
5
Informal evaluations (other)
3
Evaluations leading to invitations
3
Feedback
3
Evaluation of teaching
1
Total
72
!12

14. Basic answers:
3 most influential evaluations
Coding group
Total
!13

15. Basic answers:
3 most influential evaluations
Coding group
Total
Job applications
16
!13

16. Basic answers:
3 most influential evaluations
Coding group
Total
Job applications
16
Grants
14
!13

17. Basic answers:
3 most influential evaluations
Coding group
Total
Job applications
16
Grants
14
Informal evaluations
12
!13

18. Basic answers:
3 most influential evaluations
Coding group
Total
Job applications
16
Grants
14
Informal evaluations
12
Diplomas
6
Other
5
Paper review
4
Job-related evaluations (not appl.s)
3
Evaluations leading to invitations
1
Evaluation of teaching
1
Total
62
!13

19. Examples: informal job applications
!14

20. Examples: informal job applications
✦ A MA student’s supervisor offers a job on
a project leading to a PhD (DE AA, DE PH)
!14

21. Examples: informal job applications
✦ A MA student’s supervisor offers a job on
a project leading to a PhD (DE AA, DE PH)
✦ An informal conversation leads from a
temporary to a permanent position as lab
manager (UK EE)
!14

22. Examples: informal job applications
✦ A MA student’s supervisor offers a job on
a project leading to a PhD (DE AA, DE PH)
✦ An informal conversation leads from a
temporary to a permanent position as lab
manager (UK EE)
✦ An earlier conversation later on appears to
be an important element in a job
application (UK AA)
!14

23. Examples: evaluations leading
to invitations
!15

24. Examples: evaluations leading
to invitations
✦ A HoD asks the interviewee to lead a
project, a great opportunity for a relatively
junior researcher (UK PH)
!15

25. Examples: evaluations leading
to invitations
✦ A HoD asks the interviewee to lead a
project, a great opportunity for a relatively
junior researcher (UK PH)
✦ An editor asks the interviewee to become
the successor (UK PH)
!15

26. Examples: other
✦ Course applications (BA to PhD) (2)
✦ ‘Still to be determined’ (1)
✦ Winning a prize (1)
✦ Examinations during a study (1)
!16

27. Relevance for governance of science
✦ Evaluations seem to influence careers (but further
analysis needed)
✦ Share of informal evaluations may be more influential
than expected
•The ‘system’ is not so ‘systematic’ or
not so transparent
•Sometimes systematically opaque (Academic
freedom)
!17

29. Thank you for your attention!
!
!
Questions, suggestions?
!
!
!
www.researchaffairs.nl