A presentation building on my work on the ACUMEN project (2011-2014). 'What is important for your academic work but not evaluated at all?' I asked my interviewees. Their answer lead me to identify a few ways through which work or aspects of work are being made invisible.

1. The blind spots
of evaluations
in academic work
Frank van der Most
!
Forskningsindikatornetværket meeting
9 May 2014, Det Informationsvidenskabelige
Akademi, Københavns Universitet
1

2. Invisible work
“What will count as work does not depend a
priori on any set of indicators, but rather on
the definition of the situation.”
(Star & Strauss, 1999, p. 14)
How does that work?
What can one do about it?
2

3. Overview
✦ Introduction to ACUMEN
✦ Introduction to my sub-project
✦ The concept of invisible work
✦ Ways of making work invisible, based on
interview
✦ How to make blind spots visible again?
3

4. Frank van der Most
✦ 1988 - 1997 Computer science and Science &
Technology studies
✦ 1997 - 2004 Research in history of technology,
evaluation of Norwegian Research Council, and more
✦ 2002 - 2003 One-man company; software for research
✦ 2004 - 2009 PhD; research councils; STePS, Twente
✦ 2009 - 2014 Postdoc places; research evaluation;
CIRCLE, Lund University & e-Humanities group + DANS,
KNAW
✦ Current interest : Visualization of academic careers
www.frankvandermost.nl

5. ACUMEN
Academic Careers Understood through
Measurement and Norms
!
“Currently, there is a discrepancy between the criteria
used in performance assessment and the broader
social and economic function of scientific and
scholarly research.” (proposal, p. 3)
5

6. ACUMEN
Additional problems
✦ “lack of resources for qualitative evaluation
due to increased scale of research”
✦ “quantitative measures are often not
applicable at the individual level”
✦ “lack of recognition for new types of work”
(ACUMEN description of work, p. 3)
6

7. ACUMEN
7
✦ So, ACUMEN will develop
•a portfolio for researchers, and
•criteria and guidelines for Good
Evaluation Practices
✦ Done … see
www.research-acumen.eu

8. My research question
✦ How do evaluations influence the development of
individuals’ careers in academia?
(What is the impact of evaluations?)
•Wide notion of evaluations
•Careers + ‘work’
8

9. Method
✦ Structured, in depth interviews
✦ 4 Countries : UK, Germany, the Netherlands,
Poland
✦ 4 Disciplines : Astronomy/astrophysics,
Environmental engineering, Philosophy (+), Public
Health
✦ 3 levels of seniority: PhD + 5 to 10 years, + 11 to
25 years, + 26 years or more (or equivalents)
9

10. Invisible work
“What will count as work does not depend a
priori on any set of indicators, but rather on
the definition of the situation.”
(Star & Strauss, 1999, p. 14)
How does that work?
What can one do about it
10

11. Importance of invisible work
according to the interviewees
✦ It costs time, energy and other resources
✦ Sine qua non for research
✦ Quality
✦ ‘What makes science tick’
11

12. Question: what is invisible in
evaluations?
✦ Two ways to collect answers in the interviews
•While discussing evaluations that interviewees
brought up
•Simply ask the interviewee
12

13. Categories of invisible work
1a Care work > colleagues Collegiality / cooperative attitude
1b Care work > tools Technical/methodological support
1c Care work > output -
1d Care work > organization Administrative work
Management skills
Teaching
1e Care work > self Other
1f Care work > discipline Administrative work
Refereeing work
2 Articulation work Collegiality / cooperative attitude
Leadership skills
Management skills
Other
3a Persuasion work > credibility work Being an academic/intellectual
Societal impact or relevance
3b Persuasion work > reputation -
3c Persuasion work > position work Other
Based on Antonijevic, S., Dormans, S., & Wyatt, S. (2013)
and expanded based on interviews

14. Invisible-work category Answer Gender
F M Total
1a Colleagues Collegiality / cooperative attitude 1 3 4
1b Tools Technical/methodological support 1 2 3
1c Output 0 0 0
1d Organization Administrative work 4 1 5
Management skills 1 1 2
Teaching 2 4 6
1e Self Other 1 1 2
1f Discipline Administrative work 1 0 1
Refereeing work 3 2 5
Collegiality / cooperative attitude 1 1 2
Leadership skills 1 0 1
Management skills 2 1 3
Other 0 1 1
Credibility Being an academic/intellectual 0 2 2
Societal impact or relevance 3 2 5
Reputation 0 0 0
Position Other 0 1 1
Nothing important is not evaluated 0 2 2
Other 3 1 4
Question not asked or answered 1 3 4
Total 25 28 53
Carework
Articulation
work
Persuasion
work

15. Enforcing unrealistic categories:
research vs support
“So only who write paper, produces paper, and by paper
I mean refereed publications, is considered. And the
others are doing a different job. But, but as far as I know,
in the case of [my institute] for instance, the people that
are doing other jobs, technical jobs, are the ones that
bring the money in. So, and allow the others to do the
research. But then they don't get any credit for
that.” (Interviewee 12, 01:40:04.98)
15

16. Enforcing unrealistic categories:
not including all the work
“It's a contribution to the broader academic good if
you like. Academia couldn't go on if if people didn't
agree to peer review. Didn't take the sort of unpaid
extra roles on in terms of editing, working with
journal editors.” (Interviewee 11, 01:06:52.66)
16

17. Location-centrism
"... you are evaluated but then the basis for the
evaluation is sort of, yeah, it's, it's what you
wrote. It's fit to a traditional career, and if you
don't fit this exactly, it's sort of you fall between
the cracks or something ... which is in Europe
where we all believed it, ... the Bologna process
and everything is transparent and we work
together and blablabla" (Interviewee 4,
00:49:28.98)
17

18. Life-course fallacy
" When I look at my career and what has
happened at the universities during these years
is that when I started and when I look at the
generation before me, the criteria that they
had to ... comply to, were very different than
the criteria that were put on me." (Interviewee
2, 00:54:45.86)
18

19. Life-course fallacy
✦ "erroneously interpreting cross-sectional age
differences as if they referred to the process of
aging." (Riley 1998, p. 42)
19

20. 20
Now-10-20-30-40-50
Grosscareerage
10
20
30
40
50
Figure based on Riley 1998
Cohorts
F.V. van der Most, 2014

21. 21 Figure based on Riley 1998
Now-10-20-30-40-50
Grosscareerage
10
20
30
40
50
Life-course fallacy
F.V. van der Most, 2014

22. Life-course fallacy
✦ "erroneously interpreting cross-sectional age
differences as if they referred to the process of
aging." (Riley 1998, p. 42)
✦ When …
•comparing applicants of different cohorts
•averaging scores of all authors to calculate mean
indicator scores
•calculating averages per cohort and suggesting
career development (indicator studies + surveys)
22

23. Fallacy of cohort-centrism
✦ "erroneously assuming that members of all cohorts
will grow older in the same fashion as members of
our own cohort" (Riley 1998, p. 43)
23

24. 24
Now-10-20-30-40-50
Grosscareerage
10
20
30
40
50
Figure based on Riley 1998
Fallacy of cohort-centrism
F.V. van der Most, 2014

25. Fallacy of cohort-centrism
✦ "erroneously assuming that members of all cohorts
will grow older in the same fashion as members of
our own cohort" (Riley 1998, p. 43)
✦ When …
•comparing a younger researcher now with
oneself in that same stage x years ago
•applying criteria that describe success in one
cohort to another.
25

26. The criteria change"
while careers progress
"And the criteria that I had been confronted
with, have changed over time ... It's like
playing a game and during the game they are
changing the rules." (Interviewee 2,
00:54:45.86)
26

27. Making work invisible
✦ Un-realistic categories:
•research vs support
•not including all the work
✦ Location centrism
✦ Life-course fallacy
✦ Fallacy of cohort centrism
✦ Changing criteria too fast / during the career
✦ …
27

28. Making the blind spots
visible again
✦ Awareness of how work is made invisible
✦ Expanding or expandable categories
✦ Historical and geographical awareness
✦ Practical solutions - not always imaginable
✦ ACUMEN’s solution through a Portfolio
28

29. ACUMENPortfolio
structure
29
Expertise
- scientific /
scholarly
- technological
- communication
- organizational
- knowledge
transfer
- educational
Influence
!
- on science
- on society
- on economy
- on teaching
Your
narrative
!
!
Links your expertise, output
and influence together in a
narrative that presents your
self-perspective to your
evaluator
Thank you, Clifford Tatum for
inspiration for the figure
!
!
Output
- scholarly
- to public
- teaching
- web/social
media
!
- data sets
- software/
tools
- IPR etc.
- grant
proposals
F.V. van der Most, 2014

30. ACUMEN Portfolio
as a solution
✦ Allow a broad
presentation of the
evaluated and their
work
•‘Other category’
•Self-selection of
categories
✦ Narrative
✦ Take ‘academic age’
into account
Drawbacks
✦ Still time-consuming
✦ Information (over)load
✦ Historical development
still invisible
30

31. 31
A prospect for career visualization?

32. To all interviewees:"
!
Thank you very much!"
!
It was an honour and a pleasure!

33. 33

34. Thank you
for your attention!
Questions, comments?

35. References
ACUMEN. (2010). Academic Careers Understood through Measurement and Norms. Annex I
- ''Description of work''
ACUMEN (2014) Guidelines for Good Evaluation Practice with the ACUMEN Portfolio. Leiden
etc. : ACUMEN consortium. pdf
Antonijevic, S., Dormans, S., & Wyatt, S. (2013). Working in virtual knowledge: affective
labor in scholoraly collaboration. In P. Wouters, A. Beaulieu, A. Scharnhorst & S.
Wyatt (Eds.), Virtual knowledge. Experimenting in the humanities and the social
sciences (pp. 57 - 88). Cambridge (MA), London: MIT Press.
Bowker, G.C. & Star, S.L. (1999) Sorting things out. Classification and its consequences.
Cambridge (MA), London : The MIT Press
Riley, M. W. (1998). A life course approach. In J. Z. Giele & G. H. Elder Jr. (Eds.), Methods
of life course research (pp. 28 - 51). Thousand Oaks, London, New Delhi: SAGE
Publications.
Star, S.L. (1991) The sociology of the invisible: The primacy of work in the writings of Anselm
Strauss. In Daive R. Maines (Ed.) Social organization and social process. Essays in
honor of Anselm Strauss. New York: Aldine de Gruyter
Star, S. L., & Straus, A. (1999). Layers of silence, arenas of voice: The ecology of visible and
invisible work. In Computer Supported Cooperative Work, vol. 8, 9-30.
Strauss, A. (1985). Work and the division of labor. In Sociological Quarterly, 26(1), 1-19.
Van der Most, F. (2013) The impact of evaluations on academic careers : An study of
academics in four disciplines, four countries and three cohorts. ACUMEN Deliverable