This presentation gives an overview of the answers that my interviewees during the ACUMEN project gave on questions about their data deposition practices. It was my 'farewell' presentation at DANS.

1. Depositing and evaluating
datasets: een sketch of four
disciplines in four countries
Frank van der Most
eHumanities and DANS
!
DANS lunch-presentation, 4 March 2014
1

2. ACUMEN
Academic Careers Understood through
Measurement and Norms
www.research-acumen.eu
!
“Currently, there is a discrepancy between the criteria
used in performance assessment and the broader
social and economic function of scientific and
scholarly research.” (proposal, p. 3)
!2

3. ACUMEN
Additional problems
✦ “lack of resources for qualitative evaluation
due to increased scale of research”
✦ “quantitative measures are often not
applicable at the individual level”
✦ “lack of recognition for new types of work”
(ACUMEN description of work, p. 3)
!3

4. ACUMEN
!
✦ so, ACUMEN will develop
•a portfolio for researchers, and
•criteria and guidelines for Good
Evaluation Practices
✦ Done … see the website
!4

5. My part in ACUMEN
!5

6. Research Question
✦ How do evaluations influence the development of
individuals’ careers in academia?
(What is the impact of evaluations?)
•Wide notion of evaluations
•Careers + ‘work’
!6

7. Method
✦ Quite structured, in depth interviews
✦ 4 Countries : UK, Germany, the Netherlands, Poland
✦ 4 Disciplines : Astronomy/astrophysics, Environmental
engineering, Philosophy (+), Public Health
✦ Individual academics
•3 levels of seniority: PhD + 5 to 10 years, + 11 to 25
years, + 26 years or more (or equivalents)
✦ Deans, HoDs, or HR manager
!7

8. Method
Number of
interviews done
AA
EE
P+
PH
Total
UK
4 (3+1)
4 (3+1)
3 (2+1)
4 (3+1)
15 (11+4)
Germany
2 (2+0)
3 (3+0)
3 (2+1)
3 (3+0)
11 (10+1)
The Netherlands
3 (3+0)
4 (3+1)
3 (2+1)
4 (3+1)
14 (11+3)
Poland
2 (1+1)
1 (1+0)
4 (3+1)
1 (1+0)
8 (6+2)
Total
48
11 (9+2) 12 (10+2) 13 (9+4) 12 (10+2)
(38+10)
Numbers in brackets: individuals + deans/HR managers
!8

9. Questions relevant for DANS"
Academic individuals
✦ Have you ever deposited your research data, or
have you ever reviewed a data set?
•Could you describe the occasions in a few
words?
✦ In which evaluations is data deposition, creation,
maintenance, or review addressed?
!9

10. AI: Have you ever deposited your
research data?
Answer (coded) Discipline
AA
EE
P+
PH
Total
No
1
2
1
4
8
No, because we do not produce data
1
1
5
1
8
Yes, it is done automatically by the facilities
5
!10
5

11. European Space Agency
Example of European Southern Observatory archives
ESO / B. Tafreshi (twanight.org)
http://www.eso.org/public/archives/images/screen/potw1239a.jpg
Accessed 4 March 2014
!11

12. http://archive.eso.org/eso/eso_archive_main.html
Accessed 4 March 2014
!12

13. http://archive.eso.org/eso/eso_archive_main.html
Accessed 4 March 2014
!13

14. http://archive.eso.org/eso/eso_archive_main.html
Accessed 4 March 2014
!14

15. AI: Have you ever deposited your
research data?
Answer (coded) Discipline
AA
EE
P+
PH
Total
No
1
2
1
4
8
No, because we do not produce data
1
1
5
1
8
Yes, it is done automatically by the facilities
5
5
Question not asked
1
No, but I store privately
1
No, but I know that we should
2
No in general, but Yes, one time
1
2
2
3
3
1
3
Yes, we produce catalogues
3
2
No, but financing parties claim the data
2
2
1
1
Yes, some data is deposited at a central
place in the group
1
1
Yes, in the library of the university
1
1
10
38
Total
9
!15
10
9

16. AI: have you ever reviewed a
dataset?
✦ No, except …
✦ In AA: implicit reviewing when re-using a
dataset
•‘Reduction’ of data / processing of data
•Need to know about the instrument and
trust the data
✦ Exception of one specific project (EE)
!16

17. AI: In which evaluations is data
deposition, creation, maintenance, or
review addressed?
Answer (coded) Discipline
Do not know of any
AA
3
EE
P+
PH
Total
8
8
24
2
Question not asked
5
1
2
5
The process is evaluated
2
2
Incidentally, when data is re-processed
2
2
Through peer review of publications
1
1
2
No, it should be done but probably is not
1
1
Through peer review of claims to discoveries
1
1
Yes, through pipeline processing
1
Total
9
1
10
9
10
38

18. AI: In which evaluations is data
deposition, creation, maintenance, or
review addressed?
Answer (coded) Discipline
Do not know of any
AA
3
EE
P+
PH
Total
8
8
24
2
Question not asked
5
1
2
5
The process is evaluated
2
2
Incidentally, when data is re-processed
2
2
Through peer review of publications
1
1
2
No, it should be done but probably is not
1
1
Through peer review of claims to discoveries
1
1
Yes, through pipeline processing
1
Total
9
1
10
9
10
38

19. AI: In which evaluations is data
deposition, creation, maintenance, or
review addressed?
Answer (coded) Discipline
Do not know of any
AA
3
EE
P+
PH
Total
8
8
24
2
Question not asked
5
1
2
5
The process is evaluated
2
2
Incidentally, when data is re-processed
2
2
Through peer review of publications
1
1
2
No, it should be done but probably is not
1
1
Through peer review of claims to discoveries
1
1
Yes, through pipeline processing
1
Total
9
1
10
9
10
38

20. AI: In which evaluations is data
deposition, creation, maintenance, or
review addressed?
Answer (coded) Discipline
Do not know of any
AA
3
EE
P+
PH
Total
8
8
24
2
Question not asked
5
1
2
5
The process is evaluated
2
2
Incidentally, when data is re-processed
2
2
Through peer review of publications
1
1
2
No, it should be done but probably is not
1
1
Through peer review of claims to discoveries
1
1
Yes, through pipeline processing
1
Total
9
1
10
9
10
38

21. Questions relevant for DANS"
Deans, HR managers and Heads of
Departments
✦ Is your institution promoting data deposition
and data review?
Since when?
How?
✦ In which evaluations is data deposition,
creation, maintenance, or review addressed?
!21

22. DH: Is your institution promoting
data deposition and data review?
Answer (coded) Discipline
AA
EE
P+
PH
Total
Question not asked
1
2
1
4
Yes, university promotes
1
1
1
3
No promotion, but it may happen
1
No promotion
1
Total
2
!22
2
1
2
1
4
2
10

23. Dean in EE
✦ Motivation for researchers?
•Feels like extra bureaucracy
•Sit on their data
•What’s the point once you finish the project
•Checking for fraud
✦ Problems of depositing and re-use
•Meta-data > experimental setting
!23

24. Dean in PH
✦ Yes, the university is promoting
✦ Many questions to answer
•Should we store everything? Capacity
•Consent from patients, anonymity
✦ External institute or at the university?
•Trust in institution
•Technical reliability (15 years), back-ups etc.
!24

25. DH: In which evaluations is data
deposition, creation, maintenance, or
review addressed?
Answer (coded) Discipline
AA
EE
P+
PH
Do not know of any
1
2
1
2
Question not asked
1
Total
6
2
1
No evaluations, but interesting question
2
Total
!25
2
3
1
4
2
10

26. Conclusions
✦ Differences between disciplines, but also within
them
✦ DANS may get more busy than expected
•Outside pressure, from research councils and
through incidents such as Stapel case
•Or less busy … ?
✦ How to ‘sell’ the effort needed?
•How to turn data deposition into academic credit?
!26

27. !27

28. Thank you
for your attention
Questions, comments?
!28