Librarians are increasingly being asked to engage with bibliometrics to help with institutional decision making. However, few have professional qualifications in this area and there is an onus on universities to do metrics responsibly. A small project supported by the Lis-Bibliometrics forum and Elsevier Research Intelligence Division is developing a set of bibliometric competency statements to ensure practitioners are equipped to do their work responsibly and well. This workshop will report on progress to date and invite input into the project.

1. A N D R E W C O X - U N I V E R S I T Y O F S H E F F I E L D
E L I Z A B E T H G A D D - L O U G H B O R O U G H
U N I V E R S I T Y
What does it mean to be a competent
bibliometrician?
Creating a set of bibliometric
competency statements
LIS-Bibliometrics

2. What is bibliometrics and why does it matter?
 Bibliometrics = Biblio (books) + metrics
(measurements)
 Sub-discipline = citation analysis
 Garfield’s Journal Impact Factor (JIF) created for
Librarians to aid collection management
 Proliferation of metrics since turn of century
 Science policy & HE keen to measure performance,
value & impact

3. F
From Lorna Wildgaard, 2015. https://www.slideshare.net/LornaWildgaard/measure-
up-the-extent-authorlevel-bibliometric-indicators-are-appropriate-measures-of-
individual-researcher-performance

4. Three key calls to do metrics well
 2014: DORA (Declaration on Research Assessment) -
backlash against use of JIF for measuring individual
researchers
 2015: Metric Tide Report – could not recommend
bibliometrics as an alternative to peer review for
REF; called for all stakeholders to do metrics
responsibly
 2015: Leiden Manifesto – 10 principles for using
bibliometrics in research assessment

5. The response
 Many universities now signing DORA and
establishing Statements on the Responsible Use of
Metrics

6. University of Bath

7. University of Loughborough

8. The response
 Many universities now establishing Statements on
the Responsible Use of Metrics
 More posts in Libraries and Research Offices
established to perform bibliometrics
 Research by Sabrina Petersohn demonstrated how
Libraries were making a bid for jurisdiction in this
area

9. The challenge
 Libraries and Research Offices can struggle to fill
these posts
 Only 29% of our survey respondents had received
bibliometric training as part of an LIS qualification
 Individuals often required to learn on-the-job

10. Our response
 Loughborough University established a small set of
bibliometric competencies for Academic Librarians
 Lis-Bibliometrics Forum recognised the need for
something similar on a larger scale
 Elsevier Research Intelligence provided a small
amount of funding.
 University of Sheffield’s bid was successful

11. Why do we need bibliometric competencies?
 To communicate what a bibliometric ian does
 To ensure professionals working in this area are
trained and skilled to do bibliometrics well
 To guide Library Schools and other training
providers as to the levels of competence
professionals need to obtain
 To guide the recruitment of appropriately qualified
staff
 Ultimately, to ensure bibliometrics is done
professionally and responsibly

12. Aims of the study
Tasks
• To identify the tasks that those working in
bibliometrics undertake
Level
• To identify how those working in bibliometrics
perceive which of these tasks are entry level, core
and specialist
Model
• To produce a visual model of bibliometric
competencies and validate it with the community
Development of a community-supported set of bibliometric
competencies

13. Methods of the study
 A list of 99 bibliometric tasks assigned to 12 broad
categories was produced based on
 data from the lis-bibliometrics workshop in June 2016
 data from Sabrina’s PhD research including 10 expert
interviews with UK information professionals and various
documents such as blogs
 The list was compiled in a questionnaire and distributed to
members of lis bibliometrics, the metrics list of Arma
Association and sigmetrics@mail.asis.org during January
2017
 The results of the questionnaire were analysed by means of
descriptive statistics

14. Categorization of bibliometric tasks
A. Awareness raising and responsible use
B. Applications of bibliometrics
C. Metrics: About scholars, academic units and institutions
D. Metrics: About journals
E. Metrics: About articles/ specific outputs
F. Metrics: About Impact
G. Bibliometric tools
H. General data handling and presentation tasks
I. Training, education and advice to users
J. Systems procurement and use
K. Policy and strategy
L. Professional skills

15. Competency levels of bibliometric
tasks
 Entry level – a basic task of bibliometrics, one that a
newly qualified professional should be able to perform
 Core level – a core task of bibliometrics, one that an
established professional with a responsibility for
bibliometrics performs beyond entry level tasks
 Advanced/ Specialist level – a task involving very
specialist knowledge and evaluative skills
 Out of scope of the role

16. Responses to questionnaire
 92 valid responses, mostly from the UK (N=48)
55
20
4
12
Organisational location of respondents
Library
Research
administration
Planning
Academic
department

17. Preliminary findings
16
48
27
Distribution of levels across tasks
Entry level
Core level
Specialist level

18.  All the items in the list of tasks are considered to be part of
bibliometrics: none of the items were rated as out of scope by
a majority of respondents.
 The categories within which items were organised in the
survey, also seemed to make sense to participants: both the
task categories and the notion of entry level and core
categories.
 The data identifies a rather narrow entry level set of
competencies (16/99 tasks).
 48/99 tasks were identified as core
 64/99 items were rated as either core or entry level, thus
representing the main part of the role.
 The data identifies what is seen as specialist/advanced, as
opposed to the core of the job – 27/99 tasks.

19. The 16 Entry level tasks
A1. Explains the concept of bibliometrics 74 (80%)
A2. Explains the concept of altmetrics 65 (71%)
A10. Explains and promotes author identifiers, eg ORCID 53 (58%)
A13. Explains the benefits of open access 59 (64%)
C1. Uses bibliometric tools to find metrics on a specific scholar: H-index 56 (61%)
D1. Uses bibliometric tools to find metrics on an individual journal: JIF 68 (74%)
D2. Uses bibliometric tools to find metrics on an individual journal: 5 year impact factor 61 (66%)
D3. Uses bibliometric tools to find metrics on an individual journal: SNIP 62 (67%)
D4. Uses bibliometric tools to find metrics on an individual journal: Eigenfactor 52 (57%)
D5. Uses bibliometric tools to find metrics on an individual journal: SCImago Journal Rank 61 (66%)
E1. Uses bibliometric tools to find citations for a specific article 71 (77%)
L1. Works effectively within local institutional culture 48 (52%)
L6. Works effectively as part of a team with other library staff, colleagues in professional services and
researchers
48 (52%)
L7. Learns to update skills 57 (62%)
L8. Works independently 47 (51%)
L9. Completes work with attention to detail 65 (71%)

20. Core tasks
A3. Advises on which are the appropriate tool(s) for a particular metric 63 (69%)
A4. Explains differences in results between metrics based on different tools 62 (67%)
A6. Applies responsible use principles to specific requests/cases and in their own practices 59 (64%)
B3. ...Increase staff bibliometric literacy 67 (73%)
B13. ...Support academic bibliometric research 56 (61%)
G2. Chooses the right tool for a specific task 56 (61%)
I1. Writes documentation 60 (65%)
I2. Designs online training 63 (69%)
I3. Delivers group f2f training 69 (75%)
I4. Delivers 1:1 training 70 (76%)
L2. Creates and sustains professional networks inside the organisation 67 (73%)

21. Specialist tasks
A9. Understands the potential use of text mining in bibliometrics 59 (64%)
B6. Uses bibliometric knowledge to ...Evaluate departmental/research centre performance 57 (62%)
B8. ...Evaluate institutional performance 55 (60%)
B13. ...Support academic bibliometric research 56 (61%)
C7. Analyses collaboration patterns in a research group or department (including to compare
with competitors)
59 (64%)
C14. Examines trends in institutional performance and advises on improving its ranking 58 (63%)
F3. Gathers evidence to support a national research assessment exercise impact case study 55 (60%)
H2. Conducts manual statistical analyses outside of proprietary tools 58 (63%)
H3. Applies statistical tests of significance to analyses 73 (79%)
H4. Undertakes programming for downloading/manipulating data 68 (74%)
H5. Undertakes Network analysis for bibliometrics 76 (83%)
H6. Undertakes text mining for bibliometric purposes 72 (78%)
I5. Undertakes charged-for consultancy 61 (66%)
J4. Advises on decisions about how the institution should use specific tools 59 (64%)

22. Tasks out of scope
22
33
27
20
43
30
0 5 10 15 20 25 30 35 40 45 50
I5:Undertakes charged-for consultancy
E2:Evaluates quality of specific article
D9:Recommends a journal to publish in
purely through bibliometrics
C5:Evaluates the quality of research
group or department output
B7: Allocate funding to departments
B5: Uses bibliometric knowledge to
...Promote/employ staff

23. International differences
UK more likely to consider the following as entry
level/core:
 A5. Explains responsible use as a general set of principles
 A7. Advises on the applicability of metrics to particular
disciplines/metadisciplines (e.g. Arts and Humanities)
 E3. Advises on how to increase citations of articles
 E4. Advises on how to use social media to increase
citation
 E6. Explains metrics for research data

24. International differences
UK less likely to consider the following as entry
level/core:
 H1. Downloads, cleans and manipulates bibliometric
data
 H2. Conducts manual statistical analyses outside of
proprietary tools
 H4. Undertakes programming for
downloading/manipulating data
 I5. Undertakes charged-for consultancy

25. Growth areas in the next 5 years
Section heading Top three tasks
Awareness raising and responsible use Author identifiers 17 (46%)
Responsible use 16 (43%)
Applicability of metrics to specific disciplines 16
(43%)
Applications of bibliometrics National Research Assessment exercise 19 (68%)
Evaluating institutional performance 12 (57%)
Supporting grant applications 10 (36%)
Metrics: About scholars, academic units and
institutions
Institutional metrics and benchmarking 15
(68%)
Trends in institutional performance 15 (55%)
Identifying institutional strengths 12 (68%)
Metrics: About articles/ specific outputs Research data 20 (100%)
Metrics for books 10 (50%)
Use of social media 10 (50%)

26. Growth areas from lis-bibliometrics workshop
last month
 More use of tools for benchmarking, appraisal, university
strategy, of open access repositories/ publications – so
suggests a widening scope of application of
bibliometrics?
 “Range of tools that provide metrics continues to grow –
so really important to keep up to date” – mentions of MS
Academic
 Responsible use (including bibliometrics)
 “Awareness of responsible use ought to grow but may not, as you can
only grow what academics want to hear about”
 “More aware of indicators and asking libraries to help support them”
 “Critical approach and ethics of bibliometrics”
 “Honesty and openness about limitations”
 “More transparency with data” – maybe from the tool suppliers

27. Entry level competencies discussion
 Do you think anything should be added to our draft
statement of entry level competencies (ie the ones
required from a new graduate entering bibliometrics
work), e.g.
 Something about responsible use and the limitations of
bibliometrics
 Skills in training, be that individual, group or online
 Wider understanding of digital scholarship
 Something about understanding types of application of
bibliometrics: eg to empower academics, for monitoring and
evaluation, to support grant writing
 Are any other soft skills a priority?

28. Training discussion
 What training have you had in bibliometrics?
 Was bibliometrics included in your library degree, if you have
one?
 Or are you entirely self taught?
 What should entry level training at library school
include, in your opinion?

29. Training
 “If you have an Library/Information Studies
qualification, did it cover bibliometrics?” (55 responses)
 37 (65%) said that their library qualification had not included
bibliometrics.
 Only 16 (29%) said it had; 3 could not remember.
 “Apart from on an LIS course, have you received training
in bibliometrics? If so please give brief details.”
 Answers included courses run by commercial companies such as
Elsevier and by CWTS (at Leiden University), as well as individual
seminars and webinars and reading the literature. A few were highly
qualified with a masters or PhD in bibliometrics.

30. Careers in bibliometrics
 What is a day in the life of a bibliometrician like?
 What kind of aptitudes and values are advantageous?
 What sources of training are there?
 How is the job likely to evolve?

31. Acknowledgments
We wish to thank
 all participants in the survey for their input
 to the LIS-Bibliometrics committee for input in designing,
piloting and promoting the survey
 Elsevier and LIS-Bibliometrics for commissioning the study
 Laura Sbaffi (University of Sheffield) for the statistical
analyses

32. References
 DORA http://www.ascb.org/dora/
 Metric Tide https://responsiblemetrics.org/the-
metric-tide/
 Leiden Manifesto http://www.leidenmanifesto.org/
 Petersohn, S., 2016. Professional competencies and
jurisdictional claims in evaluative bibliometrics:
The educational mandate of academic librarians.
Education for information
http://content.iospress.com/articles/education-
for-information/efi972.