Traditionally, advanced bibliometrics have been the ‘gold standard’ in research evaluations in many fields. Due to changes in communication patterns in various fields, we now see alternative ways of assessing research appearing on the landscape. One of the major developments in scientific communication patterns is the advent of the Openness movement, through which various activities in academic life become more democratic, transparent, and hopefully fairer. This stretches out to publishing and the costs involved, how data are shared, and how peer review is organized, to name some instances in which the issue of Openness is raised. Of a somewhat more recent nature is the way assessment of scholarly activity is organized, in particular with respect to the way the various audiences with whom scholars are communicating are considered. A new way of looking at research assessment is through the recent ‘alternative metrics’ or also referred to as Altmetrics. As more classical bibliometrics are under pressure, due to international (DORA-Declaration) and national debates and initiatives (SiT) related to the organization of research assessment in various layers of the science system. This stirs a re-focus from science policy towards alternative ways to assess research performance. In this presentation we will show, by a recent example, how careful we have to be in making choices for metrics in order to support research assessment practices as well science policy decision making.

1. Slowing the pace on applying
metric techniques on Open
Science
Dr. Thed van Leeuwen, Dr Rodrigo Costas and Clifford Tatum, Msc
Contribution to the 2:AM workshop on Altmetrics,
Amsterdam, 08-10-2015

2. Issue of this lecture
• CWTS received a call on “Trends and drivers of Open
Science” in Europe.
• Starting assumptions of the call:
– Open Science is a well-established practice across the scientific
landscape, …
– that can be measured all along the full cycle of the knowledge
production process, and …
– tools for that become more and more available.
1

3. The advent of metrics in relation to
science policy issues
2
time
1950’s Start of metrics, Kuhn, De Solla Price
1960’s
1970’s
Science Citation Index, Garfield
Applying metrics in national NSF reports
1980’s Academic metrics units starting up
1990’s
2000’s
2010-2015
Applying metrics in Europe
Science mapping in Europe
Open Science and Altmetrics as next big policy issues

4. Rise of performance indicators & bibliometrics
Externally: Increasing need for formalised measures:
• ‘Push’ from science policy (from 1970s onwards)
• Independent of peer review
• New Public Management / Neo-liberalism (from 1980s onwards)
Internally: Growing pressure on the research community:
• Researchers part of international community, …
– Peer review
• ... but also part of local institutions
– Specific management practices (yearly appraisals, external evaluations)
• Institute managers not always part of international expert community
• From the 1990’s: tighter forms of management
– Distance

5. Research cycle, or knowledge
production process
4
Analysis
Publication
Review
Data gathering
Conceptualization

6. Research cycle & Open Science Trends
5
Analysis
Publication
Review
Data gathering
Conceptualization
Citizens science
Open Code
Pre-Print
Open Access
Data intensive
Open labbooks/
Open Data
Open annotation
Scientific blogs
Collaborative
Alternative reputation
systems
bibliographies
workflows

7. Adding altmetric techniques to the Open
Science model *
6
Analysis
Publication
Review
Data gathering
Conceptualization
Citizens science
Open Code
Pre-Print
Open Access
Data intensive
Open labbooks/
Open Data
Open annotation
Scientific blogs
Collaborative
Alternative reputation
systems
bibliographies
workflows
DOAJ List
RoarEprints.org
ArXiv
RunMyCode.org
SciStarter.com
FigShare.com
MyExperiment.org
dataDryad.org
OpenAnnotation.org
Researchgate.com
Mendeley.com
AltMetric.com
Academia.edu
SlideShare.com
ImpactStory.com
SlideShare.com
* Thanks to colleagues from Technopolis

8. Some conclusions …
• Classical bibliometrics mainly focuses on the output and impact
related issues.
• Altmetric techniques describe other elements of the knowledge
production process.
• But, not in all domains of scholarly activity has Open Science landed
already to the same extent.
• Nor are the altmetric techniques and data already matured so far to
be used to the full extent in a science policy context.
7

9. … and more conclusions …
• The application in the Open Science model of various altmetric
techniques throughout the whole knowledge production process
means a further metricization of the monitoring and measuring of
research activities.
• Conducting the study would mean to legitimize the assumption that
Open Science has indeed been accepted across the whole scientific
landscape.
• Furthermore, the study would also have such a legitimizing effect
on the usage of altmetrics along the full knowledge production
process
• Are Open Science advocates as well as altmetrics researchers in
favor of this development ?
8

10. Any questions?
Ask me now, or mail me
Leeuwen@cwts.nl
9