The Infrastructure for Alternative Metrics: What do we need to compare digital apples to digital apples? Todd Carpenter, Executive Director, NISO Presented at AAAS Annual Meeting: February 16, 2014 Chicago, IL

1. The Infrastructure for
Alternative Metrics
What do we need to compare
digital apples to digital apples?
!
Todd Carpenter, Executive Director, NISO
AAAS Annual Meeting
Chicago, IL | February 16, 2014

2. We hate discussing infrastructure

3. Photo: Minneapolis College
of Art and Design Library

4. About
•
Non-profit industry trade association accredited
by ANSI with 190+ members
•
Mission of developing and maintaining technical
standards related to information, documentation,
discovery and distribution of published materials
and media
•
Volunteer driven organization: 400+ spread out
across the world
•
Responsible in various wasy for standards like
ISSN, DOI, Dublin Core metadata, DAISY digital
talking books, OpenURL, MARC records, and ISBN

5. Photo by dumbledad on Flickr http://www.flickr.com/photos/dumbledad/298650884/!

6. But what is the
“infrastructure” for
assessment?

7. We stand on the
shoulders of giants...

8. The Mighty Citation

9. Impact factor

10. Thomson-Reuters
IMPACT FACTOR

11. Two Stanford Students

12. Two Stanford Students

13. They wrote a paper

14. They started a company in
someone’s garage
Photo: AP

16. A simple conclusion
can be drawn
Citations are awesome, aren’t they?
A perfect solution to all the world’s
problems, no?

17. Back to Sir Isaac Newton
“What Des-Cartes did
was a good step.You have
added much several
ways, & especially in
taking ye colours of thin
plates into philosophical
consideration. If I have
seen further it is by
standing on ye shoulders
of Giants."

18. Caveat emptor
Citations
aren’t always
what we
think they
are

19. Another problem with citations

20. So where to from here?

21. Define altmetrics,
please

22. Social Media Metrics
Image: Danny Brown, The State of Social Media Marketing 2012

23. Usage Based Metrics

24. Click Map Analysis

25. Network Analysis
Clickstream data yields highresolution maps of science
Johan Bollen, Herbert Van de Sompel, et al.
PLoS One, February 2009

26. Network Analysis
Clickstream data yields highresolution maps of science
Johan Bollen, Herbert Van de Sompel, et al.
PLoS One, February 2009

27. Behavioral Metrics

28. Sentiment Analysis

29. Non-traditional content
Assessing
impact of
new forms of
communication
isn’t as simple
as one might
think.
Image: Domenico, Caron, Davis, et al.

30. Non-traditional content
Assessing
impact of
new forms of
communication
isn’t as simple
as one might
think.
Image: Domenico, Caron, Davis, et al.

31. We’ve got what,
now how?

32. What can be measured?
Image: Flickr user karindalziel

33. How do we
measure things?

34. At what granularity do
we measure things?

35. Consistency in what is counted
Source:
Scott
Chamberlain,
Consuming
Article-­‐Level
Metrics:
Observations
And
Lessons
From
Comparing
Aggregator
Provider
Data,
Information
Standards
Quarterly,
Summer
2013,
Vol
25,
Issue
2.

36. Which measurements are
valuable?

37. How do we ensure the
validity of those
measurements?

38. How to
(or even if we should)
deal with other issues
such as gaming?

39. What infrastructure
do we need for
alternative metrics?

40. Basic Definitions
!
(So we are all talking
about the same thing)

41. Comparison across providers
Source:
Scott
Chamberlain,
Consuming
Article-­‐Level
Metrics: Observations
And
Lessons
From
Comparing
Aggregator
Provider
Data,
Information
Standards
Quarterly,
Summer
2013,
Vol
25,
Issue
2.

42. Authorship
----Disambiguation
&
Contributor Roles

43. Element Identification

44. Open exchange of
component data

45. =
TRUST

46. Where does NISO fit in?

49. Alterna(ve
Assessment
Ini(a(ve
!
Phase
1
Mee(ngs
October
9,
2013
-­‐
San
Francisco,
CA
December
11,
2013
-­‐
Washington,
DC
January
23-­‐24
-­‐
Philadelphia,
PA
!
Phase
1
report
expected
in
May
2014

50. Just
a
few
ideas
from
in-­‐person
mee(ngs
Disambigua)on
problems
Create
a
central
repository
for
data
Define
provenance
for
metrics
Foster
culture
of
DOI
use
in
mass
media
Establish
defini)ons
for
metric
types
Define
data
extract/gathering
methods
Standardize
units
of
measure
Define
levels
of
transparency/privacy
!

51. Alterna(ve
Assessment
Ini(a(ve
!
Phase
2
Presenta(ons
of
report
(June
2014)
Priori(za(on
Effort
(June
-­‐
Aug,
2014)
Project
approval
(Sept
2014)
Working
group
forma(on
(Oct
2014)
Consensus
Development
(Nov
2014
-­‐
Dec
2015)
Trial
Use
Period
(Dec
15
-­‐
Mar
16)
Publica(on
of
final
recommenda(ons
(Jun
16)

52. May 15, 2013
74

53. Thank you!
Todd Carpenter, Executive Director
tcarpenter@niso.org
@TAC_NISO
!
National Information Standards Organization (NISO)
3600 Clipper Mill Road, Suite 302
Baltimore, MD 21211 USA
+1 (301) 654-2512
www.niso.org