Todd Carpenter's plenary presentation during the UKSG conference in Glasgow, UK on March 30, 2015. Todd discussed the development of new forms of assessment, the need to build trust in metrics and that the community should stop considering them "alternative".

1. “Alt”
is
German
for
“old”
It’s
&me
to
stop
talking
about
metrics
in
terms
of
“alterna&ve”
metrics.
Todd A. Carpenter
Executive Director, NISO
@TAC_NISO
March 30, 2015

2. ! Non-­‐profit
industry
trade
associa&on
accredited
by
the
American
Na&onal
Standards
Ins&tute
! Mission
of
developing
and
maintaining
technical
standards
related
to
informa&on,
documenta&on,
discovery
and
distribu&on
of
published
materials
and
media
! Volunteer
driven
organiza&on:
400+
contributors
spread
out
across
the
world
! Responsible
(directly
and
indirectly)
for
standards
like
ISSN,
DOI,
Dublin
Core
metadata,
DAISY
digital
talking
books,
OpenURL,
MARC
records,
and
ISBN
About
March
30,
2015
2

3. 1980s Alternative
John
Taylor,
Duran
Duran

4. 1920s Alternative

5. 2010s Not-so Alternative

6. Since
Jason
Priem
coined
the
term
“altmetrics”
• There
have
been
some
1,170
ar&cles
and
posts
(per
Google
Scholar)
on
the
topic
of
altmetrics
• There
have
been
eight
altmetrics
conferences
• There
are
now
at
least
four
companies
focused
on
providing
altmetrics-­‐style
analy&cs
March
30,
2015
6

7. When it becomes a standard,
its not alterative any more…

8. But
what’s
the
value
in
coun&ng
Tweets
and
Facebook
likes?
March
30,
2015
8

9. More
than
just
popularity
Research
is
poin&ng
to
the
fact
that
there
is
a
modest
posi&ve
correla&on
between
early-­‐signal
metrics
(altmetrics)
and
later-­‐signal
metrics
(cita&ons)
Do
altmetrics
correlate
with
cita6ons?
Extensive
comparison
of
altmetric
indicators
with
cita6ons
from
a
mul6disciplinary
perspec6ve
by
Rodrigo
Costas,
Zohreh
Zahedi,
Paul
Wouters
Do
Altmetrics
Work?
TwiBer
and
Ten
Other
Social
Web
Services
by
Mike
Thelwall
,
Stefanie
Haustein,
Vincent
Larivière,
&
Cassidy
R.
Sugimoto
(2006)
Earlier
web
usage
sta&s&cs
as
predictors
of
later
cita&on
impact
by
Brody
T,
Harnad
S
&
Carr
L
March
30,
2015
9

10. More
importantly:
Scholarship
today
is
much
more
than
journals
March
30,
2015
10

11. How
do
we
measure
the
impact
of
these
different
forms
of
output?
March
30,
2015
11

12. Would
a
researcher
focus
on
only
one
data
source
or
methodological
approach?
March
30,
2015
12
Duke
University
-­‐
Informa&on
Ini&a&ve
at
Duke
(IID)

13. There
aren’t
metrics
and
“altmetrics”;
there
are
only
metrics!
March
30,
2015
13

14. We
have
been
using
non-­‐cita&on-­‐based
metrics
for
decades
March
30,
2015
14

15. March
30,
2015
15

16. March
30,
2015
16
Traverse
Area
District
Library
(TADL)

17. What
has
changed
is
our
ability
to
collect
and
analyze
data
in
new
ways
that
are
meaningful
to
others
March
30,
2015
17

18. What are we lacking?
=

19. TRUST
=

20. What are the elements
of metrics that we need
to build trust?

21. Granularity
Time Scale
Identification
Definition
Exchange

22. I
omen
sound
like
a
broken
record
• Defining
what
is
to
be
counted
=
standards
• How
to
describe
what
to
count
=
standards
• Iden&fica&on
of
what
to
count
=
standards
• Procedures
for
coun&ng
or
not
=
standards
• Aggrega&ng
counts
from
network
=
standards
• Exchange
of
what
was
counted
=
standards
March
30,
2015
22

25.
Phase
1
Mee6ngs
October
9,
2013
-­‐
San
Francisco,
CA
December
11,
2013
-­‐
Washington,
DC
January
23-­‐24
-­‐
Philadelphia,
PA
Round
of
1-­‐on-­‐1
interviews
–
March/Apr
Phase
1
report
published
in
June
2014

26. Mee&ng
Lightning
Talks
• Expecta&ons
of
researchers
• Exploring
disciplinary
differences
in
the
use
of
social
media
in
scholarly
communica&on
• Altmetrics
as
part
of
the
services
of
a
large
university
library
system
• Deriving
altmetrics
from
annota&on
ac&vity
• Altmetrics
for
Ins&tu&onal
Repositories:
Are
the
metadata
ready?
• Snowball
Metrics:
Global
Standards
for
Ins&tu&onal
Benchmarking
• Interna&onal
Standard
Name
Iden&fier
• Altmetric.com,
Plum
Analy&cs,
Mendeley
reader
survey
• Twiper
Inconsistency
March
30,
2015
26
“Lightning"
by
snowpeak
is
licensed
under
CC
BY
2.0

27. March
30,
2015
27

28. March
30,
2015
28

29. March
30,
2015
29

30. 30
One-­‐on-­‐One
Interviews
March
30,
2015
30

31. Lots
and
lots
of
ideas
March
30,
2015
31

32. Alterna6ve
Assessment
Ini6a6ve
Phase
2
Presenta6ons
of
Phase
1
report
(June
2014)
Priori6za6on
Effort
(June
-­‐
Aug,
2014)
Project
approval
(Nov
-­‐
Dec
2014)
Working
group
forma6on
(Jan
-­‐
March
2015)
Consensus
Development
(March
2015
-­‐
Feb
2016)
Trial
Use
Period
(Feb
15
-­‐
May
16)
Publica6on
of
final
recommenda6ons
(Aug
16)

33. White
Paper
Released
March
30,
2015
33

34. 200
+
Ideas
25
Poten6al
Work
Themes
Defini6ons
Applica6on
to
types
of
research
outputs
Discovery
implica6ons
Research
evalua6on
Data
quality
and
gaming
Grouping,
aggrega6ng,
and
granularity
Context
Adop6on
March
30,
2015
34

35. March
30,
2015
35
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Unimportant
Of
liple
importance
Moderately
important
Important
Very
important
Community
Feedback
on
Project
Idea
Themes
n=118

36. March
30,
2015
36
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Unimportant
Of
liple
importance
Moderately
important
Important
Very
important
n=118
Community
Feedback
on
Project
Idea
Themes

37. Top-­‐ranked
ideas
(very
important
&
important
>70%)
• Develop
specific
defini&ons
for
alterna&ve
assessment
metrics.
(87.9%)
• Promote
and
facilitate
use
of
persistent
iden&fiers
in
scholarly
communica&ons.
(82.8%)
• Develop
strategies
to
improve
data
quality
through
normaliza&on
of
source
data
across
providers.
(80.8%)
• Iden&fy
research
output
types
that
are
applicable
to
the
use
of
metrics.
(79.8%)
• Define
appropriate
metrics
and
calcula&on
methodologies
for
specific
output
types,
such
as
somware,
datasets,
or
performances.
(78.1%)
• Explore
crea&on
of
standardized
APIs
or
download
or
exchange
formats
to
facilitate
data
gathering.
(72.5%)
• Research
issues
surrounding
the
reproducibility
of
metrics
across
providers.
(70.7%)
March
30,
2015
37

38. Launching
three
working
groups
March
30,
2015
38
A)
Development
of
defini&ons
and
descrip&ons
of
use
B)
Defini&ons
for
appropriate
metrics
and
calcula&on
methodologies
for
non-­‐tradi&onal
output
types.
Also
work
toward
promo&on
and
facilita&on
of
use
of
persistent
iden&fiers
C)
Development
of
strategies
to
improve
data
quality
through
source
data
providers

39. Steering Committee
Project Site:
www.niso.org/topics/tl/altmetrics_initiative/
White Paper:
http://www.niso.org/apps/group_public/download.php/
13295/niso_altmetrics_white_paper_draft_v4.pdf
March
30,
2015
39

40. Alterna6ve
Assessments
of
our
Assessment
Ini6a6ve
White
paper
downloaded
7,400
6mes
21
substan&ve
comments
received
120
in-­‐person
and
virtual
par&cipants
at
the
mee&ngs
These
3
mee&ngs
apracted
>400
RSVPs
for
live
stream
Goal:
generate
about
40
ideas,
in
total,
generated
more
than
250
Project
materials
downloaded
more
than
22,000
&mes
More
than
550
direct
tweets
using
the
#NISOALMI
hashtag
Survey
ranking
of
output
by
118
people
Six
ar&cles
in
tradi&onal
news
publica&ons
15
blog
posts
about
the
ini&a&ve

41. For more
Project Site:
www.niso.org/topics/tl/altmetrics_initiative/
White Paper:
http://www.niso.org/apps/group_public/download.php/
13295/niso_altmetrics_white_paper_draft_v4.pdf
March
30,
2015
41

42. Questions?
Todd Carpenter
Executive Director
tcarpenter@niso.org
@TAC_NISO
National Information Standards Organization (NISO)
3600 Clipper Mill Road, Suite 302
Baltimore, MD 21211 USA
+1 (301) 654-2512
www.niso.org
March
30,
2015
42