The document discusses the evolution and significance of alternative metrics ('altmetrics') in assessing scholarly content beyond traditional citation metrics. It highlights the need for standardized criteria to evaluate these alternative measures and their relevance in real-time trends and understanding the impact of research. Todd Carpenter, the executive director of NISO, emphasizes the importance of adapting and improving assessment tools to meet the changing landscape of scholarly communication.

1. Alternative Metrics
How the future of assessment can
improve discovery and delivery of
content
Todd Carpenter, Executive Director, NISO
Tools of Change Conference, Franfurt Book Fair
October 9, 2012

2. @TAC_NISO Twitter Highlights
• So excited to present on #altmetrics today during #tocfbf at #fbf12
• Again, I’m pre-tweeting my presentation. Slides to come! #tocfbf #fbf12
• #NISO = 1/3 Publishers, 1/3 Libraries, 1/3 automation suppliers. Communities converge to develop
standards #tocfbf #fbf12
• Standards are everywhere in book publishing, we just don’t notice them anymore. #tocfbf #fbf12
• In 1955 Eugene Garfield published “Citation Indexes for Science” in Science http://bit.ly/xfylwh #tocfbf
#fbf12
• Quote: “If I have seen further it is by standing on ye shoulders of Giants." Sir Isaac Newton
#tocfbf
#fbf12
• Sounds like beginning of a joke: 2 guys at Stanford, they wrote this paper, then started a company in a
garage ... #tocfbf #fbf12
• Citations are awesome, aren’t they? A perfect solution to all the world’s problems, no? #tocfbf #fbf12
• Citations aren’t always a measure of quality. They are also a very lagging indicator of impact #tocfbf
#fbf12
• #altmetrics a suite of assessment criteria & measures being developed to assess importance of a
particular scholarly output #tocfbf #fbf12
• #altmetrics examples: usage analysis, metrics of non-traditional content, observational or behavioral
analysis, social media #tocfbf #fbf12
• How 1 can apply #altmetrics Sell related content, spot trends in near-real-time, ID new fields or new
services, marketing #tocfbf #fbf12
• Real world #altmetrics examples: http://bit.ly/OzGnGt www.tynt.com http://bit.ly/TcNIh0 http://bit.ly/
PknZb4 #tocfbf #fbf12
• Issues needing consensus regarding #altmetrics - Authorship, provenance, definitions of components,
privacy, openness #tocfbf #fbf12

3. About
• Non-profit industry trade association accredited
by ANSI with ~130 members
• Mission of developing and maintaining technical
standards related to information,
documentation, discovery and distribution of
published materials and media
• Volunteer driven organization: 400+ spread out
across the world
• Responsible for standards like ISSN, DOI, Dublin
Core metadata, DAISY digial talking books,
OpenURL, MARC records, and ISBN (indirectly)

4. Standards are familiar, even if you don’t notice
Image: DanTaylor Image: : Joel Washing

5. Who is reading published content?
Someone we all hope!
In our world, people still read content.There are editors, hopefully professors, perhaps students, perhaps college students, perhaps even the
general public or even puppies. What we tend to forget though is that in our digitally mediated environment, there is one class of readers,
we don’t often think of…. Machines.

6. Who is reading published content?
Someone we all hope!
In our world, people still read content.There are editors, hopefully professors, perhaps students, perhaps college students, perhaps even the
general public or even puppies. What we tend to forget though is that in our digitally mediated environment, there is one class of readers,
we don’t often think of…. Machines.

7. We stand on the
shoulders of giants...
In responding to a letter from a scientific rival Robert Hooke, Sir Isaac Newton wrote;
“What Des-Cartes did was a good step. You have added much several ways, & especially in taking ye colours of thin plates into philosophical consideration. If I have seen further
it is by standing on ye shoulders of Giants."

8. The Mighty Citation

9. Impact factor
Fifty-seven years ago, on July 15, 1955, Eugene Garfield, Ph.D published his groundbreaking paper on citation indexing, “Citation Indexes for Science: A New Dimension in Documentation
through Association of Ideas.” in Science magazine. This innovative paper envisioned information tools that allow researchers to expedite their research process, evaluate the impact of their
work, spot scientific trends, and trace the history of modern scientific thoughts. With that paper, essentially he launched the field of bibliometrics.
Three years later, in July 1958, Eugene Garfield laid the foundations for ISI (Institute for Scientific Information) by borrowing $500 from Household Finance. He hired his first full-time
employee and began to build an organization that included more than 500 people when it was acquired by The Thomson Corporation in 1992.

10. Thomson-Reuters
IMPACT FACTOR
The impact factor and its compilation the Journal Citation Reports have become over the past
five-plus decades since it was launched, “the” metric for assessing journal quality. Journals
live and die by this metric. In some developing countries, authors are awarded bonuses if
you’ve been published in a highly-ranked title. Many in the STM community regularly tout
their publication’s performance on charts like this one for physical chemistry. At the end of
June when the JCR is released, it is often accompanied by a stream of press releases
announcing this or that title’s Impact Factor. For all it’s importance and value, the IF is an
imperfect measure and the community has been arguing about its imperfections for years.
These include, the time delay of citation data, the inability to compare different domains, the
lack of granularity and the figure’s over-use and misapplication. If there was one metric, that
the scholarly community was interested in finding an alternate to, it probably is this one.
That said, it is an ingenious and valuable metric that really has stood the test of time.

11. Two Stanford Students
Let’s fast forward about 50 years from Mr Garfield’s publication and the launch of the impact factor. Beyond the impact factor, citation-based assessment metrics do play an
important role in our community, though not in the way you might think, and certainly well beyond the domain of STM publishing.
So these two students were working on the Stanford Digital Library Project (SDLP). The SDLP's goal was “to develop the enabling technologies for a single, integrated and
universal digital library" and was funded through the National Science Foundation among other federal agencies. They were focused on the problem of finding out which web
pages link to a given page, considering the number and nature of such backlinks to be valuable information about that page (with the role of citations in academic publishing in
mind). This research project was nicknamed "BackRub".

12. They wrote a paper

13. They started a company in
someone’s garage
Photo: AP

14. whenever you use google, you are using a variant of a bibliometric citation analysis that is a
combination of reference linking and usage data to provide your search results. Basically,
this is an “alt-metric” by a different name.

15. A simple conclusion
can be drawn
Citations are awesome, aren’t they?
A perfect solution to all the world’s
problems, no?

16. Back to Sir Isaac Newton
“What Des-Cartes did was a
good step.You have added
much several ways, &
especially in taking ye
colours of thin plates into
philosophical consideration. If
I have seen further it is by
standing on ye shoulders of
Giants."
Reflecting on how I began this talk about citations, let’s return to the famous Newtonian
quote. “What Des-Cartes did was a good step. You have added much several ways, & especially in taking ye colours of thin plates into philosophical consideration. If I
have seen further it is by standing on ye shoulders of Giants." The quote came from an exchange of letters between Robert
Hooke and Newton about a paper that Newton had written on the properties of light. Hooke
had taken umbridge over the paper as it was something he had explored some ten years
prior in his seminal work “Micrographia”. Newton was not a man to dole out praise,
particularly to men with whom he disdained or had scientific disagreements with. It is not
likely that Newton viewed Hooke as an intellectual giant since it was in part Hooke’s
criticisms that led to Newton’s self-initiated withdrawal from the Royal Society in 1874. It is
very likely as some scholars have argued that the phrase "on the shoulders of giants" as a
veiled insult of Hooke, who was quite short, and also the butt of a successful theater farce at
the time entitled, The Virtuoso.
While many believe that was the sentiment being expressed by Newton in his letter to Hooke, some researchers have suggested that he was actually using the phrase "on the
shoulders of giants" as a veiled insult of Robert Hooke, who was a rather short man. Newton had a reputation as somewhat of a petty and vindictive man whose ego clashed
with those of his rivals in the scientific and mathematical communities. One of these rivals was Robert Hooke, who had been involved in a long-running fued with Newton over
which one had discovered the inverse square law. Although Newton's letter to Hooke appeared courteous on the surface, some historians have concluded that he cleverly
employed the phrase "on the shoulders of giants" to ridicule Hooke's lack of physical stature and imply that he lacked intellectual stature as well.

17. Caveat emptor
Citations
aren’t
always what
we think
they are
The quote came from an exchange of letters between Robert Hooke and Newton about a
paper that Newton had written on the properties of light. Hooke had taken umbridge over the
paper as it was something he had explored some ten years prior in his seminal work
“Micrographia”. Newton was not a man to dole out praise, particularly to men with whom he
disdained or had scientific disagreements with. It is not likely that Newton viewed Hooke as
an intellectual giant since it was in part Hooke’s criticisms that led to Newton’s self-initiated
withdrawal from the Royal Society in 1874. It is very likely as some scholars have argued that
the phrase "on the shoulders of giants" as a veiled insult of Hooke, who was quite short, and
also the butt of a successful theater farce at the time entitled, The Virtuoso.
While many believe that was the sentiment being expressed by Newton in his letter to Hooke, some researchers have suggested that he was actually using the phrase "on the
shoulders of giants" as a veiled insult of Robert Hooke, who was a rather short man. Newton had a reputation as somewhat of a petty and vindictive man whose ego clashed
with those of his rivals in the scientific and mathematical communities. One of these rivals was Robert Hooke, who had been involved in a long-running fued with Newton over
which one had discovered the inverse square law. Although Newton's letter to Hooke appeared courteous on the surface, some historians have concluded that he cleverly
employed the phrase "on the shoulders of giants" to ridicule Hooke's lack of physical stature and imply that he lacked intellectual stature as well.

18. Another problem with citations
Citation data reflects the very last stages associated with the publication process.
Usage data can reflect earlier stages and reflect a wide range of scholarly
communication activities; it serves as an early, and potentially more comprehensive
indicator.

19. Mapping the World of Science
Clickstream data yields high-
resolution maps of science
Johan Bollen, Herbert Van de Sompel, et al.
PLoS One, February 2009
The comprehensiveness of other forms of data collection and analysis can prove much more robust. For comparison, the JCR is a curated
list of only a fraction of the overall scholarly communications community. This image was published a few years back by Johan Bollen and
Herbert Von de Sompel based on large-scale data analysis of clickstream data. It shows the connections between publications based on
usage data and user session logs.

20. Mapping the World of Science
Clickstream data yields high-
resolution maps of science
Johan Bollen, Herbert Van de Sompel, et al.
PLoS One, February 2009
The comprehensiveness of other forms of data collection and analysis can prove much more robust. For comparison, the JCR is a curated
list of only a fraction of the overall scholarly communications community. This image was published a few years back by Johan Bollen and
Herbert Von de Sompel based on large-scale data analysis of clickstream data. It shows the connections between publications based on
usage data and user session logs.

21. Define Altmetrics,
Please

22. Usage Based Metrics

23. Citation of non-traditional content
Citations to new
forms of
communciation
aren’t as simple
as one might
think.
Define a use of
this?
Image: Domenico, Caron, Davis, et al.

24. Clickstream Analysis

25. Social Media Metrics
Image: Danny Brown, The State of Social Media Marketing 2012

26. Behavioral Metrics

27. Multivariate analysis

28. OK, it’s important, but
what does it mean for
MY BUSINESS

29. Have you bought
something from
Amazon?

31. Did you buy
something that
“other people bought”?

32. Realtime Trend Spotting

33. Innovation often takes
place at the
intersection of different
fields

34. Image: http://www.crispian.net/CrispiansScienceMap.html
The names you are probably most familiar with on this model of science are the names that
are household names, Kepler, Galileo, Newton, Darwin, Einstien, Crick, Leeuwenhoek or
Wolfram. All worked at the space between (or creating new domains).

35. Mapping the World of Science
Clickstream data yields high-
resolution maps of science
Johan Bollen, Herbert Van de Sompel, et al.
PLoS One, February 2009

36. Mapping the World of Science
Clickstream data yields high-
resolution maps of science
Johan Bollen, Herbert Van de Sompel, et al.
PLoS One, February 2009

37. Improve products and
customer services

38. Editorial applications

39. Author services
(“How is my book doing?”)

40. Marketing applications

41. Is anyone really doing
any of this?
(Beyond Amazon & Google?)

42. There are lots of things that make a photo 'interesting' (or not) in the Flickr. Where the clickthroughs are coming from; who comments on it and when; who marks it as a favorite; its tags and many more
things which are constantly changing. Interestingness changes over time, as more and more fantastic photos and stories are added to Flickr.

43. 70%
Anyone here work for a company using TYNT? You may not have ever heard of TYNT, but
it’s the 9th largest data collecter on the wbe, right behind AOL’s advertising network and is
the second most popular widget on the web behind the Facebook Connect App according to
kruxdigital.com. This widget tracks people’s copy/paste behavior through specific reference
linking.

44. Ex Libris is a provide of library management software and services. In 2009, ExLibris
launched the BX service, a scholarly recommender service. Recommendations generated by
the bX system are based on an analysis of usage logs derived from OpenURL link resolution
data. So users who move from article to article during a session leave a trail of usage data,
which is gathered and analyzed to present recommendations of other articles to view, similar
in some ways to what Amazon does.

46. Last summer, Dominique Raccah, CEO of Sourcebooks, presented a very interesting session
at the IDPF Digital Book 2012 meeting prior to BookExpo discussing how Sourcebooks uses
data that includes large-scale sales data, user studies, web traffic and social analytics to
measure and optimize their business. You can listen to a recording of her presentation from
the IDPF website.

47. Issues we face with
alternative metrics

48. Image: Flickr user karindalziel

49. Authorship
-----
Disambiguation
Contributor Roles

50. Basic Definitions
Needed
(So we are all talking
about the same thing)

51. More open exchange of
component data

52. All you need is....
Large Amounts of Data
Good Analysts
Implementation Strategy
Creativity
Standards
(we can only do so much)

53. One Very Last Thing…
Image from Will Lion: http://www.flickr.com/photos/will-lion/2646213692/
Charles Mingus (22 April 1922 – 5 January 1979) was an American jazz bassist, composer, bandleader, and occasional pianist also known
for his activism against racial injustice.
Participating in this process is a very valuable thing for your organization, for your customers and end-users.

54. Thank you!
Todd Carpenter, Executive Director
tcarpenter@niso.org
National Information Standards Organization (NISO)
3600 Clipper Mill Road, Suite 302
Baltimore, MD 21211 USA
+1 (301) 654-2512
www.niso.org