Do We Need
Standards for
Alternative Metrics?
What are we hoping to
accomplish?
Todd Carpenter, Executive Director, NISO
N...
• Non-profit industry trade association accredited
by ANSI with 150+ members
• Mission of developing and maintaining techni...
We stand on the
shoulders of giants...
Friday, October 11, 13
The Mighty Citation
Friday, October 11, 13
Impact factor
Friday, October 11, 13
Thomson-Reuters
IMPACT FACTOR
Friday, October 11, 13
Two Stanford Students
Friday, October 11, 13
They wrote a paper
Friday, October 11, 13
They started a company in
someone’s garage
Photo:AP
Friday, October 11, 13
Friday, October 11, 13
Citations are awesome, aren’t they?
A perfect solution to all the world’s
problems, no?
A simple conclusion
can be drawn
F...
“What Des-Cartes did was a
good step.You have added
much several ways, &
especially in taking ye
colours of thin plates in...
Citations
aren’t
always what
we think
they are
Caveat emptor
Friday, October 11, 13
Another problem with citations
Friday, October 11, 13
So where to from here?
Friday, October 11, 13
Define altmetrics,
please
Friday, October 11, 13
Usage Based Metrics
Friday, October 11, 13
Clickstream Analysis
Friday, October 11, 13
Social Media Metrics
Image: Danny Brown,The State of Social Media Marketing 2012
Friday, October 11, 13
Clickstream data yields high-
resolution maps of science
Johan Bollen, HerbertVan de Sompel, et al.
PLoS One, February 200...
Clickstream data yields high-
resolution maps of science
Johan Bollen, HerbertVan de Sompel, et al.
PLoS One, February 200...
Behavioral Metrics
Friday, October 11, 13
Multivariate analysis
Friday, October 11, 13
Sentiment Analysis
Friday, October 11, 13
Image: Domenico, Caron, Davis, et al.
Citation of non-traditional content
Citations to new
forms of
communication
aren’t a...
We’ve got
what, now how?
Friday, October 11, 13
Image: Flickr user karindalziel
What can be measured?
Friday, October 11, 13
How do we measure
things?
Friday, October 11, 13
At what granularity do
we measure things?
Friday, October 11, 13
Which measurements
are valuable?
Friday, October 11, 13
How do we ensure the
validity of those
measurements?
Friday, October 11, 13
How to
(or even if we should)
deal with other issues
such as gaming?
Friday, October 11, 13
Issues we face
with alternative
metrics
Friday, October 11, 13
Basic Definitions
(So we are all talking
about the same thing)
Friday, October 11, 13
Comparison across providers
Source:	
  Scott	
  Chamberlain,	
  Consuming	
  Article-­‐Level	
  Metrics:
Observations	
  A...
Consistency in what is counted
Source:	
  Scott	
  Chamberlain,	
  Consuming	
  Article-­‐Level	
  Metrics:
Observations	
...
Authorship
-----
Disambiguation
&
Contributor Roles
Friday, October 11, 13
Open exchange of
component data
Friday, October 11, 13
TRUST
Friday, October 11, 13
What is NISO trying to do?
Friday, October 11, 13
Friday, October 11, 13
Friday, October 11, 13
Alterna(ve	
  Assessment	
  
Ini(a(ve	
  
Phase	
  1	
  Mee(ngs
October	
  9,	
  2013	
  	
  -­‐	
  San	
  Francisco,	
  C...
Alterna(ve	
  Assessment	
  Ini(a(ve	
  
Phase	
  2
Presenta(ons	
  of	
  report	
  (June	
  2014)
Priori(za(on	
  Effort	
...
Criticism
Friday, October 11, 13
Why now?
Isn’t it too soon?
Can NISO to do this?
Friday, October 11, 13
How soon is now?
Friday, October 11, 13
These things take time
Friday, October 11, 13
It will be hard to change
Friday, October 11, 13
Friday, October 11, 13
“All I see now is
blonde, brunette,
redhead”
Friday, October 11, 13
One Very Last Thing…
Image from Will Lion: http://www.flickr.com/photos/will-lion/2646213692/
Friday, October 11, 13
Thank you!
Todd Carpenter, Executive Director
tcarpenter@niso.org
National Information Standards Organization (NISO)
3600 ...
Upcoming SlideShare
Loading in...5
×

Alternative assessment metrics initiateve meeting 1 opening

443

Published on

Todd Carpenter's presentation at the opening of the Alternative Assessment discussion forum in San Francisco, CA on 10/09/13

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
443
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Alternative assessment metrics initiateve meeting 1 opening

  1. 1. Do We Need Standards for Alternative Metrics? What are we hoping to accomplish? Todd Carpenter, Executive Director, NISO NISO Alternative Assessment Initiative Discussion Forum #1 San Francisco, CA | October 9, 2013 Friday, October 11, 13
  2. 2. • Non-profit industry trade association accredited by ANSI with 150+ members • Mission of developing and maintaining technical standards related to information, documentation, discovery and distribution of published materials and media • Volunteer driven organization: 400+ spread out across the world • Responsible in various wasy for standards like ISSN, DOI, Dublin Core metadata, DAISY digital talking books, OpenURL, MARC records, and ISBN About Friday, October 11, 13
  3. 3. We stand on the shoulders of giants... Friday, October 11, 13
  4. 4. The Mighty Citation Friday, October 11, 13
  5. 5. Impact factor Friday, October 11, 13
  6. 6. Thomson-Reuters IMPACT FACTOR Friday, October 11, 13
  7. 7. Two Stanford Students Friday, October 11, 13
  8. 8. They wrote a paper Friday, October 11, 13
  9. 9. They started a company in someone’s garage Photo:AP Friday, October 11, 13
  10. 10. Friday, October 11, 13
  11. 11. Citations are awesome, aren’t they? A perfect solution to all the world’s problems, no? A simple conclusion can be drawn Friday, October 11, 13
  12. 12. “What Des-Cartes did was a good step.You have added much several ways, & especially in taking ye colours of thin plates into philosophical consideration. If I have seen further it is by standing on ye shoulders of Giants." Back to Sir Isaac Newton Friday, October 11, 13
  13. 13. Citations aren’t always what we think they are Caveat emptor Friday, October 11, 13
  14. 14. Another problem with citations Friday, October 11, 13
  15. 15. So where to from here? Friday, October 11, 13
  16. 16. Define altmetrics, please Friday, October 11, 13
  17. 17. Usage Based Metrics Friday, October 11, 13
  18. 18. Clickstream Analysis Friday, October 11, 13
  19. 19. Social Media Metrics Image: Danny Brown,The State of Social Media Marketing 2012 Friday, October 11, 13
  20. 20. Clickstream data yields high- resolution maps of science Johan Bollen, HerbertVan de Sompel, et al. PLoS One, February 2009 Clickstream network analysis Friday, October 11, 13
  21. 21. Clickstream data yields high- resolution maps of science Johan Bollen, HerbertVan de Sompel, et al. PLoS One, February 2009 Clickstream network analysis Friday, October 11, 13
  22. 22. Behavioral Metrics Friday, October 11, 13
  23. 23. Multivariate analysis Friday, October 11, 13
  24. 24. Sentiment Analysis Friday, October 11, 13
  25. 25. Image: Domenico, Caron, Davis, et al. Citation of non-traditional content Citations to new forms of communication aren’t as simple as one might think. Define a use of this? Friday, October 11, 13
  26. 26. We’ve got what, now how? Friday, October 11, 13
  27. 27. Image: Flickr user karindalziel What can be measured? Friday, October 11, 13
  28. 28. How do we measure things? Friday, October 11, 13
  29. 29. At what granularity do we measure things? Friday, October 11, 13
  30. 30. Which measurements are valuable? Friday, October 11, 13
  31. 31. How do we ensure the validity of those measurements? Friday, October 11, 13
  32. 32. How to (or even if we should) deal with other issues such as gaming? Friday, October 11, 13
  33. 33. Issues we face with alternative metrics Friday, October 11, 13
  34. 34. Basic Definitions (So we are all talking about the same thing) Friday, October 11, 13
  35. 35. Comparison across providers Source:  Scott  Chamberlain,  Consuming  Article-­‐Level  Metrics: Observations  And  Lessons  From  Comparing  Aggregator  Provider  Data,   Information  Standards  Quarterly,  Summer  2013,  Vol  25,  Issue  2. Friday, October 11, 13
  36. 36. Consistency in what is counted Source:  Scott  Chamberlain,  Consuming  Article-­‐Level  Metrics: Observations  And  Lessons  From  Comparing  Aggregator  Provider  Data,   Information  Standards  Quarterly,  Summer  2013,  Vol  25,  Issue  2. Friday, October 11, 13
  37. 37. Authorship ----- Disambiguation & Contributor Roles Friday, October 11, 13
  38. 38. Open exchange of component data Friday, October 11, 13
  39. 39. TRUST Friday, October 11, 13
  40. 40. What is NISO trying to do? Friday, October 11, 13
  41. 41. Friday, October 11, 13
  42. 42. Friday, October 11, 13
  43. 43. Alterna(ve  Assessment   Ini(a(ve   Phase  1  Mee(ngs October  9,  2013    -­‐  San  Francisco,  CA December  11,  2013  -­‐  Washington,  DC January  23-­‐24  -­‐  Philadelphia,  PA Phase  1  report  expected  in  May  2014 Friday, October 11, 13
  44. 44. Alterna(ve  Assessment  Ini(a(ve   Phase  2 Presenta(ons  of  report  (June  2014) Priori(za(on  Effort  (June  -­‐  Aug,  2014) Project  approval  (Sept  2014) Working  group  forma(on  (Oct  2014) Consensus  Development  (Nov  2014  -­‐  Dec  2015) Trial  Use  Period  (Dec  15  -­‐  Mar  16) Publica(on  of  final  recommenda(ons  (Jun  16) Friday, October 11, 13
  45. 45. Criticism Friday, October 11, 13
  46. 46. Why now? Isn’t it too soon? Can NISO to do this? Friday, October 11, 13
  47. 47. How soon is now? Friday, October 11, 13
  48. 48. These things take time Friday, October 11, 13
  49. 49. It will be hard to change Friday, October 11, 13
  50. 50. Friday, October 11, 13
  51. 51. “All I see now is blonde, brunette, redhead” Friday, October 11, 13
  52. 52. One Very Last Thing… Image from Will Lion: http://www.flickr.com/photos/will-lion/2646213692/ Friday, October 11, 13
  53. 53. Thank you! Todd Carpenter, Executive Director tcarpenter@niso.org National Information Standards Organization (NISO) 3600 Clipper Mill Road, Suite 302 Baltimore, MD 21211 USA +1 (301) 654-2512 www.niso.org Friday, October 11, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×