Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
“Alt”	
  is	
  German	
  for	
  “old”	
  
It’s	
  &me	
  to	
  stop	
  talking	
  	
  
about	
  metrics	
  in	
  terms	
  ...
!  Non-­‐profit	
  industry	
  trade	
  associa&on	
  	
  
accredited	
  by	
  the	
  American	
  Na&onal	
  Standards	
  I...
1980s Alternative	
  
John	
  Taylor,	
  
Duran	
  Duran	
  
1920s Alternative	
  
2010s Not-so Alternative	
  
Since	
  Jason	
  Priem	
  coined	
  the	
  term	
  
“altmetrics”	
  
•  There	
  have	
  been	
  some	
  1,170	
  ar&cles...
When it becomes a standard,
its not alterative any more…	
  
But	
  what’s	
  the	
  value	
  in	
  
coun&ng	
  Tweets	
  and	
  
Facebook	
  likes?	
  
March	
  30,	
  2015	
   8	
  
More	
  than	
  just	
  popularity	
  
Research	
  is	
  poin&ng	
  to	
  the	
  fact	
  that	
  there	
  is	
  a	
  
mode...
More	
  importantly:	
  Scholarship	
  today	
  	
  
is	
  much	
  more	
  than	
  journals	
  
March	
  30,	
  2015	
   1...
How	
  do	
  we	
  measure	
  the	
  
impact	
  of	
  these	
  different	
  
forms	
  of	
  output?	
  
March	
  30,	
  201...
Would	
  a	
  researcher	
  focus	
  on	
  only	
  one	
  data	
  
source	
  or	
  methodological	
  approach?	
  
	
  
Ma...
There	
  aren’t	
  	
  
metrics	
  and	
  “altmetrics”;	
  
there	
  are	
  only	
  metrics!	
  
	
  
March	
  30,	
  2015...
We	
  have	
  been	
  using	
  	
  
non-­‐cita&on-­‐based	
  
metrics	
  for	
  decades	
  
March	
  30,	
  2015	
   14	
  
March	
  30,	
  2015	
   15	
  
March	
  30,	
  2015	
   16	
  
Traverse	
  Area	
  District	
  Library	
  (TADL)	
  
What	
  has	
  changed	
  is	
  our	
  
ability	
  to	
  	
  
collect	
  and	
  analyze	
  data	
  	
  
in	
  new	
  ways	...
What are we lacking?	
  
=	
  
TRUST	
  
=	
  
What are the elements
of metrics that we need
to build trust?	
  
Granularity	
  
Time Scale	
  
Identification	
  Definition	
  
Exchange
I	
  omen	
  sound	
  like	
  a	
  broken	
  record	
  
•  Defining	
  what	
  is	
  to	
  be	
  counted	
  =	
  standards	...
 
Phase	
  1	
  Mee6ngs	
  
October	
  9,	
  2013	
  	
  -­‐	
  San	
  Francisco,	
  CA	
  
December	
  11,	
  2013	
  -­‐...
Mee&ng	
  Lightning	
  Talks	
  
•  Expecta&ons	
  of	
  researchers	
  
•  Exploring	
  disciplinary	
  differences	
  in	...
March	
  30,	
  2015	
   27	
  
March	
  30,	
  2015	
   28	
  
March	
  30,	
  2015	
   29	
  
30	
  One-­‐on-­‐One	
  Interviews	
  
March	
  30,	
  2015	
   30	
  
Lots	
  and	
  lots	
  of	
  ideas	
  
March	
  30,	
  2015	
   31	
  
Alterna6ve	
  Assessment	
  
Ini6a6ve	
  	
  
	
  
Phase	
  2	
  
Presenta6ons	
  of	
  Phase	
  1	
  report	
  (June	
  2...
White	
  Paper	
  Released	
  
March	
  30,	
  2015	
   33	
  
200	
  +	
  Ideas	
  	
  
25	
  Poten6al	
  Work	
  Themes	
  
Defini6ons	
  
Applica6on	
  to	
  types	
  of	
  research	
...
March	
  30,	
  2015	
   35	
  
0%	
  
10%	
  
20%	
  
30%	
  
40%	
  
50%	
  
60%	
  
70%	
  
80%	
  
90%	
  
100%	
  
Un...
March	
  30,	
  2015	
   36	
  
0%	
  
10%	
  
20%	
  
30%	
  
40%	
  
50%	
  
60%	
  
70%	
  
80%	
  
90%	
  
100%	
  
Un...
Top-­‐ranked	
  ideas	
  	
  
(very	
  important	
  &	
  important	
  >70%)	
  	
  
•  Develop	
  specific	
  defini&ons	
  ...
Launching	
  three	
  working	
  groups	
  
March	
  30,	
  2015	
   38	
  
A)	
  Development	
  of	
  defini&ons	
  and	
 ...
Steering Committee
Project Site:
www.niso.org/topics/tl/altmetrics_initiative/
White Paper:
http://www.niso.org/apps/group...
Alterna6ve	
  Assessments	
  of	
  our	
  
Assessment	
  Ini6a6ve	
  	
  
	
  
	
  
	
  
White	
  paper	
  downloaded	
  7...
For more
Project Site:
www.niso.org/topics/tl/altmetrics_initiative/
White Paper:
http://www.niso.org/apps/group_public/do...
Questions?
Todd Carpenter
Executive Director
tcarpenter@niso.org
@TAC_NISO
National Information Standards Organization (NI...
Carpenter: Lets remove alt from altmetrics - UKSG Plenary
Carpenter: Lets remove alt from altmetrics - UKSG Plenary
Upcoming SlideShare
Loading in …5
×

Carpenter: Lets remove alt from altmetrics - UKSG Plenary

2,235 views

Published on

Todd Carpenter's plenary presentation during the UKSG conference in Glasgow, UK on March 30, 2015. Todd discussed the development of new forms of assessment, the need to build trust in metrics and that the community should stop considering them "alternative".

Published in: Technology

Carpenter: Lets remove alt from altmetrics - UKSG Plenary

  1. 1. “Alt”  is  German  for  “old”   It’s  &me  to  stop  talking     about  metrics  in  terms  of   “alterna&ve”  metrics.   Todd A. Carpenter   Executive Director, NISO @TAC_NISO March 30, 2015
  2. 2. !  Non-­‐profit  industry  trade  associa&on     accredited  by  the  American  Na&onal  Standards  Ins&tute   !  Mission  of  developing  and  maintaining  technical   standards  related  to  informa&on,  documenta&on,   discovery  and  distribu&on  of  published  materials  and   media   !  Volunteer  driven  organiza&on:  400+  contributors  spread   out  across  the  world   !  Responsible  (directly  and  indirectly)  for  standards  like   ISSN,  DOI,  Dublin  Core  metadata,  DAISY  digital  talking   books,  OpenURL,  MARC  records,  and  ISBN   About     March  30,  2015   2  
  3. 3. 1980s Alternative   John  Taylor,   Duran  Duran  
  4. 4. 1920s Alternative  
  5. 5. 2010s Not-so Alternative  
  6. 6. Since  Jason  Priem  coined  the  term   “altmetrics”   •  There  have  been  some  1,170  ar&cles  and   posts  (per  Google  Scholar)  on  the  topic  of   altmetrics   •  There  have  been  eight  altmetrics  conferences   •  There  are  now  at  least  four  companies   focused  on  providing  altmetrics-­‐style  analy&cs   March  30,  2015   6  
  7. 7. When it becomes a standard, its not alterative any more…  
  8. 8. But  what’s  the  value  in   coun&ng  Tweets  and   Facebook  likes?   March  30,  2015   8  
  9. 9. More  than  just  popularity   Research  is  poin&ng  to  the  fact  that  there  is  a   modest  posi&ve  correla&on  between     early-­‐signal  metrics  (altmetrics)  and     later-­‐signal  metrics  (cita&ons)   Do  altmetrics  correlate  with  cita6ons?  Extensive  comparison  of  altmetric  indicators  with   cita6ons  from  a  mul6disciplinary  perspec6ve  by  Rodrigo  Costas,  Zohreh  Zahedi,  Paul  Wouters     Do  Altmetrics  Work?  TwiBer  and  Ten  Other  Social  Web  Services  by  Mike  Thelwall  ,  Stefanie   Haustein,    Vincent  Larivière,  &  Cassidy  R.  Sugimoto       (2006)  Earlier  web  usage  sta&s&cs  as  predictors  of  later  cita&on  impact  by  Brody  T,  Harnad  S  &   Carr  L       March  30,  2015   9  
  10. 10. More  importantly:  Scholarship  today     is  much  more  than  journals   March  30,  2015   10  
  11. 11. How  do  we  measure  the   impact  of  these  different   forms  of  output?   March  30,  2015   11  
  12. 12. Would  a  researcher  focus  on  only  one  data   source  or  methodological  approach?     March  30,  2015   12  Duke  University  -­‐    Informa&on  Ini&a&ve  at  Duke  (IID)  
  13. 13. There  aren’t     metrics  and  “altmetrics”;   there  are  only  metrics!     March  30,  2015   13  
  14. 14. We  have  been  using     non-­‐cita&on-­‐based   metrics  for  decades   March  30,  2015   14  
  15. 15. March  30,  2015   15  
  16. 16. March  30,  2015   16   Traverse  Area  District  Library  (TADL)  
  17. 17. What  has  changed  is  our   ability  to     collect  and  analyze  data     in  new  ways  that  are   meaningful  to  others   March  30,  2015   17  
  18. 18. What are we lacking?   =  
  19. 19. TRUST   =  
  20. 20. What are the elements of metrics that we need to build trust?  
  21. 21. Granularity   Time Scale   Identification  Definition   Exchange
  22. 22. I  omen  sound  like  a  broken  record   •  Defining  what  is  to  be  counted  =  standards   •  How  to  describe  what  to  count  =  standards   •  Iden&fica&on  of  what  to  count  =  standards   •  Procedures  for  coun&ng  or  not  =  standards   •  Aggrega&ng  counts  from  network  =  standards   •  Exchange  of  what  was  counted  =  standards     March  30,  2015   22  
  23. 23.   Phase  1  Mee6ngs   October  9,  2013    -­‐  San  Francisco,  CA   December  11,  2013  -­‐  Washington,  DC   January  23-­‐24  -­‐  Philadelphia,  PA   Round  of  1-­‐on-­‐1  interviews  –  March/Apr     Phase  1  report  published  in  June  2014  
  24. 24. Mee&ng  Lightning  Talks   •  Expecta&ons  of  researchers   •  Exploring  disciplinary  differences  in  the  use  of  social  media  in   scholarly  communica&on   •  Altmetrics  as  part  of  the  services  of  a  large  university  library   system   •  Deriving  altmetrics  from  annota&on  ac&vity   •  Altmetrics  for  Ins&tu&onal  Repositories:  Are  the  metadata   ready?   •  Snowball  Metrics:  Global  Standards  for  Ins&tu&onal   Benchmarking   •  Interna&onal  Standard  Name  Iden&fier   •  Altmetric.com,  Plum  Analy&cs,  Mendeley  reader  survey   •  Twiper  Inconsistency   March  30,  2015   26   “Lightning"  by  snowpeak  is  licensed  under  CC  BY  2.0  
  25. 25. March  30,  2015   27  
  26. 26. March  30,  2015   28  
  27. 27. March  30,  2015   29  
  28. 28. 30  One-­‐on-­‐One  Interviews   March  30,  2015   30  
  29. 29. Lots  and  lots  of  ideas   March  30,  2015   31  
  30. 30. Alterna6ve  Assessment   Ini6a6ve       Phase  2   Presenta6ons  of  Phase  1  report  (June  2014)   Priori6za6on  Effort  (June  -­‐  Aug,  2014)   Project  approval  (Nov  -­‐  Dec  2014)   Working  group  forma6on  (Jan  -­‐  March  2015)   Consensus  Development  (March  2015  -­‐  Feb  2016)   Trial  Use  Period  (Feb  15  -­‐  May  16)   Publica6on  of  final  recommenda6ons  (Aug  16)  
  31. 31. White  Paper  Released   March  30,  2015   33  
  32. 32. 200  +  Ideas     25  Poten6al  Work  Themes   Defini6ons   Applica6on  to  types  of  research  outputs   Discovery  implica6ons   Research  evalua6on   Data  quality  and  gaming   Grouping,  aggrega6ng,  and  granularity   Context     Adop6on   March  30,  2015   34  
  33. 33. March  30,  2015   35   0%   10%   20%   30%   40%   50%   60%   70%   80%   90%   100%   Unimportant   Of  liple  importance   Moderately  important   Important   Very  important   Community  Feedback  on  Project  Idea  Themes   n=118  
  34. 34. March  30,  2015   36   0%   10%   20%   30%   40%   50%   60%   70%   80%   90%   100%   Unimportant   Of  liple  importance   Moderately  important   Important   Very  important   n=118   Community  Feedback  on  Project  Idea  Themes  
  35. 35. Top-­‐ranked  ideas     (very  important  &  important  >70%)     •  Develop  specific  defini&ons  for  alterna&ve  assessment  metrics.   (87.9%)       •  Promote  and  facilitate  use  of  persistent  iden&fiers  in  scholarly   communica&ons.  (82.8%)   •  Develop  strategies  to  improve  data  quality  through  normaliza&on   of  source  data  across  providers.  (80.8%)   •  Iden&fy  research  output  types  that  are  applicable  to  the  use  of   metrics.  (79.8%)   •  Define  appropriate  metrics  and  calcula&on  methodologies  for   specific  output  types,  such  as  somware,  datasets,  or  performances.   (78.1%)   •  Explore  crea&on  of  standardized  APIs  or  download  or  exchange   formats  to  facilitate  data  gathering.  (72.5%)   •  Research  issues  surrounding  the  reproducibility  of  metrics  across   providers.  (70.7%)   March  30,  2015   37  
  36. 36. Launching  three  working  groups   March  30,  2015   38   A)  Development  of  defini&ons  and  descrip&ons  of  use   B)  Defini&ons  for  appropriate  metrics  and  calcula&on   methodologies  for  non-­‐tradi&onal  output  types.  Also   work  toward  promo&on  and  facilita&on  of  use  of   persistent  iden&fiers   C)  Development  of  strategies  to  improve  data  quality   through  source  data  providers  
  37. 37. Steering Committee Project Site: www.niso.org/topics/tl/altmetrics_initiative/ White Paper: http://www.niso.org/apps/group_public/download.php/ 13295/niso_altmetrics_white_paper_draft_v4.pdf March  30,  2015   39  
  38. 38. Alterna6ve  Assessments  of  our   Assessment  Ini6a6ve           White  paper  downloaded  7,400  6mes     21  substan&ve  comments  received   120  in-­‐person  and  virtual  par&cipants  at  the  mee&ngs     These  3  mee&ngs  apracted  >400  RSVPs  for  live  stream       Goal:  generate  about  40  ideas,  in  total,  generated  more  than  250   Project  materials  downloaded  more  than  22,000  &mes   More  than  550  direct  tweets  using  the  #NISOALMI  hashtag   Survey  ranking  of  output  by  118  people     Six  ar&cles  in  tradi&onal  news  publica&ons     15  blog  posts  about  the  ini&a&ve  
  39. 39. For more Project Site: www.niso.org/topics/tl/altmetrics_initiative/ White Paper: http://www.niso.org/apps/group_public/download.php/ 13295/niso_altmetrics_white_paper_draft_v4.pdf March  30,  2015   41  
  40. 40. Questions? Todd Carpenter Executive Director tcarpenter@niso.org @TAC_NISO National Information Standards Organization (NISO) 3600 Clipper Mill Road, Suite 302 Baltimore, MD 21211 USA +1 (301) 654-2512 www.niso.org March  30,  2015   42  

×