• Save
NISO Update June 2014 Assessment Carpenter
 

NISO Update June 2014 Assessment Carpenter

on

  • 312 views

The NISO Update provides the latest news about NISO's current efforts, including standards, recommended practices and community meetings covering many areas of interest to the library community. ...

The NISO Update provides the latest news about NISO's current efforts, including standards, recommended practices and community meetings covering many areas of interest to the library community. Working group members will provide updates on projects newly underway or recently completed.

NISO Altmetrics Initiative, Todd Carpenter, Executive Director, NISO

Statistics

Views

Total Views
312
Views on SlideShare
253
Embed Views
59

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 59

http://www.niso.org 58
http://www.google.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

NISO Update June 2014 Assessment Carpenter NISO Update June 2014 Assessment Carpenter Presentation Transcript

  • Comparing digital apples to digital apples: An update on NISO’s Alternative Assessment Initiative Todd Carpenter Executive Director, NISO ALA Annual June 30, 2014June 30, 2014 1
  • 2
  • What are the infrastructure elements of alternative assessments? June 30, 2014 3
  • Basic Definitions (So we are all talking about the same thing) June 30, 2014 4
  • June 30, 2014 5
  • Element Identification June 30, 2014 6
  • At what granularity? June 30, 2014 7
  • How long do we measure? June 30, 2014 8
  • June 30, 2014 9
  • Consistency across providers Source: Scott Chamberlain, Consuming Article-Level Metrics: Observations And Lessons From Comparing Aggregator Provider Data, Information Standards Quarterly, Summer 2013, Vol 25, Issue 2.June 30, 2014 10
  • TRUST = June 30, 2014 11
  • What is NISO working toward?June 30, 2014 12
  • June 30, 2014 13
  • June 30, 2014 14
  • Steering Committee • Euan Adie, Altmetric • Amy Brand, Harvard University • Mike Buschman, Plum Analytics • Todd Carpenter, NISO • Martin Fenner, Public Library of Science (PLoS) (Chair) • Michael Habib, Reed Elsevier • Gregg Gordon, Social Science Research Network (SSRN) • William Gunn, Mendeley • Nettie Lagace, NISO • Jamie Liu, American Chemical Society (ACS) • Heather Piwowar, ImpactStory • John Sack, HighWire Press • Peter Shepherd, Project Counter • Christine Stohn, Ex Libris • Greg Tananbaum, SPARC (Scholarly Publishing & Academic Resources Coalition) June 30, 2014 15
  • Alternative Assessment Initiative Phase 1 Meetings October 9, 2013 - San Francisco, CA December 11, 2013 - Washington, DC January 23-24 - Philadelphia, PA Round of 1-on-1 interviews – March/Apr Phase 1 report published June 9, 2014 June 30, 2014 16
  • Meetings’ General Format • Collocated with other industry meeting • Morning: lightning talks, post-it brainstorming • Afternoon: discussion groups – X – Y – Z – Report back/react • Live streamed (video recordings are available) June 30, 2014 17
  • Meeting Lightning Talks • Expectations of researchers • Exploring disciplinary differences in the use of social media in scholarly communication • Altmetrics as part of the services of a large university library system • Deriving altmetrics from annotation activity • Altmetrics for Institutional Repositories: Are the metadata ready? • Snowball Metrics: Global Standards for Institutional Benchmarking • International Standard Name Identifier • Altmetric.com, Plum Analytics, Mendeley reader survey • Twitter Inconsistency June 30, 2014 18 “Lightning" by snowpeak is licensed under CC BY 2.0
  • June 30, 2014 19
  • SF Meeting Discussions • Business & Use cases – Publishers want to serve authors, make money – People don’t value a standard, they value something that helps them – … Couldn’t identify a logical standard need that actors in the space would value, and best practices are of interest • Quality & Data science – Themes: context, validation, provenance, quality, description & metadata – We'll never get to the point where assessment can be done without a human in the loop, but discovery and recommendation can • Definitions – Define “ALM” and “Altmetrics” – Map the landscape – We'll never get to the point where assessment can be done without a human in the loop, but discovery and recommendation can June 30, 2014 20
  • DC Meeting Discussions • Business and Use Cases • Discovery – metrics only get generated if material is discovered • Qualitative vs. Quantitative • Identifying Stakeholders and their Values – stakeholders in outcomes / stakeholders in process of creating metrics – shared values but tensions – branding • Definitions/Defining Impact – metrics and analyses – what led to success of citation? – how to be certain we are measuring the right things • Future Proofing – what won't change – impact - hard to establish across disciplines June 30, 2014 21
  • Philly Meeting Discussions • Definitions – Define life cycle of scholarly output and associated metrics – Qualitative versus Quantitative aspects - what is possible to define here – Consider other aspects of these data collections • Standards – Develop definitions (what is a download? what is a view?) – Differentiate between scholarly impact versus popular/social use – Define sources/characteristics for metrics (social, commercial, scholarly) • Data Integrity – Counter biases/gaming – Association with credible entities - e.g. ORCID ID v. gmail account – Reproduceability is key – Everyone needs to be at the table to establish overall credibility • Use cases (3X) June 30, 2014 22
  • 30 One-on-One Interviews June 30, 2014 23
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption June 30, 2014 24
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption June 30, 2014 25
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption June 30, 2014 26
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption June 30, 2014 27
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption June 30, 2014 28
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption June 30, 2014 29
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption June 30, 2014 30
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption June 30, 2014 31
  • Potential work themes Definitions Application to types of research outputs Discovery implications Research evaluation Data quality and gaming Grouping, aggregating, and granularity Context Adoption & Promotion June 30, 2014 32
  • Alternative Assessment Initiative Phase 2 Presentations of Phase 1 report (June 2014) Prioritization Effort (June - Aug, 2014) Project approval (Sept 2014) Working group formation (Oct 2014) Consensus Development (Nov 2014 - Dec 2015) Trial Use Period (Dec 15 - Mar 16) Publication of final recommendations (Jun 16) June 30, 2014 33
  • We all want our own June 30, 2014 34
  • For more information: www.niso.org/topics/tl/altmetrics_initiative/ June 30, 2014 35
  • Questions? Todd Carpenter Executive Director tcarpenter@niso.org National Information Standards Organization (NISO) 3600 Clipper Mill Road, Suite 302 Baltimore, MD 21211 USA +1 (301) 654-2512 www.niso.org June 30, 2014 36