Getting the Most Out of Your E-Resources: Measuring Success Todd Carpenter Managing Director, NISO
Where are we headed this afternoon? <ul><li>A bit about NISO </li></ul><ul><li>Overview of usage measurement of    e-resou...
What is NISO? <ul><li>NISO - National Information Standards Organization </li></ul><ul><li>NISO is the only ANSI-accredite...
Big Challenges, Modest Resources <ul><li>Revenue: $900K, up 20% in 2007 </li></ul><ul><li>Primary income: Member dues (60%...
Standards – Why should I care? <ul><li>Standards accelerate production, ordering/sales, dissemination, locating, storing a...
Meaningless? <ul><li>Certainly, there’s a lot of data </li></ul><ul><li>The difference between meaningless and meaningful ...
The Early Days of E-Resources <ul><li>Thinking back to 1998 - Many open questions </li></ul><ul><li>How do you record traf...
Slow development of consensus <ul><li>ICOLC - Guidelines for Statistical Measurement of Usage of Web-Based Information Res...
Toward Formalization <ul><li>ANSI/NISO Z39.7: 2004 - Information Services and Use Metrics & Statistics for Libraries and I...
Project COUNTER <ul><li>COUNTER (Counting Online Usage of NeTworked Electronic Resources) </li></ul><ul><li>Formed in 2002...
COUNTER Codes of Practice <ul><li>Definitions </li></ul><ul><li>Specifications for Usage Reports </li></ul><ul><ul><li>Wha...
COUNTER: Current Codes of Practice <ul><li>1) Journals and databases </li></ul><ul><ul><li>Release 1 Code of Practice laun...
Journal  Report 1 Example
Need for easier access to usage data <ul><li>As a community, we need to reduce the time and effort necessary to collect, f...
One example of the need <ul><li>One university with more than 75 online resources from which they draw usage data </li></u...
Briefly:  SUSHI <ul><li>Need:  Simplify and automate the gathering of usage data for librarians </li></ul><ul><ul><li>Libr...
Content Provider Library SUSHI Server Usage Data SUSHI Client Internet ERM SUSHI is a Web Service which sends an XML reque...
SUSHI: Where now, where to? <ul><li>Passed unanimously by NISO membership in September, 2007 </li></ul><ul><li>Formally ap...
Careful the conclusions you draw
What is the most used resource? <ul><li>BioOne’s most viewed article </li></ul><ul><li>The nest architecture of the Florid...
Link Prefectching affecting data? <ul><li>Link prefetching is a browser mechanism, which utilizes browser idle time to dow...
User Interface Issues Affecting  Data? <ul><li>How a publisher system is designed could affect reported usage </li></ul><u...
Metasearch affecting data? <ul><li>Metasearch engines conduct multiple searches simultaneously </li></ul><ul><li>Retrieve,...
Applying Usage Data Data  -  All  facts Information  - Facts within  context Knowledge  -  Interrelationships  among relev...
Basic Measures <ul><li>Cost-per use - Are we getting comparative value from this resource? </li></ul><ul><li>Are my system...
Using Usage as a Quality Measure <ul><li>The amount of traffic an item receives is separate, but valuable, metric for asse...
Comparing usage measures <ul><li>Development project underway within COUNTER in partnership with UKSG </li></ul><ul><li>Go...
Two methods for assessing quality <ul><li>Impact Factor </li></ul><ul><ul><li>Established, understood and generally accept...
What’s in store in the future <ul><li>MESUR - MEtrics from Scholarly Usage of Resources </li></ul><ul><li>Mellon funded pr...
MESUR - Some Examples <ul><li>Discerning methods of citation networks </li></ul><ul><li>Describing journal usage compariso...
Privacy Concerns <ul><li>If you have enough data, you can pinpoint exact people </li></ul><ul><li>Say you have domain expe...
Other Areas for Development <ul><li>Non-Journal content, A&I, bibliographies </li></ul><ul><li>Content on multiple platfor...
Thank you! <ul><li>Todd Carpenter, Managing Director </li></ul><ul><li>[email_address] </li></ul><ul><ul><li>One North Cha...
Upcoming SlideShare
Loading in...5
×

Getting the Most Out of Your E-Resources: Measuring Success

1,426

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,426
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Getting the Most Out of Your E-Resources: Measuring Success

    1. 1. Getting the Most Out of Your E-Resources: Measuring Success Todd Carpenter Managing Director, NISO
    2. 2. Where are we headed this afternoon? <ul><li>A bit about NISO </li></ul><ul><li>Overview of usage measurement of e-resources </li></ul><ul><li>COUNTER & SUSHI </li></ul><ul><li>The application of usage data </li></ul><ul><li>Issues and concerns with use data </li></ul><ul><li>A glimpse into the future </li></ul>
    3. 3. What is NISO? <ul><li>NISO - National Information Standards Organization </li></ul><ul><li>NISO is the only ANSI-accredited organization tasked with the development of standards in the field of Information and Documentation </li></ul><ul><li>Work with publishers, libraries, agents and other systems vendors to develop community consensus </li></ul><ul><li>Develop wide range of standards </li></ul><ul><ul><li>Paper permanence and steal shelving </li></ul></ul><ul><ul><li>Accessibility issues </li></ul></ul><ul><ul><li>Bibliographic formats and exchange </li></ul></ul><ul><ul><li>Web-based delivery, OpenURL, Metasearch, SUSHI </li></ul></ul>
    4. 4. Big Challenges, Modest Resources <ul><li>Revenue: $900K, up 20% in 2007 </li></ul><ul><li>Primary income: Member dues (60%) </li></ul><ul><li>Other income: Seminars, Publishing (20%) </li></ul><ul><li>New sources of revenue in 2007 - Grants </li></ul><ul><ul><li>Mellon $196K, IMLS - $24K (20%) </li></ul></ul><ul><li>Staff: 4 Professional full-time </li></ul><ul><li>Virtual staff: 10+ (Consultants, Partners) </li></ul><ul><li>83 Voting Members, 25 LSA members as of 2007 </li></ul><ul><li>Maintenance Agencies: 12 </li></ul><ul><li>Volunteers: 300+ spread out across the world </li></ul>
    5. 5. Standards – Why should I care? <ul><li>Standards accelerate production, ordering/sales, dissemination, locating, storing and preserving information </li></ul><ul><li>Key standards which NISO has developed and helping to bring consensus around </li></ul><ul><ul><li>ISSN, OpenURL, Z39.50, NCIP </li></ul></ul><ul><ul><li>In development: DOI, SUSHI, SERU, LEWG </li></ul></ul><ul><ul><li>In planning: Institutional ID, Performance Measures, OpenURL Expansion, </li></ul></ul>
    6. 6. Meaningless? <ul><li>Certainly, there’s a lot of data </li></ul><ul><li>The difference between meaningless and meaningful data is APPLICATION </li></ul>
    7. 7. The Early Days of E-Resources <ul><li>Thinking back to 1998 - Many open questions </li></ul><ul><li>How do you record traffic? </li></ul><ul><li>What is a hit? </li></ul><ul><li>Is a hit different than a download? </li></ul><ul><li>What about reloading? </li></ul><ul><li>What about images and links? </li></ul><ul><li>What should a report include? </li></ul><ul><li>Eventually, counting different versions of texts </li></ul><ul><li>How often do you need to provide stats </li></ul><ul><li>Traffic to abstracts, TOCs or other elements? </li></ul><ul><li>Content over multiple pages? </li></ul>
    8. 8. Slow development of consensus <ul><li>ICOLC - Guidelines for Statistical Measurement of Usage of Web-Based Information Resources </li></ul><ul><li>Released in 1998, updated in 2000 </li></ul><ul><ul><li>Minimum Requirements– Data elements, timeframe, etc </li></ul></ul><ul><ul><li>Confidentiality </li></ul></ul><ul><ul><li>Access </li></ul></ul><ul><ul><li>Delivery </li></ul></ul><ul><ul><li>Definitions </li></ul></ul><ul><ul><li>Formats </li></ul></ul><ul><li>National Commission on Libraries and Information Science (NCLIS) Electronic Access and Use-Related Measures </li></ul><ul><li>Released in 2001 </li></ul>
    9. 9. Toward Formalization <ul><li>ANSI/NISO Z39.7: 2004 - Information Services and Use Metrics & Statistics for Libraries and Information Providers -- Data Dictionary </li></ul><ul><ul><li>ONLINE: www.niso.org/emetrics/current/index.html </li></ul></ul><ul><li>Technical Committee 46 - Information and Documentations, SC 8 - Statistics and Performance Indicators </li></ul><ul><ul><li>ISO 2789: 2006 Information and documentation -- International library statistics </li></ul></ul><ul><ul><li>ISO 11620: 1998 Information and documentation -- Library performance indicators </li></ul></ul>
    10. 10. Project COUNTER <ul><li>COUNTER (Counting Online Usage of NeTworked Electronic Resources) </li></ul><ul><li>Formed in 2002 </li></ul><ul><li>Membership organization (as of 3/15/08) </li></ul><ul><ul><li>Industry Organizations - 13 </li></ul></ul><ul><ul><li>Library Consortia - 62 </li></ul></ul><ul><ul><li>Libraries - 84 </li></ul></ul><ul><ul><li>Publishers - 66 </li></ul></ul><ul><li>Establishes Codes of Practice on the gathering, compiling and storage of publishing usage data </li></ul>
    11. 11. COUNTER Codes of Practice <ul><li>Definitions </li></ul><ul><li>Specifications for Usage Reports </li></ul><ul><ul><li>What they should include </li></ul></ul><ul><ul><li>What they should look like </li></ul></ul><ul><ul><li>How and when they should be delivered </li></ul></ul><ul><li>Data processing guidelines </li></ul><ul><li>Auditing (New in 2006) </li></ul><ul><li>Compliance </li></ul><ul><li>Maintenance and development of the Codes of Practice </li></ul><ul><li>Governance of COUNTER </li></ul>
    12. 12. COUNTER: Current Codes of Practice <ul><li>1) Journals and databases </li></ul><ul><ul><li>Release 1 Code of Practice launched January 2003 </li></ul></ul><ul><ul><li>Release 2 replaced Release 1 in January 2006 </li></ul></ul><ul><ul><li>Release 3 under consideration - to include SUSHI compliance and consortia reporting </li></ul></ul><ul><ul><li>Now a widely adopted standard by publishers and librarians </li></ul></ul><ul><ul><li>60%+ of Science Citation Index articles now covered </li></ul></ul><ul><li>2) Books and reference works </li></ul><ul><ul><li>Code of Practice for Books was launched March 2006 </li></ul></ul><ul><ul><li>Relevant usage metrics less clear than for journals </li></ul></ul><ul><ul><li>Different issues than for journals </li></ul></ul><ul><ul><ul><li>Direct comparisons between books less relevant </li></ul></ul></ul><ul><ul><ul><li>Understanding how different categories of book are used is more relevant </li></ul></ul></ul>
    13. 13. Journal Report 1 Example
    14. 14. Need for easier access to usage data <ul><li>As a community, we need to reduce the time and effort necessary to collect, format and compile usage data </li></ul><ul><li>“ Time for meaningful analysis is compromised by the time required just to gather and record the statistics.” </li></ul><ul><ul><li>Median percentage of time spent on analysis is only 25 percent </li></ul></ul><ul><ul><li>More than half of the time is spent on gathering and formatting </li></ul></ul><ul><ul><li>Average number of hours spent working on usage data is 96 hours, but ranged on the high end up to 1-2 FTEs entirely focused on data </li></ul></ul><ul><ul><li>Usage reports to help them make subscription decisions (94%) and justify expenditures (86%) for their electronic resources </li></ul></ul><ul><ul><li>DATA FROM: Gayle Baker, Eleanor J. Read, Vendor Usage Data for Electronic Resources: A Survey of Libraries http://smartech.gatech.edu/handle/1853/13611 </li></ul></ul>
    15. 15. One example of the need <ul><li>One university with more than 75 online resources from which they draw usage data </li></ul><ul><li>They have a 80-page booklet containing the details of how to access and gather usage data! </li></ul><ul><li>How much time does it take not just compiling and maintaining this notebook, but even just going through it? </li></ul>
    16. 16. Briefly: SUSHI <ul><li>Need: Simplify and automate the gathering of usage data for librarians </li></ul><ul><ul><li>Librarians spending months gathering data </li></ul></ul><ul><li>Solution </li></ul><ul><ul><li>Server/Client system to exchange COUNTER reports </li></ul></ul><ul><ul><li>Easily incorporated into usage systems (on publisher side) or into ERM (on library side) </li></ul></ul><ul><ul><li>Client calls to server, asks for report, and server runs the report and sends it on </li></ul></ul><ul><ul><li>Data exchange is taking place by machine talking with machine </li></ul></ul>
    17. 17. Content Provider Library SUSHI Server Usage Data SUSHI Client Internet ERM SUSHI is a Web Service which sends an XML request to a content provider to obtain an XML response containing the usage report. ? Response COUNTER Request SOAP Slide courtesy of Oliver Pesch, EBSCO Information Services, Co-Chair SUSHI
    18. 18. SUSHI: Where now, where to? <ul><li>Passed unanimously by NISO membership in September, 2007 </li></ul><ul><li>Formally approved as ANSI/NISO Z39.93:2007 </li></ul><ul><li>Working toward broad ADOPTION </li></ul><ul><li>Ask that it be included in your ERM solution </li></ul><ul><li>Demand your content providers become SUSHI and COUNTER (rev 3) compliant </li></ul><ul><li>Talk to your vendors and your consortia </li></ul>
    19. 19. Careful the conclusions you draw
    20. 20. What is the most used resource? <ul><li>BioOne’s most viewed article </li></ul><ul><li>The nest architecture of the Florida harvester ant, Pogonomyrmex badius </li></ul><ul><li>Walter R. Tschinkel </li></ul><ul><li>“ Coolest images on the net” </li></ul>
    21. 21. Link Prefectching affecting data? <ul><li>Link prefetching is a browser mechanism, which utilizes browser idle time to download or prefetch documents that the user might visit in the near future. </li></ul><ul><li>Based on previous use data, the site provides a set of prefetching hints to the browser, and after the browser is finished loading the page, it begins silently prefetching specified documents and stores them in its cache. </li></ul><ul><li>When the user visits a prefetched document, it can be served up quickly out of the browser's cache. </li></ul><ul><ul><li>Source: Mozzila .org </li></ul></ul>
    22. 22. User Interface Issues Affecting Data? <ul><li>How a publisher system is designed could affect reported usage </li></ul><ul><ul><li>For example: </li></ul></ul><ul><ul><li>If you have to visit the HTML page to get the PDF </li></ul></ul><ul><ul><li>Source: Price & Davis, JASIST, 2006 arxiv.org/pdf/cs/0602060 </li></ul></ul>
    23. 23. Metasearch affecting data? <ul><li>Metasearch engines conduct multiple searches simultaneously </li></ul><ul><li>Retrieve, consolidate and ranks results based on algorithmic and semantic analysis </li></ul><ul><li>Provide users with resource selections </li></ul><ul><li>However, in many cases this search and retrieve results in hits and downloads </li></ul>
    24. 24. Applying Usage Data Data - All facts Information - Facts within context Knowledge - Interrelationships among relevant facts Wisdom - Actionable knowledge
    25. 25. Basic Measures <ul><li>Cost-per use - Are we getting comparative value from this resource? </li></ul><ul><li>Are my systems working? </li></ul><ul><li>Are there barriers to use - why are similar products experiencing different use patterns? </li></ul><ul><li>Expressing value to administration, contributors or government sponsors </li></ul>
    26. 26. Using Usage as a Quality Measure <ul><li>The amount of traffic an item receives is separate, but valuable, metric for assessing quality </li></ul><ul><li>Citation measures capture only one type of use - scholarly citation, not necessarily quality </li></ul><ul><li>Teaching or clinical use is extremely valuable </li></ul>
    27. 27. Comparing usage measures <ul><li>Development project underway within COUNTER in partnership with UKSG </li></ul><ul><li>Goal: Derive a meaningful calculation for assessing quality through COUNTER data </li></ul><ul><li>Usage of items / Period of time </li></ul><ul><li>Questions primarily related to the denominator used in calculation </li></ul>
    28. 28. Two methods for assessing quality <ul><li>Impact Factor </li></ul><ul><ul><li>Established, understood and generally accepted </li></ul></ul><ul><ul><li>Funding agencies, researchers rely on its data </li></ul></ul><ul><ul><li>Limitation in the fields of scholarship it covers </li></ul></ul><ul><ul><li>Reflects value of journals to researchers, but not all users </li></ul></ul><ul><ul><li>Over-emphasis on IF distorts the behaviour of authors </li></ul></ul><ul><ul><li>Over-used, mis-used and over-interpreted </li></ul></ul><ul><li>Usage Factor </li></ul><ul><ul><li>Usage-based alternative perspective </li></ul></ul><ul><ul><li>Would cover all online journals </li></ul></ul><ul><ul><li>Would reflect value of journals to all categories of user </li></ul></ul><ul><ul><li>Would be easy to understood </li></ul></ul>
    29. 29. What’s in store in the future <ul><li>MESUR - MEtrics from Scholarly Usage of Resources </li></ul><ul><li>Mellon funded project to study assessment of the impact of scholarly communication items, and hence of scholars, with metrics that derive from usage data </li></ul>Data analysis imaese from Johan Bollen
    30. 30. MESUR - Some Examples <ul><li>Discerning methods of citation networks </li></ul><ul><li>Describing journal usage comparisons </li></ul><ul><li>Describing potential connectedness measures </li></ul><ul><li>Relatedness of items based on use patterns </li></ul><ul><li>“ Readers who viewed this also views…” </li></ul>Image source: www.mesur.org
    31. 31. Privacy Concerns <ul><li>If you have enough data, you can pinpoint exact people </li></ul><ul><li>Say you have domain expertise </li></ul><ul><li>You see person X looks at this article, then that article, and onto this series of articles </li></ul><ul><li>You’ll probably be able to figure who the person is and what they’re working on </li></ul>
    32. 32. Other Areas for Development <ul><li>Non-Journal content, A&I, bibliographies </li></ul><ul><li>Content on multiple platforms </li></ul><ul><li>Different versions existing on the network </li></ul><ul><li>Institutional repository systems </li></ul><ul><ul><li>Limited usage tracking and consistent reporting </li></ul></ul><ul><li>Multimedia content, streaming content </li></ul><ul><li>Mash-ups and multi-feed content </li></ul><ul><li>Research data and visualization tools </li></ul>
    33. 33. Thank you! <ul><li>Todd Carpenter, Managing Director </li></ul><ul><li>[email_address] </li></ul><ul><ul><li>One North Charles Street </li></ul></ul><ul><ul><li>Suite 1905 </li></ul></ul><ul><ul><li>Baltimore, MD 21201 USA </li></ul></ul><ul><ul><li>(301) 654-2512 </li></ul></ul><ul><ul><li>(410) 685-5278 </li></ul></ul><ul><ul><li>www.niso.org </li></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×