Beyond COUNTER Compliant: Ways to Assess E-Resources Reporting Tools
•Download as PPTX, PDF•
1 like•468 views
Kelly Marie Blanchat, presenter The need to continually evaluate electronic resources should not limited to a metric for how resources perform. The reporting tools that monitor and collect e-resource usage need to have their performance evaluated as well. This presentation will cover how vendor-provided systems -- designed to aid in the decision making process of the e-resources lifecycle -- can be assessed for reporting accuracy. Following this session, participants will have an understanding of what data points to review when assessing vendor-provided usage statistic tools, and will have a method to begin evaluating their own systems. In summer 2015, Yale Library brought up ProQuest’s 360 COUNTER Data Retrieval Service (DRS), a service in which COUNTER-compliant usage statistics are uploaded, archived, and normalized into consolidated reports twice per year. To date 360 COUNTER has freed up a significant amount of time for Yale's E-Resources Group, allowing for staff resources to be allocated elsewhere in the e-resources lifecycle. This extra staff time also allowed time to “kick the tires” of the system, which resulted in an assessment workflow using Microsoft Excel to compare how raw COUNTER data uploaded to the system was affected by title normalization in the knowledgebase. This assessment workflow helped to identify the volume of data available in the system, and also gave clarity to how the 360 COUNTER system works and what steps need to be taken–by both ProQuest and Yale Library–to improve reporting accuracy. Please note that this presentation will touch on issues found within the system, and how ProQuest worked with Yale to identify the source through title normalization decisions, and correct errors when possible. The primary purpose is to bring awareness for the need of reporting tool assessment, which can be applied to any assessment tool, not just 360 COUNTER.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Beyond COUNTER Compliant: Ways to Assess E-Resources Reporting Tools
Similar to Beyond COUNTER Compliant: Ways to Assess E-Resources Reporting Tools (20)
More from NASIG
More from NASIG (20)
Recently uploaded
Recently uploaded (20)
Beyond COUNTER Compliant: Ways to Assess E-Resources Reporting Tools
- 1. BEYOND COUNTER-COMPLIANT THE IMPORTANCE OF ASSESSING E-RESOURCES REPORTING TOOLS University Library Kelly Blanchat Electronic Resources Support Librarian Excel Handout: https://tinyurl.com/y7bvlg27
- 2. 360 COUNTER @ YALE Out-source usage statistics harvesting To allocate staff resources elsewhere Consolidate usage across time and providers To enhance reporting NOT to calculate Cost-Per-Use, not using SUSHI
- 3. 360 COUNTER: HOW IT WORKS Part 2: INTOTA ASSESSMENT A. Consolidates usage • Across time • Across providers B. Assessment (i.e.: CPU) Part 1: 360 COUNTER A. Collection of usage statistics • “Data Retrieval Service” • SUSHI B. Archives usage data Jan Feb 0 0 1 42 3 7 89 0 Jan Feb 2 4 16 0 0 0 1 0 Jan Feb 2 0 7 6 0 0 26 87 Jan Feb 23 7 0 0 1 1 0 9 Jan Feb 27 11 32 48 4 8 106 96
- 4. IMPLEMENTATION TIMELINE Prior to 2015 Spring / Summer 2015 Administrative credentials added to 360 COUNTER for Data Retrieval Service (DRS) YUL staff manually uploads historical statistics to 360 COUNTER YUL staff manually retrieves usage statistics every 6 months YUL staff uploads statistics to a local website The DRS team finalizes the first retrieval of Yale’s usage statistics, for January – June 2015 September 2015 November 2015 Collection Development retrieves usage statistics for 2015 OUP BR2 within IA Oxford English Dictionary is MISSING from consolidated report, as are 342 titles, and 29,170 total uses October 2015 Begin exploring the consolidated system Open tickets for data corrections Pull sample reports from Intota Assessment (IA)
- 5. …WHAT DID THAT SAY? Oxford English Dictionary is MISSING from consolidated report, as are 342 other titles, and 29,170 total uses November 2015
- 7. PHASE 1: TITLE-LEVEL ANALYSIS Excel V-LOOKUP on ISSN/ISBN and title between COUNTER report and consolidated report
- 8. PHASE 1: SAMPLE FINDINGS 360 COUNTER Missing Titles (20 total titles): • A Dictionary of Geography • The Oxford Dictionary of Plays Variant Usage (34 total titles): • A Dictionary of Economics • The Oxford Classical Dictionary Variant ISBN (63 total titles): • Between Two Empires: Race, History, and Transnationalism in Japanese America INTOTA ASSESSMENT Duplicate entries / editions scrubbed Usage from distinct editions merged ISBN changed from raw COUNTER 2015 Oxford BR2 REASON REASON REASON
- 9. PHASE 1: OUTCOME Title-level example errors submitted to ProQuest for the 2015 Oxford BR2. After response and fix, only 2 titles and 20 uses remained outstanding.
- 10. PHASE 1: SUCCESSFUL, NOT SUSTAINABLE Favorable results! Time consuming and really not fun at all Yale has over 100 providers in 360 COUNTER
- 11. PHASE 2: SIMPLIFIED TO COLLECT TOTALS Collection Date: June 2016 VERY SIMPLE SUBTRACT FORMULA BUILT-IN
- 12. PHASE 2: IN PRACTICE CONSTANT: REPORTS VARIABLE: TIME RESULT: ACCURACY, OVER TIME
- 13. High-level data collection from 8 content providers for JR1 & BR2 submitted to ProQuest’s 360 COUNTER team. PHASE 2: OUTCOME 360 COUNTER INTOTA ASSESSMENT HOW Received granular information on HOW data is consolidated through normalization to the Authority Title. Jan Feb 0 0 1 42 3 7 89 0 Jan Feb 2 4 16 0 0 0 1 0 Jan Feb 2 0 7 6 0 0 26 87 Jan Feb 23 7 0 0 1 1 0 9 Jan Feb 27 11 32 48 4 8 106 96
- 14. CONSOLIDATION = NORMALIZATION Normalization to the Authority Title will affect overall title count because… • When duplicates have matching ISSN and title (i.e.: full match), usage is merged onto 1 entry • When a title has variant data points (DOI, ISSN) over time, titles may display multiple times • When duplicate titles have the same ISSN but distinct titles, usage is picked from 1 version
- 15. EXAMPLE: SAME ISSN, VARIANT NAMES 2015 Springer JR1: When duplicate titles have the same ISSN but distinct titles, usage is picked from 1 version
- 16. EVALUATION Yale has so much data. High-level analysis has helped us understand WHAT exactly what happening to our data better behind the scenes. And… it is complex because it is connected to the knowledgebase (and the knowledgebase is complicated).
- 17. 1) Do the “rules” for title normalization/consolidation make sense? • How does it affect potential CPU reporting? 2) Which results …. • should trigger a bug fix? • are a result of incorrect COUNTER data? • are a result of over title normalization? EVALUATION
- 18. EVALUATION: USE MY TEMPLATE! https://tinyurl.com/y7bvlg27
- 19. WHERE WE ARE IN 2017 Prior to 2015 Spring – Summer 2015 Administrative credentials added to 360 COUNTER for Data Retrieval Service (DRS) YUL staff manually uploads historical statistics to 360 COUNTER YUL staff manually retrieves usage statistics every 6 months YUL staff uploads statistics to a local website The DRS team finalizes the first retrieval of Yale’s usage statistics, for January – June 2015 OED is MISSING from consolidated report, as are 342 titles, and 29,170 total uses Sept – Nov 2015 April 2016Dec 2015 – March 2016 Phase 1 title-level analysis comparing raw COUNTER reports to consolidated reports in Intota Assessment Phase 2 begins with high- level analysis, gathering totals between reports August 2016 Lots of conference calls with 360 COUNTER Planning for Phase 3 begins, using ARL statistics as a pilot project; transforming COUNTER as data source in Tableau Lots more conference calls with 360 COUNTER January 2017 Pilot with ARL stats is complete, internal testing begins in Access, Tableau March 2017 Add more stats for ARL providers to Tableau Discuss more robust data solutions (MySQL, Python) June 2017 NASIG – HELLO! The Future….
- 20. YOU BET THERE’S A PHASE 3! August 2016 Planning for Phase 3 begins Transforming COUNTER reports as data source for Tableau Yay!
- 21. • We’re putting ourselves in charge, removing guess-work and assuming “burden” of accepting all data • Still leveraging the use of 360 COUNTER’s data retrieval – YAY! • More easily import ILS $$$ data into Tableau to merge with usage PHASE 3: “LET COUNTER BE COUNTER”
- 22. PHASE 3: COUNTER AS DATA SOURCE STANDARD COUNTER REPORT, JR1 COUNTER REPORT TRANSFORMED AS A DATA SOURCE WITH EXCEL TABLEAU PLUG-IN
- 23. REMEMBER THIS SPRINGER JOURNAL?
- 24. PHASE 3: TABLEAU, PROOF OF CONCEPT 2015 Springer JR1: When duplicate titles have the same ISSN but distinct titles, usage is picked from 1 version – now fixed!
- 25. Problem: We collect data on calendar years, but ARL needs fiscal years…(UGH) PHASE 3: PILOTING
- 26. PHASE 3: PILOT WITH ARL STATS • It’s self service! We can set-up standard views set-up in Tableau for subject librarians to retrieve
- 27. • Move from less manual (Excel, Access) to more automated and robust (Python, SQL) • Data visualization! • Self-service for renewals • NO QUESTIONS about data, built on COUNTER standards NEXT GOALS…
- 28. PREPARING FOR WHAT’S AHEAD Phases 1-3 SUSHI, COUNTER R5
Editor's Notes
- My name is Kelly Blanchat, and my talk today, titled “Beyond Counter-compliant: the importance of assessing e-resources reporting tools” will discuss Yale’s experience bringing up an ERMS for usage statistics, the methods we used to assess the output from the tool, and how we have modified our use of it over time. This talk focuses on ProQuest’s 360 COUNTER tool and how we worked with ProQuest to uncover how title normalization factored into the usage consolidation. Though this talk is specific to one tool, my hope is that it will give each of you with ERMS for usage statistics a method and purpose for replicating within your own systems.
- For Yale, 360 COUNTER was implemented in order to OUT SOURCE the harvesting of usage statistics in order to ALLOCATE staff resources elsewhere. We also wanted to leverage it to consolidate usage in order to enhance and expand our reporting capabilities. 360 COUNTER was NEVER intended to calculate CPU at Yale. At Yale, adding in usage data to another system would be unnecessarily replicating work happening in our ILS. The time it would take to add that information into 360 COUNTER didn’t seem worth it. We’re also only using the Data Retrieval Service option, which involves team members from ProQuest manually retrieving our stats for us, and not using an automated retrieval like SUSHI. One thing that I’ll talk about later is the approaching iceberg that is both SUSHI and COUNTER R5.
- Now that I’ve talked a bit about our intentions with the usage ERM, I want to clarify a bit about how the system works, since I’ll be using some language specific to 360 COUNTER in this presentation. What I’ll be referring to as “360 COUNTER” is actually made up of two parts: 360 COUNTER itself and the reporting piece, Intota Assessment. For any ProQuest ExLibris customers out there, I should clarify that Yale is only an Intota customer for this one piece – Intota Assessment – and we don’t subscribe to Intota as a whole. Essentially, how the two pieces break down is that 360 COUNTER is the AUTOMATED COLLECTION and STORAGE of usage data. Once the data is available in 360 COUNTER, it is brought into Intota Assessment, part 2, where usage is CONSOLIDATED over both TIME and PROVIDER. In short: 360 COUNTER does the collection, and Intota Assessment does the consolidated reporting, and it can generate cost-per-use calculations as well.
- Prior to implementing 360 COUNTER, usage statistics were retrieved and uploaded to a local server twice per year by a staff member in e-resources. This process was time consuming. In Spring / Summer 2015 we began implementing 360 COUNTER (which incidentally was also around the time when I began at Yale). Implementation involved entering all of our administrative data into the system, as well as uploading our historical usage reports. This, like our former processes, was time consuming, but was intended to streamline our workflows down the line. The ultimate payoff. In September / October 2015 we received our first set of outsourced statistics from the 360 COUNTER Data Retrieval Service, and we began exploring both parts of the system, 360 COUNTER and Intota Assessment. Then in November, Collection Development went in to review the usage statistics for the 2015 Oxford University Press BR1.
- It was in November 2015 that we really started kicking the tires in the system, because while retrieving these Oxford statistics, the OED was completely missing from the consolidated report in Intota Assessment, while it was present in the raw data available in 360 COUNTER. Remember, Intota Assessment is the consolidated reporting piece – and the consolidated report was also 299 titles short, and was missing 29,170 uses from the original COUNTER report.
- And if anyone asks WHY we started assessing our reporting tools in e-resources, the missing Oxford English Dictionary was a big reason why. Up until this point we had been finding smaller errors and had submitted individual tickets for ProQuest to correct. It is important to note that these tickets had been resolved quickly, and we stayed in touch with ProQuest the entire time we were implementing and testing. Our support team here was invaluable to for the next parts of my presentation. However, once the OED went missing, we started doing a much more detailed analysis.
- In order to actually get the numbers that I rattled off about the Oxford BR2 – again that was 342 total missing titles from the consolidated report and 29K uses -- we had to find a way to compare the raw COUNTER report with the consolidated usage report from Intota Assessment. To do this, we did a title level analysis using Excel’s VLOOPUP function, first matching ISSNs between raw COUNTER reports and the consolidated report, and then matching titles. A VLOOKUP can identify where data points match EXACTLY, and therefore it also isolates data points that do not have an exact match. The result was akin to the “spot the differences” comic strips from kid’s magazines. If you’ve ever smashed data together in this way, you’ll know its not a perfect method, but it does the trick when you’re trying to do high-level comparisons.
- The title-level analysis helped us identify 3 major categories of what had happened to the consolidated 2015 Oxford BR2 – --duplicate titles had been scrubbed --distinct editions of books with unique ISBNs had been merged into a single entry --ISBNs had changed from the raw COUNTER report to the consolidated report. Though this information was useful, it still didn’t answer why OED was missing. You’ll see that the totals I have here – 20, 34, 63 – don’t add up to the full 342 title difference between the reports. Again -- VLOOKUP isn’t perfect -- but what we discovered AND what was still missing lead us to the next steps in our exploration.
- Though they weren’t quite complete, we submitted our title-level findings to ProQuest for review. After about a month fixes were put into place and only 2 titles and 20 uses remained outstanding.
- The results from ProQuest were favorable, and for us the remaining 2 titles and 20 uses was nominal and likely wouldn't affect decision making. It’s still not perfect though, and we weren’t sure why. We did know that embarking on a title-level process of examining ALL consolidated reports would be EXTREMELY time consuming, and not sustainable to evaluate all providers and all reports. Figure we have 100+ providers in 360 COUNTER, times 1-8 different types of COUNTER R4 reports. This process was simply not sustainable, but was a good exercise in understanding the system and pin pointing areas for improvement and correction.
- Start Phase 2, where we moved to high-level data collection, focusing on the totals, using an Excel spreadsheet to keep track of the data over time. This image is a snipped of what the working assessment tool looks like, which is an Excel spreadsheet with a very simple SUBTRACT formula built-in. You can see here that I have a collection date noted in my header, followed by the provider, report type, and year, and then distinct areas for the TOTALS collected from both the 360 COUNTER original report and the consolidated report from Intota Assessment. Its simple math, with a big impact.
- During testing, we updated collection data if there is a big system update, or when a new batch of statistics became available. The worksheet builds as time passes, with new sections created for each distinct collection date. Therefore, we always have CONSTANTS – provider, report type, and year – as well as a variable, which is the DATE OF COLLECTION. Combined, these figures give us a rate of change over time. Remember: collection over time is intended for the SAME COUNTER REPORT, and those numbers shouldn’t be changing. With each assessment submitted to ProQuest, our goal was to see these totals getting closer to zero, where 0 equals exact data matches between 360 COUNTER and Intota Assessment. I do want to note the COUNTER-validation tool from Usus quickly here. I have tested the Usus validation tool on these reports, both the native COUNTER reports and re-formatted consolidated reports from IA. This tool cannot pick up what I am describing here, and technically both are COUNTER-compliant.
- When we moved to submitting high-level error reports to ProQuest, we were able to provide MORE data on a variety of different report types and providers, and the result gave us insight on what happens behind the scenes as data moves from 360 COUNTER collection to consolidated reports within Intota Assessment.
- Animated slide. A look at other things we’ve learned from ProQuest after submitting our high-level assessment for review.
- Here’s an example of what ProQuest found from our high-level assessment for normalization when “duplicate titles have the same ISSN and distinct titles, usage pulled from 1 version”. Since we didn’t build this system, and because the normalization process occurs in a ‘black box’, knowing this information helps us understand the system better, and also helps in our decision making about whether the system totals are valuable for our assessment. In this case, while ~900 some uses might be nominal compared to the total uses of the Springer report, the lost 847 uses compared to the normalized title’s 0 uses is significant. For a provider smaller than Springer, this kind of normalization would have an even bigger impact; and for a provider such as JSTOR, these types of occurrences will be more prevalent.
- The system ADDS usage by month instead of copying RPT from COUNTER report Titles without ISSN data cannot be normalized Titles with the same ISSN for print and online cannot be normalized
- Though there are many different usage consolidation systems, this template can work to evaluate them due to its simplicity.
- With ProQuest’s help, phases 1 & 2 challenged the assumption that our data would do what we expected. 360 COUNTER fully implemented at Yale for 1.5 years. After phase 2, we paused on our workflows assessing the 360 COUNTER system, due to the following: In the process we provided ProQuest with so much data, in addition to enhancement requests. This data helped us evaluate the system, and also helped us re-evaluate what we wanted from the system. We can’t rebuild their system, but we can make recommendations We needed to get going on consolidating usage so that we could enhance our assessment metrics 360 COUNTER was already freeing up our time in collecting reports – which is great!
- We’re putting ourselves in charge, and in IA we couldn’t see what was happening, and had to guess or put in a ticket to ask The burden is now on us, but we have a clearer view of what’s happening
- We’re now transforming our raw COUNTER data so that it is formatted as a data source, which can be lined up with our ILS data and matched behind the scenes in Tableau. Tableau is a data visualization tool, and with COUNTER as a data source we are essentially able to consolidate our own usage, on our terms. This method lets us leverage 360 COUNTER by out-sourcing the retrieval, while avoiding that pesky title normalization piece. This new process means that we’re accepting every single title already represented in the COUNTER report. Knowing that we are not over-working the report or over-normalizing our data -- and potentially creating misleading or inaccurate information -- is more important to us than pulling apart or merging title history.
- Remember: when consolidating usage using 360 COUNTER and Intota Assessment, this journal title had been represented as having 0 uses, when in fact it had 847 uses.
- Now, with our home-grown usage consolidation, we can consolidate the usage for this title, see its title history (even with its flaws), and still see its total usage count for the year in question (2015).
- Now that we know phase 3 can work, we wanted a proof of concept. Because we struggle to collect ARL statistics every year – since the switch from calendar year reporting to fiscal year reporting can be cumbersome – using ARL providers was a very clear choice.
- To create the proof of concept for ARL providers, we went through the process of turning the COUNTER reports into a data source for each provider, for the 2013 to present. Remember: since we’re still using the retrieval service from ProQuest, we had extra time to go through this process, and because the transformation into a data source happens with the click of a button, it was quick! The ARL statistics were imported into Tableau, which enabled us to easily switch from calendar year reporting to fiscal year reporting, for FY14 to FY16. The picture here is the result – success!
- Right now the process of transforming COUNTER reports into a data source is manual, requiring an Excel plug-in and an Access database. Moving forward, we’re looking into making this process more automatic by having Python pick up the reports from 360 COUNTER and transform them with a script. Additionally, when we have more and more reports as part of this process, we’ll need to switch to a database structure more robust than Microsoft Access, which we are in the process of testing with SQL. Tableau also provides our subject librarians with more opportunities for self-service usage retrieval. Right now we’re still doing some degree of emailing usage to librarians. Tableau can allow us to set-up standard and highly-requested views (or analysis types) so that any librarian can enter the system and be able to retrieve what they want.
- So far we have tackled bringing up a new system, testing it, assessing it, and starting our own home-grown version. We’re currently working within COUNTER R4 and with manual usage harvesting. On the horizon we see as potential road blocks: COUNTER R5 (and reports that may be formatted in a different way, with different or revised metrics) and SUSHI. To over come these potential blocks, we see automation as part of the solution – for both SUSHI and COUNTER 5. Python can help us create a pipeline so that it is just handled in the background and we don’t have to worry about it. Once we move fully to SUSHI we’ll be re-using the assessment worksheet to evaluate the delivery of SUSHI reports and the consistency of the data.