Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Measuring Our Relevancy:
Comparing Results in a Web-Scale Discovery Tool,
Google & Google Scholar
Elizabeth Namei, Univers...
unexpected
results
“I could throw an author
and title in [Summon] but
I know that works very
erratically….I often get
inundated by irrelevant...
In search of evidence
methods
We pulled
out a random
sample of 384 queries
There were 1.2 million searches
entered in Summon in Fall 2014.
From the 384 queries we eliminated:
● 63 queries for known items that USC did not
own
● 22 queries that had unrecognizable...
defining relevance
1. Relevant: a match for the
known item shows up 1st or 2nd
in the list of results.
2. Partially Releva...
results
76%
91%
79%
24%
100,000
(69/278)
(183/278)
(9/278)
(11/278)
(6/278)
Partial Citation queries (with 2 metadata elements)
that did not find relevant results in Summon
and then there were none ...
Quotes were used in only 6%
of the queries (16/278)
but……
100% of these queries
returned relevant results
Formatted citations (with 3 or more metadata
elements) that did not find relevant results in Summon
Block G Hartman AM Dre...
if fewer metadata elements had been used...
Block A data-based approach to diet
questionnaire design and testing.
shows up...
where
do we
go from
here?
Continue teaching users to be more strategic
searchers
● Explain the whys of strategic
searching and not just the hows
● E...
follow the user
Push for In-house solutions
● Using APIs
● Developing programs to
“clean” problematic queries
Lean on vendors
• to improve...
personalization
questions?
comments?
Image credits
Slide 1 image https://flic.kr/p/9pfXxq
Slide 2 image: http://bit.ly/1NkcLNJ
Slide 4 image: http://bit.ly/1G0...
Acrl2015 presentation gotime
Acrl2015 presentation gotime
Acrl2015 presentation gotime
Upcoming SlideShare
Loading in …5
×

Acrl2015 presentation gotime

761 views

Published on

How successful are discovery tools in providing relevant results? Can these single search boxes really displace/replace the library catalog? This presentation presents the results of an analysis of user queries sampled from the University of Southern California's discovery tool and how results using these same queries compare when entered into Google and Google Scholar in terms of relevancy. This session will provide insight into user search behavior and examine if discovery tools are actually delivering on their promise to deliver relevant and expected results.

Published in: Education
  • Be the first to comment

Acrl2015 presentation gotime

  1. 1. Measuring Our Relevancy: Comparing Results in a Web-Scale Discovery Tool, Google & Google Scholar Elizabeth Namei, University of Southern California Libraries, namei@usc.edu and Christal Young, University of Southern California Libraries, youngc@usc.edu #summonrelevancy ACRL 2015 Conference
  2. 2. unexpected results
  3. 3. “I could throw an author and title in [Summon] but I know that works very erratically….I often get inundated by irrelevant things…” “[Summon] just doesn’t generally turn up reliable results for me. I’ll search a very obvious keyword or a very specific keyword and it won’t turn up the most relevant results first even though I know the highly relevant results are in there...” English professor Senior music major
  4. 4. In search of evidence
  5. 5. methods
  6. 6. We pulled out a random sample of 384 queries There were 1.2 million searches entered in Summon in Fall 2014.
  7. 7. From the 384 queries we eliminated: ● 63 queries for known items that USC did not own ● 22 queries that had unrecognizable or foreign characters ● 21 queries that were for “non-scholarly” formats Our final sample size = 278
  8. 8. defining relevance 1. Relevant: a match for the known item shows up 1st or 2nd in the list of results. 2. Partially Relevant: a match for the known item is found 3rd- 10th in the list of results. 3. Not relevant: known item is not listed in the first 10 results or no results were returned.
  9. 9. results
  10. 10. 76% 91% 79%
  11. 11. 24%
  12. 12. 100,000
  13. 13. (69/278) (183/278) (9/278) (11/278) (6/278)
  14. 14. Partial Citation queries (with 2 metadata elements) that did not find relevant results in Summon and then there were none agatha christie palladio four books on architecture showed up 74th shows up 75th
  15. 15. Quotes were used in only 6% of the queries (16/278) but…… 100% of these queries returned relevant results
  16. 16. Formatted citations (with 3 or more metadata elements) that did not find relevant results in Summon Block G Hartman AM Dresser CM Carroll MD Gannon J Gardner L. A data-based approach to diet questionnaire design and testing. Am J Epidemiol 1986; 124:453-469. does not show up in first 10 results (6 metadata elements) no results returned (9 metadata elements) Brooks M. K. (2010). Hospice services: The dilemmas of technology at the end of life. In T. S. Kerson J. L. M. McCoyd & Associates (Eds.) Social work in health settings: Practice in context (3rd ed. pp. 235-246). New York NY: Routledge.
  17. 17. if fewer metadata elements had been used... Block A data-based approach to diet questionnaire design and testing. shows up 1st (2 metadata elements) Shows up 1st (5 metadata elements) Brooks M. K. (2010). Hospice services: The dilemmas of technology at the end of life. In T. S. Kerson J. L. M. McCoyd & Associates (Eds.) Social work in health settings: Practice in context
  18. 18. where do we go from here?
  19. 19. Continue teaching users to be more strategic searchers ● Explain the whys of strategic searching and not just the hows ● Explain the organization of information systems and the benefits field searching (especially for one word/common word titles) ● Provide troubleshooting tips for how to deal with the inevitable failed searches
  20. 20. follow the user
  21. 21. Push for In-house solutions ● Using APIs ● Developing programs to “clean” problematic queries Lean on vendors • to improve relevancy • to improve “Did you mean” suggestions • To provide query reformulation suggestions Create Smarter Library Systems
  22. 22. personalization
  23. 23. questions? comments?
  24. 24. Image credits Slide 1 image https://flic.kr/p/9pfXxq Slide 2 image: http://bit.ly/1NkcLNJ Slide 4 image: http://bit.ly/1G0s5fY Slide 5 image: http://bit.ly/1DQRREd Slide 6 image: http://thevisualharvest.com/lib/img/elements/to-the-drawing-board-graphic.svg Slide 8 image: http://relevantinfo.org/wp-content/uploads/2013/09/good-o-meter.jpg Slide 9 images: ● Jeopardy: http://www.ibm.com/smarterplanet/us/en/ibmwatson/assets/img/tech/watson_on_jeopardy.jpg ● Summon: http://www.almanhal.com/ResourceImages/Publisher/StartigicPublisherLogos/ProQuest.png ● Google: http://bit.ly/1I0b0RZ ● Google Scholar: http://www.redcad.org/members/hadjkacemm/images/GS2.png Slide 12 image: http://wishflowers.tumblr.com/post/10244886760 Slide 16 image: http://bit.ly/1EdFthI Slide 21 image: http://www.healthcarereformmagazine.com/voluntary-benefits/where-do-we-go-from-here-voluntary/ Slide 22 image: http://samanthashorey.blogspot.com/2011_02_01_archive.html Slide 23 image: http://blog.cityspoon.com/2012/02/08/gathering-followers/ Slide 24: Image: http://www.csfieldguide.org.nz/ArtificialIntelligence.html Slide 25: image: http://bloomreach.com/snap Slide 26: image: http://www.gtcw.org.uk/gtcw/index.php?lang=cy

×