Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

“More than Meets the Eye” - Analyzing the Success of User Queries in Oria

326 views

Published on

Our presentation of an extended analysis of queries in the University of Oslo's library system

Published in: Education
  • Be the first to comment

  • Be the first to like this

“More than Meets the Eye” - Analyzing the Success of User Queries in Oria

  1. 1. “More than Meets the Eye” Analyzing the Success of User Queries in Oria Hugo Huurdeman, Mikaela Aamodt, Dan Michael Heggø University of Oslo Library VIRAK conference, 2017-06-13
  2. 2. 1. Introduction • More insights are needed into library search issues, what goes right and what goes wrong • Opportunity: analyze data gathered at UiO during the last two years
  3. 3. Research Questions 1. Which insights can we gain from classifying user queries within Oria by their popularity, specificity and intended target resources? 2. To what extent are the most popular user queries successful? 3. What underlying reasons for unsuccessful queries can be determined?
  4. 4. 2. Data: Primo Analytics • Actions • Device usage • Facets • Popular Searches (monthly) • Timeframes: Jan-Jun 2015; Nov-Dec 2015; Jan-Sep 2016 • Sessions • Zero result queries (daily) • Aug 7 2015-Sep 30 2016
  5. 5. 2. Processing & Analysis • Basic normalization of queries • "stanislav andreski" -- stanislav andreski • Manual annotation of • (1) top 50 popular queries • (2) random selection of 50 "zero result" queries • If derivable from query, determine • nature of query? • for which resource type? • curriculum-related? [in pensum lists UiO] • successful? [result on first results page]
  6. 6. 3. Previous work • Dominance of commercial search engines in students’ information seeking (Griffiths & Brophy, 2005) • Students adopting search behavior from commercial search engines in library OPACs (Shiv, 2012), Willson & Given, 2010) • In particular, undergraduates just entering academia (Novotny, 2014)
  7. 7. 4. Nature of user queries RQ1 Which insights can we gain from classifying user queries within Oria by their popularity, specificity and intended target resources? • Popular Searches and Zero Result Searches
  8. 8. 4.1 Popular Searches dataset • 5,776 different queries, 115,590 searches (UiO) • monthly totals (500) • 2015 & 2016 • 4.9% of all search actions • Analyzing top 50 • Together issued almost 20,000 times
  9. 9. Top 50 • What types of queries are most popular? titles (33x, 66%) • det kvalitative forskningsintervju • books, journals, databases topics / titles (6x, 12%) • spesialpedagogikk • neurology topics (4x, 8%) • cessio legis
  10. 10. Top 50 • What resource types are sought for? Books (50%) • det kvalitative forskningsintervju • det norske samfunn Journals (12%) • science • lancet Databases (10%) • atekst • pubmed • duo
  11. 11. Curriculum? • Were the queries related to pensum books? • Quite often: at least 58%! • menneskets fysiologi • det kvalitative forskningsintervju • Or other things (9%) :) • aftenposten; morgenbladet • harry potter
  12. 12. Curriculum? • Were the queries related to pensum materials? • Quite often: at least 58%! • menneskets fysiologi • det kvalitative forskningsintervju • Or other things (34%) :) • aftenposten; morgenbladet
  13. 13. 4.2 “Zero result” queries • 39,925 different queries • In total 52,257 searches • Aug 2015-Sep 2016 • 2.2% of all search actions • Annotating random sample (50) al azm sadik «orientalisme og omvendt orientalisme» 28 curr eye res 27 allmenningen olaf 24 12136173x 22 am j ophthalmol 21 821017268 21 askim j. 19 900317809 19 cheng 19 (direkte krav or direktekrav) and subrogasjon 17 151582998 16 142734500 16 direkte krav or direktekrav 16 961675616 16 9780549547303 16 agirdag phalet 16 andresen steinar elin l. boasson og geir hønneland (2012) international environmental agreements 15
  14. 14. Queries without results What is the nature of the “zero result” queries performed in Oria? • Pasted reference (20x, 40%) • Browning, N. (2015). The ethics of two-way symmetry and the dilemmas of dialogic kantianism. Journal of Media Ethics • Title (15x, 30%) • Sentralbankens oppgaver i dag og i fremtiden • Author (8x, 16%) • Christopher Hotchens
  15. 15. Queries without results • Book (28%) • Prcopius Secret History • Book Chapter (12%) • Solhaug, (2006). Kapittel 13: Strategisk læring i samfunnsfag. I • Article (24%) • E.g., pasted references Which resource types are not found?
  16. 16. Degree of pensum queries? At least 28% of the unsuccessful queries are for pensum materials • Fukuyama, F. (2013): What Is Governance? Governance, Vol. 26, No. 3, July 2013 (s. 347–368). • basic immubology
  17. 17. 5. Search Success RQ2 To what extent are the most popular user queries successful?
  18. 18. Top queries 1. atekst 1425 2. pubmed 1221 3. exphil 719 4. det kvalitative forskningsintervju 711 5. nature 669 6. medical genetics 598 7. direktekrav 562 8. menneskets fysiologi 540 9. 998903677444702000 500 10. science 483 11. spesialpedagogikk 475 12. lancet 438 13. jussens venner 369 14. 991645464 364
  19. 19. 1. atekst 1425 2. pubmed 1221 3. exphil 719 4. det kvalitative forskningsintervju 711 5. nature 669 6. medical genetics 598 7. direktekrav 562 8. menneskets fysiologi 540 9. 998903677444702000 500 10. science 483 11. spesialpedagogikk 475 12. lancet 438 13. jussens venner 369
  20. 20. 1. atekst 1425 2. pubmed 1221 3. exphil 719 4. det kvalitative forskningsintervju 711 5. nature 669 6. medical genetics 598 7. direktekrav 562 8. menneskets fysiologi 540 9. 998903677444702000 500 10. science 483 11. spesialpedagogikk 475 12. lancet 438 13. jussens venner 369
  21. 21. 1. atekst 1425 2. pubmed 1221 3. exphil 719 4. det kvalitative forskningsintervju 711 5. nature 669 6. medical genetics 598 7. direktekrav 562 8. menneskets fysiologi 540 9. 998903677444702000 500 10. science 483 11. spesialpedagogikk 475 12. lancet 438 13. jussens venner 369 Alma
  22. 22. Were the queries successful? • Often, yes: • main result in first 10 results: 58% • not (easily) found: 20% • For example: • pubmed • nature • science
  23. 23. What are causes for unsuccessful queries? • Ambiguous names • nature, science • At the time of writing, no entries for some databases • pubmed
  24. 24. 6. Queries without results RQ3 What underlying reasons for zero result queries can be determined?
  25. 25. Why no results? Pasted reference
  26. 26. Why no results? • Query being too specific (pasted reference, pasting quote) (22%) • Browning, N. (2015). The ethics of two-way symmetry and the dilemmas of dialogic kantianism. Journal of Media Ethics • Misspellings, reference mistakes (e.g. wrong year) (20%) • svennevig j.: ledelsesretorikk i nedbemanningssituasjoner 2009 • Using incorrect query syntax (2%) • "McLuhan" AND/OR "Understanding media" • 978-147996410-9 • Wrong scope (12%), wrong field (4%)
  27. 27. Why no results (2) • Searching for an ISBN number (specific edition not in library) • 9780618721566 • Searching for journal titles, ISSNs, DOIs • journal of speech and hearing disorders • Searching for course codes • MED1100 • Resource not (indexed) in Oria (16%) • Haugianerliberalistene: En analyse av haugianere som politikere og næringslivsaktører
  28. 28. Suggestions for misspellings? • 10 out of 50 queries (20%) were caused by misspellings. •For half, a correct spelling suggestion exists •(April 17)
  29. 29. Do queries still return 0 results? • In 72% of the cases, no improvement, but 28% is now resolved. • April ’17
  30. 30. 7. Conclusion & discussion • Library catalog containing more than meets the eye • Even though materials are available, they do not always show in searches • Issues in: • query formulation • system support...
  31. 31. Discussion • Refine and extend search suggestions • Enhanced spell check / query corrections • why students underacheive → 0 results • why students underachieve → 463 results • Query suggestions and autocomplete • Could be based on previous (successful) queries • especially recurring queries should be supported (50% of popular queries!) • ISBN suggestions • Automatically search for the book title and author (using information derived from ISBN number?)
  32. 32. Discussion • 0 results: get more helpful information • Requesting assistance, material • Contextual search suggestions • e.g. broaden search, simplify query, alternate formulations • Available scopes • suggest how many searches in other scope
  33. 33. Discussion • Importance of "pensum" queries • Better integration with pensum lists • Detecting curriculum queries • E.g. widget which suggests pensum materials directly // referring students to UiO fagsider // etc. • On the cataloging side: • Course codes (e.g. INF2260) "to catalog or not to catalog" • More support for Database queries • Atekst could be found, not pubmed
  34. 34. Discussion • Monitoring searches • Detect sudden ‘spikes’ in zero result queries • Finding errors in curriculum lists • Detecting ‘holes’ in collections (acquisition staff), and e.g. popular books with too few copies
  35. 35. 8. Future work • Plans for further analysis • Comparing queries with most frequent loans • Alma Analytics • Current analysis: many known-item queries. Also look at common exploratory queries • medicine; math; united nations; economics • how can we support those types of queries better? • Obtaining more data (Limits Primo Analytics) • e.g. analyzing "struggling sessions", stats location, etc • Look at use of external databases, link resolver stats
  36. 36. bit.ly/the-visualisation-project @BookNavigationFind this presentation here
  37. 37. “More than Meets the Eye” Analyzing the Success of User Queries in Oria Hugo Huurdeman, Mikaela Aamodt, Dan Michael Heggø 2017-06-13

×