Letting Users Lead:
Analyzing Search Queries & Relevancy
in USC’s Web-Scale Discovery Tool
California Association of Resea...
Web Scale Discovery Services in a nutshell
image: http://www.colinburnett.com/wp-content/uploads/2014/01/livingunderrock.j...
The hype - Have we chosen wisely?
image: http://jessicaknauss.blogspot.com/2012/09/the-grail-knight-as-inspiration.html
Why USC got Summon
#1: To provide better discoverability of our subscription
and purchased content (via a unified access p...
Our study
We re-executed a sample of Summon search
queries to see how successful our users were in
retrieving relevant res...
April 2010 - Pre-Summon homepage
July 2010 - Summon is launched as the default tab
July 2012 - Our current Summon-centric homepage
(Catalog tab is removed)
Motivation for our study
• There were a lot of anecdotal complaints.
• To get evidence about how successful Summon
was in ...
our
methodology
image: http://www.rsc.org/chemistryworld/issues/2007/december/thechemistrysetgeneration.asp
Transaction Log Analysis (TLA)
A transaction log is a history of actions executed in a system.
TLA involves looking at the...
Advantages of TLA
• Unobtrusive
• Large quantities of data that can reveal large-scale
patterns
• Low cost
Limitations of TLA
• Does not capture contextual information such as the
user’s emotions, motivations, intentions and need...
Summon searches in Fall 2013: 1,243,250
# of unique searches: 184,076
Our sample: 384 searches
Margin of error: +/- 5%
Defining success (and failure)
image: http://abovethelaw.com/wp-content/uploads/2013/12/thumbs-up.jpg
Success = Relevant results
"We are used to hearing people talk about the 'simple
search box' as the goal” of Discovery ser...
Information overload
“Surveys of...users
reveal a consistent
theme: most are
overwhelmed and
confused by the
disorganized ...
“Surveys of Internet users reveal a consistent theme:
most are overwhelmed and confused by the
disorganized flood of infor...
Information used to
be scarce….
Image: http://allaboutalpha.com/blog/wp-content/uploads/2011/04/iStock_000014587595XSmall....
But today our attention
span and time are scarce
while information is
abundant and easily
accessible.
“There are so very m...
Summon’s Relevance Ranking
Looks at:
• term proximity - how close query terms are to one
another.
• term frequency - how o...
Also considers:
• Some content types are boosted over others
o Books and journal articles over newspaper articles and book...
A “Success Engine”
Relevancy Enhancements: “to ensure users don’t miss out on the most
relevant content.”
This includes au...
Measuring Relevancy in our Study
We used “systems-oriented relevance” to rate the
success of search queries.
We rated the ...
Inter-rater reliability
mage: - http://www.artofmanliness.com/2008/08/21/manly-feats-of-strength/
Types of searches
Known item searches
Had to be specific enough for
us to recognize a definitive
match. If the search got
...
Our Relevancy Rubric
Known Item Searches
Relevant:
1st item in the
list of results
Partially
Relevant:
2nd-10th item in
th...
Our Relevancy Rubric
Keyword Searches
Relevant: ALL search
terms appear in item's title or
record.
All search terms appear...
Summon vs. Google vs. Google Scholar
Known Item
Searches:
For Google & Google
Scholar, the relevancy
was determined by a
m...
our findings...
Top 100 most frequent queries in Fall 2013
Consisted of 23,813 individual searches
Long Tail of unique searches
87% (160,263) of total searches
make up the long tail
Top 100 unique
searches make up
13% of ...
36%
62%
2%
Types of Searches
Image: http://seanarchy.files.wordpress.com/2013/12/educated-poor.jpg
Summon vs. Google vs. Google Scholar
Which do you think did better?
http://bit.ly/summon-faceoff
Successful Searches:
54% of all our sample searches
(204) retrieved relevant results.
Summon’s Overall Relevancy Report Ca...
Successful Searches:
85% (318) of the sample
searches retrieved
relevant results
Google’s Relevancy Report Card = B
After ...
Successful Searches:
54% of the sample
searches (203) retrieved
relevant results.
Google Scholar’s Relevancy Report Card =...
Relevancy of all Keyword Searches
n=236
Relevancy of all known item searches
n=139
Failed searches in Summon
32% (45) of all known item searches did not locate the
item being searched for
• 33% (15) of the...
Relevancy of “academic” known item searches that USC
owns
Summon
improved 13%
Google Scholar
improved 15%
Google improved ...
Revised Relevancy Report Cards
Google Scholar = F
57% (192) retrieved
relevant results
(up from 53%)
After Curve = C+
79% ...
User errors
18% (66) of the
searches in our
sample had a
user input error
Image: http://human-error.sarkisozlerik.com/huma...
“Did you mean?”
Showed up 24 times
• 83% (20) triggered by user input errors.
• 42% (10) of the time the “Did you mean” li...
Google automatically redirects searches with errors
Summon lets the user
decide whether to follow
a “corrected” path
Median: 3
Mean: 3.182
Mode: 2
# of words used in Keyword Searches
Impact of # of search terms entered on relevancy
Type of Keyword Searches Executed
n=236
Impact of search type on relevancy
Duplicates
image: http://amarkedman.com/wp-content/uploads/2011/07/Matrix-Clones.jpg
Known item searches:
15 searches had ...
Linking to full-text
"Linking users to full text as
quickly as possible after
discovery results are available
is a paramou...
What we learned:
Summon has some work to do to improve the
relevancy of its results.
But, it is doing better in other area...
Summon added
as default search
tab (July 2010)
Catalog search
tab removed
from homepage
(July 2012)
Winning! (sort of…)
image: http://beautelicious.com/2011/09/charlie-sheen-warner-bros-close-settling-wrongful-termination-...
Final Report Card:
Relevancy = C+
Intuitive starting place =
Fast =
Bringing users back to the library =
Maximizing usage ...
Moving forward - Following our users’ lead
image: http://blog.cityspoon.com/2012/02/08/gathering-followers/
Leadership Strategy #1: Learn and use your
library’s discovery tool
• 3 million searches in 2013!
• Will give you insight ...
Leadership Strategy #2: Change what and how
we teach
”Librarians must reconsider training students to use
advanced search ...
Leadership Strategy #3: Be a squeaky wheel
• Many users will get frustrated and abandon the library
without out ever letti...
Leadership Strategy #4: Imporvise and fall in
front of students
• Show students how to troubleshoot a failed search.
• Sho...
"If we think like users (instead of as librarians) it is easy
to understand the frustration. Our tools must seem
broken or...
“The trouble with Summon is that students don’t need to be
taught how to use it, but librarians do”
-Matt Borg, Sheffield ...
English Faculty Member:
“I want an easy way to find a book with
author and title, and an easy way to move
from that to jou...
Leadership strategy #6: Empathize with users
AND colleagues
• Try not to criticize or judge
• Invite skeptical librarians ...
Common complaints
• “I think it's a cheat. Too many students don't learn basic searching skills that
would make any search...
Complaining is not a (constructive) strategy
“When you invent something new, if customers come to
the party, its disruptiv...
Leadership Strategy #7: Solicit feedback AND
then listen
• Look for positive AND negative feedback
• Re-frame negatives as...
Leadership Strategy #8: Turn negativity to
your advantage
“Engage and transform the
most negative person in your
library s...
Leadership Strategy #9: Gather evidence
• Test your assumptions
• Test your colleagues’ assumptions
• Study user behavior,...
Leadership Strategy # 10: Redefining ourselves
References
Asher, Andrew D., Lynda M. Duke, and Suzanne Wilson. “Paths of Discovery: Comparing the
Search Effectiveness of...
Holman, Lucy. “Millennial Students’ Mental Models of Search: Implications for Academic Librarians
and Database Developers....
Pan, Bing, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. “In
Google We Trust: Users’ Decis...
Carl 2014 slides_gotime
Upcoming SlideShare
Loading in …5
×

Carl 2014 slides_gotime

807 views

Published on

California Association of Research Libraries 2014 Conference, San Jose, CA

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
807
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Carl 2014 slides_gotime

  1. 1. Letting Users Lead: Analyzing Search Queries & Relevancy in USC’s Web-Scale Discovery Tool California Association of Research Libraries, 2014 April 5, 2014 Beth Namei, University of Southern California Christal Young, University of Southern California
  2. 2. Web Scale Discovery Services in a nutshell image: http://www.colinburnett.com/wp-content/uploads/2014/01/livingunderrock.jpg
  3. 3. The hype - Have we chosen wisely? image: http://jessicaknauss.blogspot.com/2012/09/the-grail-knight-as-inspiration.html
  4. 4. Why USC got Summon #1: To provide better discoverability of our subscription and purchased content (via a unified access point) #2: Provide relevant results to our users (most urgently to provide relevance ranked results for items in our SIRSI OPAC) #3: To provide a better user experience with the library’s website
  5. 5. Our study We re-executed a sample of Summon search queries to see how successful our users were in retrieving relevant results.
  6. 6. April 2010 - Pre-Summon homepage
  7. 7. July 2010 - Summon is launched as the default tab
  8. 8. July 2012 - Our current Summon-centric homepage (Catalog tab is removed)
  9. 9. Motivation for our study • There were a lot of anecdotal complaints. • To get evidence about how successful Summon was in leading users to relevant sources. • To learn more about user search behavior with this single search box
  10. 10. our methodology image: http://www.rsc.org/chemistryworld/issues/2007/december/thechemistrysetgeneration.asp
  11. 11. Transaction Log Analysis (TLA) A transaction log is a history of actions executed in a system. TLA involves looking at the data captured in a transaction log to investigate interactions between users and a search tool.
  12. 12. Advantages of TLA • Unobtrusive • Large quantities of data that can reveal large-scale patterns • Low cost
  13. 13. Limitations of TLA • Does not capture contextual information such as the user’s emotions, motivations, intentions and needs. • Does not capture demographic information. • Does not capture user satisfaction with the overall search experience.
  14. 14. Summon searches in Fall 2013: 1,243,250 # of unique searches: 184,076 Our sample: 384 searches Margin of error: +/- 5%
  15. 15. Defining success (and failure) image: http://abovethelaw.com/wp-content/uploads/2013/12/thumbs-up.jpg
  16. 16. Success = Relevant results "We are used to hearing people talk about the 'simple search box' as the goal” of Discovery services. But a simple search box has only been one part of the Google Formula. PageRank has been very important in providing a good user experience, and Google has progressively added functionality as it included more resources in the search results" (Dempsey, 2012, 4).
  17. 17. Information overload “Surveys of...users reveal a consistent theme: most are overwhelmed and confused by the disorganized flood of information” (Riley 24). image: https://www.flickr.com/photos/youreyestellies/7984526358
  18. 18. “Surveys of Internet users reveal a consistent theme: most are overwhelmed and confused by the disorganized flood of information. As a result, librarians have an opportunity to carve their niche as the de facto information navigators of the Digital Age” Riley, Margaret. "Riley's Guided Tour: Job Searching On The Net." Library Journal 121 (1996): 24-27.
  19. 19. Information used to be scarce…. Image: http://allaboutalpha.com/blog/wp-content/uploads/2011/04/iStock_000014587595XSmall.jpg
  20. 20. But today our attention span and time are scarce while information is abundant and easily accessible. “There are so very many search engines now and so much information; I preferred the ‘good old days’ when there weren’t so many ways to access information” - USC Journalism faculty member. image: http://gcn.com/articles/2013/09/30/smarter-cities-cloud.aspx
  21. 21. Summon’s Relevance Ranking Looks at: • term proximity - how close query terms are to one another. • term frequency - how often do the query terms appear in the record • field weighting - where is the query term found? Finding query terms in some fields are more important than others.
  22. 22. Also considers: • Some content types are boosted over others o Books and journal articles over newspaper articles and book reviews. o The Journal over the articles in that journal • Publication Date - Generally, items with newer publication dates are favored over older items. • Citation Counts - Items cited more are boosted • Local Collections - Content from institutional catalog(s) and repositories are boosted. • Content Size - Longer works aren’t necessarily more relevant even though search terms might appear in them more often
  23. 23. A “Success Engine” Relevancy Enhancements: “to ensure users don’t miss out on the most relevant content.” This includes automatically searching for synonyms, term-stemming and smart searching of stop-words depending on their importance to the search phrase. image: http://blog.dappersnappers.com/wp-content/uploads/2012/02/baby-laptop-computer.jpg
  24. 24. Measuring Relevancy in our Study We used “systems-oriented relevance” to rate the success of search queries. We rated the relevancy of Summon’s results without knowing the context of the original query. We looked at how well the topic of the search was represented in the topics of the results retrieved in Summon, Google and Google Scholar. - Maglaughlin & Sonnenwald 2002, 328-9.
  25. 25. Inter-rater reliability mage: - http://www.artofmanliness.com/2008/08/21/manly-feats-of-strength/
  26. 26. Types of searches Known item searches Had to be specific enough for us to recognize a definitive match. If the search got numerous matches and was general, we would likely not categorize it as a known item. Examples: • marketing alterity moore • 978-0078026676 • An Empirical Analysis of Cigarette Addiction • "happy days are here again" garland streisand Keyword/topic/exploratory searches Encompassed broad, general or ambiguous searches. Named persons were put into this category as well. Examples: • Catherine the Great, • vulnerability and happiness • PTSD and Substance use in african american women • supersitions
  27. 27. Our Relevancy Rubric Known Item Searches Relevant: 1st item in the list of results Partially Relevant: 2nd-10th item in the list of results Not relevant: Not listed in the first 10 results or no results retrieved. (could mean we don’t own the item or there is a user input error)
  28. 28. Our Relevancy Rubric Keyword Searches Relevant: ALL search terms appear in item's title or record. All search terms appear to have a relationship to one another, they are not just randomly placed throughout the title/record. First 5 items look like a "perfect"/solid match or clearly seem to be ABOUT the topic as identified through the search terms. Partially Relevant Not all search terms are visible in the title or record of the items. At least 3 of the first 5 results appear to be somewhat related to the topic as entered, even if broadly or tangentially. Would a user easily recognize a connection between the results and the topic as it was entered? Not Relevant: At least four of the first 5 items appear to be false hits. Even if one or more of the search terms (or synonyms of those terms) appear in the title, abstract or record, results appear to be only about a portion of the search terms entered and not about all of them combined.
  29. 29. Summon vs. Google vs. Google Scholar Known Item Searches: For Google & Google Scholar, the relevancy was determined by a match, did not have to be a full-text match. Links to Amazon, Google Books, WorldCat, imdb.com, and Youtube were matches. Image: http://www.geekwire.com/2013/ibm-takes-watson-cloud/
  30. 30. our findings...
  31. 31. Top 100 most frequent queries in Fall 2013 Consisted of 23,813 individual searches
  32. 32. Long Tail of unique searches 87% (160,263) of total searches make up the long tail Top 100 unique searches make up 13% of total queries
  33. 33. 36% 62% 2% Types of Searches
  34. 34. Image: http://seanarchy.files.wordpress.com/2013/12/educated-poor.jpg
  35. 35. Summon vs. Google vs. Google Scholar Which do you think did better? http://bit.ly/summon-faceoff
  36. 36. Successful Searches: 54% of all our sample searches (204) retrieved relevant results. Summon’s Overall Relevancy Report Card = F After the curve = C Moderately Successful + Successful Searches: 73% (273) retrieved partially relevant - relevant results image: http://bleedingedge.pynchonwiki.com/wiki/images/b/b8/Dunce-cap.jpg
  37. 37. Successful Searches: 85% (318) of the sample searches retrieved relevant results Google’s Relevancy Report Card = B After the curve: A Moderately Successful + Successful Searches: 95% (356) of the searches retrieved partially relevant - relevant results.
  38. 38. Successful Searches: 54% of the sample searches (203) retrieved relevant results. Google Scholar’s Relevancy Report Card = F After the curve: C Moderately Successful + Successful Searches: 75% (281) retrieved partially relevant - relevant results.
  39. 39. Relevancy of all Keyword Searches n=236
  40. 40. Relevancy of all known item searches n=139
  41. 41. Failed searches in Summon 32% (45) of all known item searches did not locate the item being searched for • 33% (15) of these failed searches are for items USC does not own Of the items we do own: • 57% (17) did not show up due to user error • 40% (12) did not show up due to a Summon problem (bad metadata, poor relevancy, not indexed in Summon) • 3% (1) had irregular characters and found no results
  42. 42. Relevancy of “academic” known item searches that USC owns Summon improved 13% Google Scholar improved 15% Google improved 1% n=109
  43. 43. Revised Relevancy Report Cards Google Scholar = F 57% (192) retrieved relevant results (up from 53%) After Curve = C+ 79% (272) retrieved partially relevant - relevant results (up from 73%) Summon = F 59% (202) retrieved relevant results (up from 53%) After Curve = C+ 79% (271) retrieved partially relevant - relevant results (up from 73%) Google = B 84% (291) retrieved relevant results (down from 85%) After Curve = A 95% (328) retrieved partially relevant - relevant results (no change)
  44. 44. User errors 18% (66) of the searches in our sample had a user input error Image: http://human-error.sarkisozlerik.com/human-error/a-lifetime-by-design.html
  45. 45. “Did you mean?” Showed up 24 times • 83% (20) triggered by user input errors. • 42% (10) of the time the “Did you mean” links took users to relevant results.
  46. 46. Google automatically redirects searches with errors Summon lets the user decide whether to follow a “corrected” path
  47. 47. Median: 3 Mean: 3.182 Mode: 2 # of words used in Keyword Searches
  48. 48. Impact of # of search terms entered on relevancy
  49. 49. Type of Keyword Searches Executed n=236
  50. 50. Impact of search type on relevancy
  51. 51. Duplicates image: http://amarkedman.com/wp-content/uploads/2011/07/Matrix-Clones.jpg Known item searches: 15 searches had 2 or more duplicates (11%) Keyword searches: 51 searches had 2 or more duplicates (22%)
  52. 52. Linking to full-text "Linking users to full text as quickly as possible after discovery results are available is a paramount concern" (NISO ODI Report, 2013, 7). There were only 3 bad links (out of 55 known item article searches) image: https://flic.kr/p/mqjHgR
  53. 53. What we learned: Summon has some work to do to improve the relevancy of its results. But, it is doing better in other areas….
  54. 54. Summon added as default search tab (July 2010) Catalog search tab removed from homepage (July 2012)
  55. 55. Winning! (sort of…) image: http://beautelicious.com/2011/09/charlie-sheen-warner-bros-close-settling-wrongful-termination-suit/
  56. 56. Final Report Card: Relevancy = C+ Intuitive starting place = Fast = Bringing users back to the library = Maximizing usage of collections =
  57. 57. Moving forward - Following our users’ lead image: http://blog.cityspoon.com/2012/02/08/gathering-followers/
  58. 58. Leadership Strategy #1: Learn and use your library’s discovery tool • 3 million searches in 2013! • Will give you insight into what users are experiencing, both good and bad. • Discovery tools are taking up prime real estate on many of our websites. image: http://www.creativity4us.com/wp-content/uploads/2012/02/blinders-crop.jpg
  59. 59. Leadership Strategy #2: Change what and how we teach ”Librarians must reconsider training students to use advanced search features or Boolean logic if students purposefully choose not to use them or fail to use them correctly. Rather than teaching students more effective search syntax, more attention should be placed on developing critical thinking and evaluative skills" (Holman, 2011, 24).
  60. 60. Leadership Strategy #3: Be a squeaky wheel • Many users will get frustrated and abandon the library without out ever letting us know about problems • We cannot depend on other people to report problems
  61. 61. Leadership Strategy #4: Imporvise and fall in front of students • Show students how to troubleshoot a failed search. • Show searches with typos or how to revise a search that gets no results)
  62. 62. "If we think like users (instead of as librarians) it is easy to understand the frustration. Our tools must seem broken or outdated to them….Are we in the business of promoting library databases or the business of helping users accomplish their tasks?” (Matthews, 2013) Leadership Strategy #5:
  63. 63. “The trouble with Summon is that students don’t need to be taught how to use it, but librarians do” -Matt Borg, Sheffield Hallam University, 2012
  64. 64. English Faculty Member: “I want an easy way to find a book with author and title, and an easy way to move from that to journal articles if that’s what I want….” Religion Faculty Member: “If I need to search something I don’t go to any USC search engine, which is totally a waste of time. I go to Google where I can get things ten times as fast.” image: http://www.morvimmer.com/blog/free-download-staples-easy-button/
  65. 65. Leadership strategy #6: Empathize with users AND colleagues • Try not to criticize or judge • Invite skeptical librarians into your classes to watch you teach w/the discovery service • Showing vs telling - talking can only get you so far
  66. 66. Common complaints • “I think it's a cheat. Too many students don't learn basic searching skills that would make any search better - like planning before typing. They just throw in anything and take what comes up first” - 3/1/14 Survey of USC Instruction Librarians • “It is so imprecise" (Buck & Steffy, 2013, 76). • Perpetuates "the homogenization of information" - when "everything looks and feels the same" (Bawden & Robinson, 2008, 181). • “pandering to the 'principle of least effort'” (Richardson 2013; Meadow & Meadow 163-4). • They “impinge upon the development of critical research skills” (Wiles & Hofmann, 2013, 156). • “these systems...reinforce unreflective research habits” (Asher, 2013, 6)
  67. 67. Complaining is not a (constructive) strategy “When you invent something new, if customers come to the party, its disruptive to the old way….The internet is disrupting every media industry...people complain about that but complaining is not a strategy. Amazon is not happening to bookselling, the future is happening to bookselling.” -Jeff Bezos, 60 Minutes, [8:55]. 12/1/2013
  68. 68. Leadership Strategy #7: Solicit feedback AND then listen • Look for positive AND negative feedback • Re-frame negatives as positives or as opportunities for dialogue
  69. 69. Leadership Strategy #8: Turn negativity to your advantage “Engage and transform the most negative person in your library system into a productive team member.” By “converting the most negative person has a huge impact on the rest of the staff” (Cuillier, 2011, 439). Image: http://www.salon.com/2013/03/12/why_is_francis_underwood_a_democrat/
  70. 70. Leadership Strategy #9: Gather evidence • Test your assumptions • Test your colleagues’ assumptions • Study user behavior, formally and informally • Assess the tool and then assess it again
  71. 71. Leadership Strategy # 10: Redefining ourselves
  72. 72. References Asher, Andrew D., Lynda M. Duke, and Suzanne Wilson. “Paths of Discovery: Comparing the Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional Library Resources.” College & Research Libraries, 74.5 (2013): 464-488. Bawden, D., and L. Robinson. “The Dark Side of Information: Overload, Anxiety and Other Paradoxes and Pathologies.” Journal of Information Science 35 2 (November 21, 2008): 180-91. doi:10.1177/0165551508095781. Buck,, Stefanie, and Christina Steffy. “Promising Practices in Instruction of Discovery Tools.” Communications in Information Literacy, 7.1 (2013). Cuillier, Cheryl. “Choosing Our Futures … Still!” Journal of Library Administration 52.5 (July 2012): 436–51. doi:10.1080/01930826.2012.700806. Dempsey, Lorcan. “Thirteen Ways of Looking at Libraries, Discovery, and the Catalog: Scale, Workflow, Attention.” Educause, December 10, 2012.
  73. 73. Holman, Lucy. “Millennial Students’ Mental Models of Search: Implications for Academic Librarians and Database Developers.” The Journal of Academic Librarianship 37.1 (January 2011): 19–27. doi:10.1016/j.acalib.2010.10.003. Maglaughlin, K. L., and D. H. Sonnenwald. “User Perspectives on Relevance Criteria: A Comparison among Relevant, Partially Relevant, and Not-Relevant Judgments.” Journal of the American Society for Information Science and Technology, 2002. Matthews, Brian. “Database vs. Database vs. Web-Scale Discovery Service: Further Thoughts on Search Failure (or: More Clicks than Necessary?) (or: Info-Pushers vs. Pedagogical Partners).” Chronicle of Higher Education. Ubiquitous Librarian, August 21, 2013. Meadow, Kelly, and James Meadow. “Search Query Quality and Web-Scale Discovery: A Qualitative and Quantitative Analysis.” College & Undergraduate Libraries 19.2-4 (2012): 163–75. doi:10.1080/10691316.2012.693434. NISO ODI Working Group. National Information Standards Organization ODI Survey Report: Reflections and Perspectives on Discovery Services, January 2013.
  74. 74. Pan, Bing, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. “In Google We Trust: Users’ Decisions on Rank, Position, and Relevance.” Journal of Computer-Mediated Communication 12.3 (April 2007): 801–23. doi:10.1111/j.1083-6101.2007.00351.x. Richardson, Hillary A. H. “Revelations From the Literature: How Web-Scale Discovery Has Already Changed Us.” Information Today, May 2013. Rose-Wiles, Lisa M., and Melissa A. Hofmann. “Still Desperately Seeking Citations: Undergraduate Research in the Age of Web-Scale Discovery.” Journal of Library Administration 53.2–3 (February 2013): 147–66. doi:10.1080/01930826.2013.853493.

×