USABILITY TESTING REPORT:DISCOVERY (SEARCH) TOOLS OF ENCORE AND SUMMONSAuthors: Heather Mathieson, Debbie Shultz, Nikki Kerber IDIA 642 - Lucy Holman May 12, 2010
Table of Contents Executive Summary Participants Methodology Results Recommendations Appendices Heat-map Wireframes Consent Form Faculty Screener Graduate Student Screener Script Task List
Executive SummaryThis report contains the results of a user research project conducted between April 14 and May 12, 2011, todevelop recommendations for the University System of Maryland and Affiliated Institutions (USMAI) LibraryConsortium in their consideration of several commercial discovery tools to purchase for use in USMAIlibraries. Before a product is purchased, it was determined that usability testing should be performed on thefour main tools under consideration: EDS, Primo, Summons, and Encore. Dr. Lucy Holman’s IDIA 642Research Methods class at the University of Baltimore conducted the testing as their ﬁnal project.To gain insight into each tool’s usability in terms of both negative issues and positive attributes, weconducted usability testing on Summons and Encore using four participants: two graduate students and twofaculty members. Participants selected had to be from a university or college within the state universitysystem of Maryland. To conduct the test, we used the Tobii T60 Eyetracker and Software and Morae, ausability testing and market research software, at the University of Baltimore Usability Research Lab. Thisreport concerns the ﬁndings of our usability testing.What Are Discovery Tools?Universities within the state university system of Maryland currently use Research Port, developed by theUniversity of Maryland at College Park as a front end for Federated Search. (Federated Search is aninformation retrieval technology that allows the simultaneous search of multiple searchable resources.) Theuniversity system of Maryland is planning to replace Research Port by purchasing a commercial Discoverytool.Currently, when a user logs onto a library website to search for information, he or she faces a daunting task:how to ﬁnd something doing independent searches on 75 article databases, local digital content on thelibrary server, the library catalog, and electronic collections across libraries.
Executive Summary (cont.)This is where Discovery tools come in. Discovery tools, which are middleware that is used as a searchingfront end, make searching faster and easier. By entering a single query, Discovery tools index contentsimultaneously from a variety of sources such as library catalogs, article databases, and electroniccollections. Discovery tools are available as open source, but the most robust ones come from commercialvendors.Summary of FindingsIn general, our tests showed that Summons was easier to use than Encore. Both the quantitative data and thequalitative data proved that this was true. The quantitative data, which measured such things as the taskcompletion rate, the number of searches needed to complete each task, and the ocular ﬁxations and scanpaths, indicate that participants took longer and needed more searches to complete a task in Encore thanthey did in Summons. The qualitative data, which primarily included an evaluation of participant commentsand observations of body language and facial expressions, led us to infer that participants were moreconfused and frustrated in Encore than they were in Summons.Both tools were lacking in feedback to the user, especially in cases in which a search fails because of amisspelled word. Neither tool offers adequate search hints for frustrated users. However, Encore had manymore issues of major or moderate severity, while most of Summons’ issues were minor. Perhaps the mostserious error that occurred in Encore had to do with the links to additional content that were located on theright-hand side of the page. Not one user saw these links right away and without clicking on them, searchafter search failed. The “Results” section explains the ﬁndings in more detail.
Executive Summary (cont.)One of the most surprising ﬁndings had to do with one participant’s comments as he was searching. Hespoke about how he tries to use the subject matter terms to get to the right Library of Congress catalog,indicating that he applies the traditional thought patterns of a library catalog to his search. Discovery toolsare supposed to provide a different search experience for the user than that provided by a traditional librarycatalog search. This made us question whether the two tools we tested are providing that experience, orwhether it is a case of users beginning to change old paradigms as they become more familiar withDiscovery tools.RecommendationsFor most issues, our recommendations include suggestions to reorganize screen elements to makesearching less confusing, easier, and faster. Adding more user feedback in the form of search tips and “doyou mean” prompts when a word is misspelled would make the user experience less frustrating. In somecases, a simple cosmetic change might make things clearer.
Participant Overview User 1 User 2 User 3 User 4 Gender Female Female Male Female Age 18-29 18-29 60+ 50-59 UB Affiliation Graduate Student Graduate Student Faculty Faculty Frequency >10 times per 4-10 times per Conducting >5 classes 1-2 classes month month ResearchWebsite used ﬁrst Langsdale library Langsdale library Google Google in research website website Academic Search Academic Search Academic Search Premier; ABI/ Premier; ABI/Online Databases Premier; Inform; Research Port Inform; Lexis Utilized Psychinfo PyschInfo; Nexis; Research EBSCO; Business Port Source Premier
Participants (cont.) User 1 User 2 User 3 User 4-- If none, which -- -- -- -- statement? ** Statement about online #2 #1 #2 #1 databases Online Research Good Good Good Excellent Skills Length of time 1-2 hours 31-60 minutes 0-30 minutes 0-30 minutes researching Frequency 1-3 times per 1-3 times per <1 time per visiting UB <1 time per week week week week library’s website ** Note: The statement numbers, #1 and #2, refer to the appropriate statements on the screeners. Please refer to the screener to read those statements.
MethodologyRECRUITMENTGraduate students and faculty of the University of Baltimore community were contacted aboutparticipating in a usability test of two discovery search tools. Team members, as well as Dr. LucyHolman, used a variety of ways to contact prospective participants, including but not limited to e-mailblasts, individual e-mails, the UB Students Daily Digest, the UB Faculty Daily Digest and word ofmouth. A total of four participants who were all affiliated with the University System of Maryland werefound to participate in the usability test. Two participants were UB faculty members and twoparticipants were UB graduate students; all were over the age of 18. Participants were tested insessions that ran approximately 60 minutes each.PREPARATIONPrior to the test dates, the team reviewed two discovery tools, Encore and Summons, to determine howeach tool worked. A screener, test-script, and consent forms were written and duplicated for eachparticipant. Projects were created in each of the testing software tools, Tobii and Morae.TESTING PROCEDUREEach testing procedure began with introductions and a description of the project. The participant wasasked to ﬁll out an informed consent form. A screener was also given to the participant to collect userproﬁle information (as shown in the proﬁle matrix in the previous section). Each of the fourparticipants was asked to perform ﬁve tasks in each of the two discovery tools, Encore and Summons.The speciﬁc tasks included combinations of searching for articles, books, and audio-visual materials.A Task List is included in the Appendix, pages 48-50.
Methodology (cont.)Participants were met and welcomed into the reception area of the Usability Lab. The members of the grouptook turns being the moderator for each participant. Each participant was set up in the testing room andlogged into the system. If Tobii was used, it was calibrated to the user’s eye movements.During the test, the moderator followed a written script and emphasized two important points at thebeginning and throughout the test: ﬁrst, that we were testing the discovery tools and not the user or theuser’s ability to succeed; and second, that because of the test environment, participants might not be able tofully complete a task. Participants were also encouraged to talk aloud and comment on anything theywanted to during the test.The other two team members sat in the Observation Room behind a one-way mirror. They took notes,observed the users’ actions, and made note of users’ comments. They also recorded any quantitative data.After the test, participants were asked to summarize their reactions to the two Discovery tools and provideus with any additional feedback that could help us in our evaluations.
Methodology (cont.)CRITERIA OF USABILITYWe used the following criteria as a measure of the usability in Encore and Summons:Quantitative Measures Measure of time participants needed to complete each task Task completion rate Number of searches needed to complete each task Average number of searches needed to complete each taskQualitative Measures Conversations with participants after the usability test and answers from the screener questionnaires Comments from participants Facial expressions Body language
11 Results Pages 15-24 provide an extensive results overview. Please refer to the Task List, located in the Appendix, pages 48-50, to get acquainted with all of the tasks we asked our users to complete.
Results Features of Encore and Summons that Support Users Tool Good Usability Practices Current Features of WebBoth Tools Allow users to use search criteria in a logical way Left side bar with various search options to further narrow search results produced by a displayed when searching articles in Encore. basic search. This paradigm will ﬁt users’ mental models of the search process.
Results Features of Encore and Summons that Support Users Tool Good Usability Practices Current Features of WebSummons Allow users to select as much criteria for an Material Type ﬁeld on the Advanced Search advance search with as little clicks or selections as screen includes an option that combines books possible. This way users will spend less time and journals together. searching and more time looking through meaningful search results. Encore Provide a clean and simple user interface, clear of Encore provides a clean and simple user advertisements, banners, or other added features interface that is fairly intuitive to navigate. that do not provide a purpose in the search process.
Results Features of Encore and Summons that Support Users Tool Good Usability Practices Current Features of WebSummons Give users plenty of search criteria options in Summons offers a nice Advanced Search form. order to help narrow their search from the beginning.
Results Error Severity The most critical level - The user was unable to correctly complete Major task. Moderate Signiﬁcant problems caused for the user. Minor annoyances that slow the user down during the tasks. The user Minor leaves frustrated. Good Indicates that an object is well designed.
Results Tool Issue Results Level FrequencyEncore Users were blind in most cases It took most users anywhere from 5 to 10 Major 4 to ﬁnding and/or clicking on minutes to locate this link. One user never “Other Sources” (WorldCat and found it. Until the user located and clicked Link+) “Other Sources,” searches failed.Encore Tool does not clearly show how The user could not understand why certain Moderate 1 user failed it uses search terms results were listed. The results list did not to complete highlight keywords to see why certain data task; was included. Sometimes data was included that did not appear to contain any of the 3 users had search terms. signiﬁcant problemsEncore Irrelevant search results Search results sometimes seemed completely Moderate 2 irrelevant to the search terms entered. The “Did you mean” entries often appeared to be somewhat or very unrelated to what the user was looking for.Encore Tool manipulates keywords It appeared that the tool treated the whole Moderate 4 search string as individual keywords and not speciﬁcally as Title, Author, or Subject.
Results (cont.) Tool Issue Results Level FrequencyBoth Both tools are unforgiving of When a user misspelled a word, the search Moderate 2Tools spelling errors. failed. No feedback was provided to bring the misspelling to the user’s attention. Unless the user noticed the misspelling and corrected it, additional searches could also fail.Both Neither tool offers search hints Encore did not provide any assistance or Moderate 4Tools for frustrated users suggestions to users when a search failed. It did not offer any suggestions as to how the user could get an unmanageable list (too many items) down to a manageable list. Summons offered search tips but they were below the fold of the Advanced Search page and users did not always see them.Both Tool offers multiple ways to Clicking in various search ﬁelds produced Moderate 4tools search for items. search results pages each with a different look/feel and with different ﬁltering options. Navigating through these search options required the user to click multiple times.
Results (cont.) Tool Issue Results Level FrequencyEncore Links for Catalog, Images, and Placement of these links at the very top of Minor 2 Articles are hard to ﬁnd and the screen made them hard to locate. Their behavior can be confusing behavior is sometimes puzzling. For example, when the user clicks Articles, then clicks Advanced Search, the Articles link disappears and only the Catalog link displays, making the user wonder whether the search is actually being performed on the catalog.Summons Users appeared to hesitate when clicking The tool has two Submit this button as if unsure of which one to use. buttons on the Advanced Minor 3 Some clicked the top button; others clicked Search form. the bottom. Not obvious when the tool Because users did not see the “No entries displayed “No entries found” on found” message, they were unsure whetherSummons Minor 2 the top of the Advanced Search the search was still processing or had form. ﬁnished.Encore Confusing search results Search displayed “No Catalog Results Minor 1 Found,” at the top of the page; at the bottom of the page, it displayed a whole series of results (in one case over 40,000 articles).
Results (cont.) Tool Issue Results Level FrequencySummons Not obvious when the tool Because users did not see the “No entries Minor 2 displayed “No entries found” on found” message, they were unsure whether the top of the Advanced Search the search was still processing or had form. ﬁnished. Both Screens in both tools were It was not always clear where users should Minor 3 tools cluttered, with options for go when starting a search. searching on the left-hand side, the center, and sometimes on the right-hand side of the page. Both Discovery tools are not Participant tried to use subject matter terms Minor 1 tools providing a different search to get to the right Library of Congress experience catalog, applying the traditional thought patterns of a library catalog to his search. Discovery tools are supposed to provide a different search experience for the user than that provided by a traditional library catalog search.
Results Number of searches for each task Encore Summons Encore Summons User 1 User 2T1 6 1 T1 6 2T2 7 1 T2 6 2T3 4 2 T3 4 2T4 4 1 T4 5 1 Encore Summons Encore Summons User 3 User 4T1 10 1 T1 7 2T2 10 2 T2 5 1T3 9 6 T3 6 4T4 3 2 T4 10 1 Overall, users had the most problem with successfully completing task 1 in Encore and task 3 in Summons
Results How successful were users in completing tasks? Participant completed all of the components of a certain task successfully. Complete Example: Participant found many articles on Artiﬁcial Intelligence that were no more than ﬁve years old and are peer-reviewed. Participant only completed some of the components of a certain task. Incomplete Example: Participant successfully found an article but did not save an article for retrieval. Participant was not able to complete any portion of the task. Failed Example: Participant did not ﬁnd a known item: Beowulf
Results How successful were users in completing tasks? Failed Tasks 15% Failed Tasks 35%Completed Tasks 45% Incomplete Tasks Completed Tasks 25% 60% Incomplete Tasks 20% Encore Summons
Results Average Time on Task in Encore and Summons Based on our findings, Task 2 in Encore gave users the most problem. It took an average of 7:50 minutes for users to complete the task or end the task before asking to move to the next task. Similarly, it took users an average of 5:15 minutes to complete or end Task 3 in Summons.
Results Time on Task in Encore and Summons User 1 User 2 User 3 User 4 Encore Task 1 4:31 2:44 5:40 2:51 Task 2 7:39 5:29 14:25 3:49 Task 3 1:30 3:49 9:50 4:45 Task 4 4:43 4:39 4:31 3:27 Task 5 0:14 Summons Task 1 0:42 0:48 1:44 2:44 Task 2 0:40 0:42 4:03 0:50 Task 3 2:41 2:40 7:18 8:21 Task 4 1:29 2:14 6:16 2:22 Task 5In general, it took users much more time to complete or end tasks in Encore than in Summons. Interestingly, Task 2 inEncore gave users the most problem, as their completion times are higher.In Summons, users were much more successful in completing tasks. Interestingly, Task 3 gave users the most troublewith time rates varying between 2:40 - 8:21 minutes, however these times do not compare to the problems Task 2gave users in Encore. Comparing Task 2 in Encore and Summons, users cut their time rate by more than half.
Interesting ObservationsThe observations described below include those things that we felt were important or interesting enough tomake a note of while we were conducting our testing. • Only one participant found the + button in the Format ﬁeld of the Advanced Search screen in Encore. • Only one participant typed in “peer reviewed” as a search term in the advanced search criteria. The other participants were unsuccessful in ﬁltering articles by this criteria because there was no speciﬁc option and it didn’t occur to them to include it as a search term. • One participant had no successful searches in Encore because this participant never located the “Get More Results in: Link+ and WorldCat” on the right-hand side of the screen. It appeared from our observations that the right-hand side of the screen was totally ignored during the usability testing for this tool with this participant. • Only one participant was successful in saving the article for retrieval in Task 5. The other participants found the article but did not save. • Participants who used a basic search and then narrowed their search results using additional search criteria seemed to have better results with both tools (provided that they had already clicked Link+ or WorldCat in Encore) than those who started with Advanced Search.One of the most surprising ﬁndings we came across in our usability testing had to do with one participant’scomments as he was searching. He spoke about how he tries to use the subject matter terms to get to theright Library of Congress catalog, indicating that he applies the traditional thought patterns of a librarycatalog to his search. Discovery tools are supposed to provide a different search experience for the userthan that provided by a traditional library catalog search. This made us question whether the two tools wetested are providing that experience.
RecommendationsEncore Recommendations Highlight words in results page to show user how terms are being used. All results should appear below the primary search ﬁeld. Use better labeling such as “Get More Results in: Link+ and WorldCat.” If no search results are found, this should be located where the results would be. If results are found, move this to the top left sidebar where there is a higher probability a user would look for more information on the page.Summons Recommendations Have only one Submit button on Advance Search Form page. Make the words “No Entries Found” in a larger font size and in the color red, located below the search ﬁelds where results would normally appear, so that user has a clear understanding that their search returned zero results and that the search has completed.Common Recommendations Have a Modify Search Link below the search ﬁeld to edit search or make changes to the search directly from the Search Results page. Have a list of the top search term suggestions. Spelling suggestions are needed so that a search does not always result in “No Entries Found”. Clear call-out button for help or instructions on how to use the discovery tool. Perhaps a video tutorial or PDF guide on how to use the system. Help link/button should also be clearly labeled on all internal web pages.
Overall Recommendation Should a purchasing decision need to be made between Encore and Summons, our group feels that Summons would be the better discovery search tool for the graduate students and faculty members of University of Baltimore, and as a whole for the University System of Maryland and Affiliated Institutions (USMAI). While some changes should be made to Summons before it is integrated into USMAI, we feel that these changes are minor in comparison to the issues Encore would need to address. The results show that many users felt Summons was a lot easier to use compared to Encore. Users successfully completed 60% of the ﬁve tasks in Summons versus users only completing 45% of the tasks in Encore. The overall failure rate of successfully completing tasks shows an even greater disparity; users testing Summons only had a 15% failure rate compared to Encore’s 45% failure rate. We feel that if our recommendations were taken into consideration, the failure rate in Summons would drop signiﬁcantly.
29 Heat Map Heat Maps show how long each part of the screen has been looked at, as well as the main areas of the website users looked at and focused on.
Heat-MapsThese heat maps show that users have a right-side “blindness” in both Encore and Summons. In otherwords, most users did not notice that there was a portion on the right side bar in Encore that said“Other Sources”. We assume this “blindness” occurs because advertisements and other bannersnormally appear on the right hand side on webpages. On average, it took participants approximately1:34 minutes to ﬁnd “Other Sources” in Encore. User 4 in particular never even found the “OtherSources” side bar. A similar pattern of “blindness” was also found in Summons. For example, none ofour participants looked to the right sidebar in Summons to see what features or links were available.User 2 even commented that her ﬁeld of vision was in the center of the screen and to the left.
WireframesThe following wireframes were created in response to our recommendations, based on both ofthe discovery search tools we tested. We envisioned a discovery search tool that incorporatesall of our recommendations. Our wireframes also include the positive features we found inboth systems.
ScriptIntroductionM: Good <time of day>, thank you for making time in your schedule to participate today. My name is <name>, and I am a part of a groupworking on a research study to learn about discovery (search) tools for the University System of Maryland and Affiliated Universities (USMAI).We have asked you here <time of day> to help us evaluate two discovery tools through your interaction with them.At a basic level, a discovery tool is library search engine that allows you to search multiple databases, along with the librarys catalog in onesearch interface. During today’s session, you will be using <name discovery tool> and <name discovery tool> to complete some basic tasksthat a <student/faculty member> might carry out during their search process.Please realize that we are not testing your ability. Instead, we’re testing the effectiveness of the tool. Do you feel comfortable in proceeding?P: Participant acknowledgesM: Very good, if you will, please sign our release form here. It says: “I, the undersigned, agree to be part of a usability study conducted at the University of Baltimore. As a participant. I agree to be videotapedand to have my activities on the computer recorded. I allow my comments and observations about my experiences to be become part of theﬁndings of the usability study.”P: Participant signs release form.M: Thank you. Before we begin with the usability test, we could like you to ﬁll out this basic questionnaire in order to get an idea of yourbackground in searching online databases and online search engines for resources such as books, articles, or audiovisual items. (Hands theparticipant the screener).P: Participant ﬁlls out screener and hands it back to the moderator.M: Thank you.
Script (cont.)System Introduction and Calibration (Tobii)M: In order to start the usability test we need to set up the eye-tracking software. This computer system is equipped with eye-tracking software, which is a device for measuring eye positions and eye movement which is helpful in analyzing how you use thediscovery tools. Before we begin, we will need to set up a session in the software. Would it be OK if I used your ﬁrst name toidentify this session?P: Participant responds.M: O.K. <name of user>.The next step is to calibrate the eye tracker to your eyes. During this calibration and throughout this session, please relax andcontinue to look at the screen. While we will be conversing throughout the session, please try to keep your focus on the screenrather than looking at me. Do you have any questions?P: Participant responds.M: OK, here we go… (start and run through calibration)FAILUREM: OK, the system had a little trouble picking up all the information it needed for calibration. I would like to recalibrate the areasof the screen on which the system had some trouble. Are you ready?P: Participant respondsM: OK – Here we goSUCCESSM: Excellent! I will now open the ﬁrst discovery tool.
Script (cont.)Alternate paragraph if using Morae:(Will the Morae session be set up prior to beginning the test, or will we have to set it up before the user starts working with thetool?)M: The computer system you will be working on today is equipped with Morae software, which is used for usability testing anduser experience research. It records your interactions with the computer so they can be analyzed after your session. I’m nowgoing to set up a session in the software. Would it be OK if I used your ﬁrst name to identify this session?P: Participant AgreesFamiliarizationM: Here is the ﬁrst discovery tool you will be working with. As you can see, <name of tool> includes options that allow you tosearch library databases for books, articles, and other types of materials.Overview of TasksM: To gauge the usability of the tool, we will be asking you to work your way through some tasks that we would expect a<student or faculty member> to do. These will be tasks that you may or may not have done before.Because of the test environment, you may not be able to FULLY complete a task and may be stopped at the point of a login orclick a link that is unavailable. If you can’t ﬁnd something or if it doesn’t make sense, just tell us, and we can inform librarianswho are making decisions about the tools.Remember: We are testing the discovery tools, not you. There is no right or wrong answers, no right or wrong way to do things.Every action you take, no matter how you may feel about how it turns out, helps us to evaluate the product. Do you have anyquestions before we begin?P: Participant responds
Script (cont.)Tasks – Discovery Tool 1Insert Discovery Tool Tasks and Scenarios.Tasks- Discovery Tool 2M: Here is the second discovery tool that you will be working with. As you can see, <name of tool> includes options that allowyou to search library databases for books, articles, and other types of materials.To gauge the usability of this tool, we will be asking you to work your way through the same tasks that you did using the previoustool. Again, because of the test environment, you may not be able to FULLY complete a task and may be stopped at the point of alogin or click a link that is unavailable. If you can’t ﬁnd something or if it doesn’t make sense, just let us know. Do you have anyquestions before we begin?P: Participant respondsInsert Discovery Tool Tasks and Scenarios.ConclusionM: Those are all of the tasks we have today. Thank you for your assistance in evaluating the discovery tools; we appreciate yourtime completing them for us. Your participation today will help us make recommendations that can make a difference to the UBcommunity. Do you have any further questions about the study before you leave today?P: Participant responds.M: (Answers questions if any are asked). Once again, thank you for spending your (morning/afternoon) with us today. As a thankyou, please accept this gift of appreciation. (Hand over $10 Starbucks gift card)
Task List Task Script1. Find Your professor has asked you to read the Seamus Heaney translation of Beowulf. Find out if a copy isknown item available in the library and, if so, where it is located. If a copy is NOT available, to check out, are(book) there any other options for obtaining or accessing the book? Alternate wording for faculty: You are interested in reading Seamus Heaney’s translation of Beowulf. Find out if a copy is available in the library and, if so, where it is located. If a copy is NOT available, to check out, are there any other options for obtaining or accessing the book?2. Find Your professor suggested that you consult an article entitled “Modernism and the Harlem Renaissance” thatknown item was published in American Quarterly for a paper you’re doing. You want to see if it’s available full-text from the(article) & save library’s databases. Once you have found the article, can you save that information for later use?article citation Alternate wording for faculty: There is an article you recall seeing in American Quarterly on “Modernism and the Harlem Renaissance” that you think you might assign as a class reading. You want to see if it’s available full-text from the library’s databases. Once you have found the article, can you save the information to pass on to your students? (Full citation, do not give to subject: Baker, Houston A. (1987). Modernism and the Harlem Renaissance. American Quarterly, 39 (1), 84-97.)3. Conduct You are interested in learning basic Portuguese for an upcoming trip to Brazil. You want to see if the library hastopic search for any elementary Portuguese language textbooks or audiovisual materialsprint or AVmaterials
Task List Task Script4. Conduct a You have to research Artificial Intelligence for a project, and your professor has asked you to use a variety oftopic search for resources including books and journal articles. The articles should be no more than five years old and be peer-all types of reviewed. As a preliminary assignment she wants you to report on how many relevant books and articles youmaterials found. Alternate wording for faculty: You are researching recent developments in Artificial Intelligence. As a preliminary step, you want to compile a list of relevant books and peer-reviewed articles from the last five years that are available through your library.5. Retrievearticle You’re ready to use the Modernism article you found earlier, and you want to retrieve it.