Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (13)
Similar to Ethnographically-inspired usability testing
Similar to Ethnographically-inspired usability testing (20)
Recently uploaded
Recently uploaded (20)
Ethnographically-inspired usability testing
- 1. Ethnographically-inspired usability testing Leah Rosenblum Emary i3 Conference Aberdeen, 24 June 2015
- 3. Background
- 4. Goals • Observe the user’s first experience of the catalog • Questions to ask: – What do they think? – What works? – What doesn’t work? • Optimize the design and functionality before launch (August 2015)
- 5. Test design • Ethnographically-inspired (Emary, forthcoming) – Small sample size drawn from our team’s contacts – Relationship and rapport increase the richness – Relaxed, discussion-based lab setting – Freeform test scenarios – Think aloud protocol
- 6. The test computer • 1 moderator, 1 greeter, 1 technical support • 12 participants thought aloud during individual time slots • 3 scenarios (20-30 minutes) • Video capturing gesture and affect as well as their screencast.
- 7. Testing day(s) • Everyone showed up, engaged and enjoyed themselves • Sweets, arts and crafts and discussion • Voucher for a free hot drink from Costa Coffee • Tests took place over the course of 2 weeks Reception area outside the testing room. Participants also created process maps while waiting for their test to begin.
- 8. How we managed things
- 9. Initial results • Major and minor problems led to changes in: – layout – colours – link prominence – language – advanced search • Confident that users will like the ‘look and feel’ of the new catalogue Comments and problems coded in NVivo
- 10. Initial results • The think aloud protocol is powerful • The user believing they were doing the ‘right’ thing when they had actually made an error • This happened consistently and pointed to a ‘major error’
- 11. Initial results ‘Thank you for testing us properly. So many times we’re asked for feedback on something in a way that’s not productive.’ –RA1
- 12. Challenges • Time, time, time • Qualitative data is voluminous • No undergraduate participants • Prioritizing what we change • Managing user expectations • Balancing consistency with a natural, relaxed environment
- 13. What next? Through a Jungle by andywon. https://flic.kr/p/6XrXyT Used under CC BY-NC-ND 2.0
- 14. Want to talk or reuse? • Leah Emary – l.emary@yorksj.ac.uk – @LeahEmary – LeahEmary • Design document (CC BY 4.0) https://docs.google.com/document/d/1iXlVef_1klXZvYYIjp9azXx3UwuqhA 88iOankfwQ_Ck/edit?usp=sharing • Planning document (CC BY 4.0) https://docs.google.com/document/d/1AooKerzP5w3JsPqgCWI27ZanTBBr w8rDNn9lRNoaOvw/edit?usp=sharing
- 15. Bibliography & reference list Asher A.D. and Miller S. (various) ERIAL Publications. Available from http://www.erialproject.org/publications/ [Accessed 19/11/2014]. Asher A.D. and Miller S. ([2011]) So You Want to do Anthropology in Your Library? Or A Practical Guide to Ethnographic Research in Academic Libraries. Available from http://www.erialproject.org/publications/toolkit/ [Accessed 11/11/2014]. Barnum C.M. (2010) Usability Testing Essentials: Ready, Set... Test!. Burlington, MA, Morgan Kaufmann (Elsevier). Bowler L., Koshman S., Oh J.S., He D., Callery B.G., Bowker G. and Cox R.J. (2011) Issues in user-centered design in LIS. Library Trends, 59 (4), pp.721-752. Delcore H.D., Mullooly J., Scroggins M., Arnold K., Franco E. and Gaspar J. (2014) The Library Study at Fresno State. Fresno, CA, Institute of Public Anthropology, California State University, Fresno. Emary L.F.R. (forthcoming) When (and when not) to use ethnography for LIS research. In: Priestner A and Borg M eds. UX in Libraries. Surrey, Ashgate. Available from http://ray.yorksj.ac.uk/id/eprint/641 [Accessed 17/06/2015]. Foster N.F. and Gibbons S. (2007) Studying Students: The Undergraduate Research Project at the University of Rochester. Chicago, Association of College and Research Libraries. Gabridge T., Gaskell M. and Stout A. (2008) Information Seeking through Students’ Eyes: The MIT Photo Diary Study. College & Research Libraries, 69 (6), pp.510-523. Greifeneder E. (2012) Does it Matter Where we Test? Online User Studies in Digital Libraries in Natural Environments. PhD Thesis, Institut für Bibliotheks- und Informationswissenschaft, Humboldt Universität zu Berlin. Seadle M. (2000) Project Ethnography: An Anthropological Approach To Assessing Digital Library Services. Library Trends, 49 (2), pp.370-85. Usability Consulting Services (2002) User Centered Design Services. Bloomington, IN, Indiana University.
- 16. Thank you Especially to Matt Tams for moderating and Charlotte Elwell for her technical expertise (and patience)
Editor's Notes
- Who I am, what I did, what I’m going to share with you today in 15 minutes. Have planning documents open
- Origins as a teacher training college founded in 1841 Became a University in 2006 with a diverse academic portfolio (not just teacher training) We have a strong widening participation focus Achieved RDAP in March 2015
- Changing search tool from EDS to Capita
- Synchronous, formative usability testing. If they’re interested in the types of usability tests you can run, have a look at the Barnum book at the end. The testing also included process mapping, which I don’t have time to discuss today. This was exploratory, to see if it would be a fruitful method for collecting data in future.
- What do you mean by ethnographically-inspired? Saturation is reached quite quickly, usually with 5-6 participants (Nielsen). Triangulate what people say against what they do. Not imposing structure from the beginning allows us to find errors we wouldn’t have otherwise, and truly see things as our users see them, not with our own professional blinders on.
- See end for the scenarios themselves. Very grateful to Matt Tams as moderator and Charlotte Elwell for technical expertise.
- 100% participation
- 2 person team with input from 7 others, technical assistance and equipment from the Pscychology department. Unbelievably heavy on the admin and planning. Coordinating everything and designing the tests took the most time. Actual testing was not so time consuming. Transcription and analysis were short changed. You can read more about timing, planning and project management in the documents linked to at the end.
- Think aloud protocoll is powerful but stressful. Anecdote of users thinking they had filtered by ‘format’ when really they had filtered results by ‘subject’. This is classed a major error and needed immediate fixing.
- Doing good research improves your liaison with academics and graduate researchers.
- Catalog was 2 weeks late, got it at 4PM the day before testing started ergo 24 hours in which to finalize the scenarios. Only book data was loaded. Doing my normal job on top of this project. Transcription involved not only what they said but transcribing observations as well. Particularly important because Out of the dozens of suggestions made, what’s the most important. Figuring that out in a small amount of time. Users might expect that all of their suggestions will be acted upon. Some of them were poor suggestions, some of them not possible.
- UAT 2. Recruit undergraduates in the wild when they have more time. Have a moderator who is similar to them (young, non authority figure, recent graduate?). No recording. Have them respond to the look and feel, let them have a play around/search. Brief 5 minutes.
- Time for questions at the end