Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to IIIF Pre-conference - Usability testing conducted on the UV and Mirador
Similar to IIIF Pre-conference - Usability testing conducted on the UV and Mirador (20)
Recently uploaded
Recently uploaded (20)
IIIF Pre-conference - Usability testing conducted on the UV and Mirador
- 1. The International Image Interoperability Framework (IIIF): raising awareness of the user benefits for scholarly editions Results of the usability testing conducted on the UV and Mirador for my bachelor’s thesis Julien A. Raemy, LIS Student (HEG-ID) 2017 IIIF Conference – The Vatican
- 2. Background Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 3. Context (1) • IIIF Wonder • Heard of the IIIF initiative at the University of Basel’s Digital Humanities Lab • IIIF Working Groups Meeting in the Hague in October 2016 • Appraised the potential of IIIF-compliant viewers such as the Universal Viewer and Mirador • IIIF 3rd Goal • To develop, cultivate and document shared technologies, such as image servers and web clients, that provide a world-class user experience in viewing, comparing, manipulating and annotating images. Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 4. Context (2) •Library and Information Science (LIS) Bachelor’s thesis: February-July 2017 •NIE-INE – National infrastructure for (scholarly) editions in Switzerland • Project funded by Swissuniversities (like e- codices) • Coordination conducted at the University of Basel Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 5. Objectives 1. Writing a comprehensive description of the International Image Interoperability Framework (IIIF) for potential new implementers. 2. Conducting usability tests to show the benefits of the UV and Mirador in terms of efficiency and satisfaction. 3. Assessing the interests of deploying and using IIIF-compliant technologies for complex and large scientific editions. Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 6. Usability versus user experience • Usability • A system which makes users able to achieve their goals in a specified context • Learnability, Efficiency, Memorability, Few errors, Satisfaction (Jakob Nielsen, Usability Engineering, 1993) • UX • ‘Everything that touches upon someone’s experience with a product’ (Don Norman) • Much broader concept than usability Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 7. Usability testing Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 8. Scenario and tests • A scenario that comprised tasks and surveys • on the UV • on Mirador • An A/B session • IIIF Sandbox (or ‘SandbUX’) on GitHub. Installation on a server for the duration of the tests • UV 2.0.1 – Mirador 2.3.0 • Remote usability tests with Loop11 and in-person tests with Morae Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican Assessed!
- 9. Measurement approach (1) • Satisfaction • Expectations, Fun, Pleasure • 3 criteria from Usefulness, Satisfaction, and Ease (USE) – Arnold M. Lund, 2001 • Average grade from 1 to 5 based on a Likert scale • Efficiency • Overall relative efficiency – Anton Sergeev, 2010 • Ratio of the time taken by the participants who successfully completed a given task in relation to the total time taken by all participants Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 10. Measurement approach (2) • Perceived usability • System Usability Scale (SUS) – John Brooke, 1986 • 10 items (positive and negative statements) on a Likert scale • Giving a score from 0 to 100 (percentile ranking) • Interpretation by giving the score a letter-grade (A+ to F) – Jeff Sauro, 2011 • A/B testing • Observational task followed by a survey • 7 aspects tested (size and choice of icons, metadata presentation, overall aesthetic, etc.) • 4 possibilities: UV, Mirador, Both are equally good, neither Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 11. Pilot test •Remote-ish test conducted at the HEG on the 5th April with 29 LIS students during a user-centred design course •Scenario carried out in French •To test the test •Buffering issues with the first task on Mirador Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 12. Target test •Remote test conducted between the 20th April and the 8th May 2017 with the broader DH community: 45 participants •Dissemination of the URL on Slack and Twitter. Discussed during a IIIF Community Call (26th April) •The test was not accessible for 2 days Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 13. In-person test • Tests conducted between the 25th April and the 10th May 2017. The thinking aloud protocol was applied. • Recorded sessions (voice and screen) with the target audience. 7 people participated. • Slight modifications of the scenario/measurement approach • No SUS to make it shorter • Task completion instead of efficiency Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 14. Results and findings Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 15. Feedback + • Intuitive and simple • Search within • Share and embed • Download – • Doesn’t offer enough possibilities for researchers • Confusion around numbering (image/page) • The pinpoint Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 17. Feedback • Powerful • Comparison* • Annotation • Image manipulation • Finding how to rotate an image • Confusion between Change Layout and View type • Pan and zoom controls Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican + –
- 20. Outcomes •Both interfaces are overall satisfying or very satisfying •The UV is highly efficient and simple to use •Mirador is efficient and powerful •Their perceived usability ranks from average to excellent Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican
- 21. Aggregated results Julien A. Raemy @julsraemy 2017 IIIF Conference – The Vatican Pilot test Target test In-person test UV Mirador UV Mirador UV Mirador Satisfaction (out of 5) 3.75 3.49 4.26 3.81 4.19 3.67 Efficiency 88.9% 73% 100% 82% - - Task completion - - - - 89.29% completed with ease 10.71% completed with difficulty 0% failed to complete 42.86% completed with ease 50% completed with difficulty 10.71% failed to complete SUS 72.76 (B-) 64.05 (C-) 86.33 (A) 74.67 (B) - - A/B UV: 6/7 UV, Mirador, and Both: 1/7 UV: 2/7 Mirador: 1/7 Both: 4/7 UV: 2/7 UV and Both: 3/7 Both: 2/7
Editor's Notes
- Hi everyone, I’m Julien Raemy and I am Library and Information Science student in Geneva. I’m very happy to be here to talk about the usability research I did on the Universal Viewer and Mirador for my bachelor’s thesis.
- First, some background information
- I heard about IIIF last year during an internship at the University of Basel. I thought it was a very interesting inititiative but still a bit vague to me, so I decided to go to the Working Groups Meeting in the Hague where I assessed the potential of the IIIF ecosystem. IIIF Website – 3rd Goal
- As a student in my last semester, I've got to do a bachelor's thesis. I started this assignement in February and will be done next month. I already thought to take IIIF as a research topic right after the WG meeting and... when I talked about it to one of my professors, he told me to include the NIE-INE project as my target audience. NIE-INE aims to build a national platform in Swizerland for scholarly editions.
- When my research topic was validated, I desined a document where I laid out 3 objectives... 1) Write a comprehensive description of IIIF. Especially in Switzerland where e-codices is the only IIIF-compliant collection available 2) Usability testing on the UV and Mirador. 2 attributes: efficiency and satisfaction 3) NIE-INE Assessment
- 68 is considered as average – C
- SUS: enough inputs from remarks and interactions
- Some results and feedbacks that I received. + 1 video + heat maps
- Here is a recorded task where participants had first to go to a specific page. Almost all participants during the in-person tests tried to use the pinpoint to navigate through digital assets. They were quite frustrated that I didn't work as they expected.
- As you may have noticed, the specificity of each viewer was greatly appreciated. However, I must say that adding a new item is also the feature that was the most problematic during the usability test.
- Heat maps – Pilot test Have a look at the Pan and zoom controls
- Heat maps – Target test
- Even if it might be a bit too much of information, I wanted to show you some figures. My thesis and the full results will of course be availabe as soon as it’s been reviewed. The target test gave both viewers better results (which is a good sign)