Collaborative heuristic evaluation - masters project presentation
•
2 likes•2,168 views
A short presentation given in Sept 2009 to finish my masters degree. The project experiment found that evaluators working collaboratively could identify more usability problems and reach a significantly higher level of inter-evaluator agreement than using the traditional method of heuristic evaluation.
Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
Similar to Collaborative heuristic evaluation - masters project presentation
Similar to Collaborative heuristic evaluation - masters project presentation (20)
More from Lucy Buykx
More from Lucy Buykx (7)
Recently uploaded
Recently uploaded (20)
Collaborative heuristic evaluation - masters project presentation
- 1. Improving heuristic evaluation through collaborative working Lucy Buykx
- 2. “discount usability engineering” Jakob Nielsen, 1992
- 3. Heuristics
- 4. John 38
- 5. Alesha 25 John 38
- 6. Usability problems 56
- 7. Carolyn 29 56
- 8. Usability problems 29 2 3 78 1 7
- 9. Nielsen & Landauer, 1993
- 10. Still very popular • Usability scores for Supermarket websites Chen 2005 • Generate user protocols for medical equipment Zhang 2003 • Comparing library websites Peng 2004
- 11. so what is the question?
- 12. Usability problems 29 2 3 78 1 7
- 13. Carolyn 29 2 3 John 1 Alesha 31 7 18
- 14. Inter - evaluator agreement is very low
- 15. Standard Heuristic Evaluation (SHE) “each individual evaluator inspect the interface alone...... to ensure independent and unbiased evaluations from each evaluator’’ Nielsen 1994
- 16. Collaborative Heuristic Evaluation (CHE) • Collaborative inspection to find potential problems • Individual evaluation with secret voting
- 17. Experimental design Group 1 Group 2 National Rail Easy Jet CHE Visit Britain British Towns Easy Jet National Rail SHE British Towns Visit Britain
- 18. Research questions • find same problems ? • find more severe problems ? • find more reliable problems ? • use heuristics better ?
- 19. Research questions • find same problems ? - NO • find more severe problems ? - maybe • find more reliable problems ? - YES ! • use heuristics better ? - NO
- 20. SHE results confirm previous findings
- 21. CHE findings
- 22. SHE CHE 60 45 Problems found 30 15 0 1 2 3 4 5 Number of evaluators
- 23. What causes effect? • Group Think ? (Janis 1982) • Social loafing ? (Karau 1995)
- 25. Severity ratings • Unlikely to be influenced the same way • Show high inter-evaluator agreement • 24-48% agreement • 76-95% within one agreement
- 26. Conclude : the effect is real
- 27. Future research • Explore social processes • e.g. remote voting • Explore how experts perform their role • e.g. compare heuristics & severity between experts • Use it to train & evaluate novices
Editor's Notes
- scenarios + think aloud user testing + heuristic evaluation
- 10 heuristics - available on his website useit.com So how does it work... take John
- then Alesha
- she finds 25, merge them together...
- and makes 56 ... add Carolyn
- she finds 29 and merge them together...
- and fab we have 78
- Lots of studies and found 50-75% of problems at 5 evaluators / users had advantage of knowing how many problems there were
- Add severity ratings of all problems in each heuristic, weight them then sum to find global score (Sainsburys won!) Infusion pumps cant be changed but can spot the problems and help ensure people dont make them again
- been going so long, what is left to ask? back to the 78 usability problems...
- ..they are made up of three individual sets
- most found by just one evaluator If you were making a change based on problem found by one would you feel confident? Was it overlooked by the other 4 (assuming we have quorum) or was it rejected as a problem by 4?
- Confirmed in the literature - Gilbert Cockton and Alan Woolrych Sunderland Problem matching - Ebba Hvannberg and Effie Law Why? work independently so see different pages, or in different ways
- Together in same room, see the same interface at the same time And keep the unbiased approach with no discussion and secret voting
- 2x2 individual differences - superstar evaluators compare problems, two conditions same website
- imagine 6th evaluator - dont expect to overlap much, and viewed different pages 2 way ANOVA CHE vs SHE - but most found by only one so question the premise Vague heuristics question - no pattern of usage, very little agreement
- National Rail most problems found by only 1 evaluator, none by all
- Stoven Karau - good round up of studies Irving Janis - group think - discussion comes to wrong conclusions, but we banned discussion Social loafing - he said it was so I’m just gonna agree,
- Could not loaf on their decision of severity rating & could not know what other evaluators were voting