Collaborative Heuristic Evaluation, UPA2010


Published on

Standard heuristic evaluation (SHE) continues to be very popular evaluation method in spite of serious criticisms of its validity and reliability. The paper proposes a new collaborative heuristic evaluation (CHE) and presents a study that shows the increased reliability and effectiveness of CHE in comparison to SHE.

Presentation given at 9:30 on Thursday 27 May by Helen Petrie and Lucy Buykx.

Published in: Technology, Education
  • Be the first to comment

Collaborative Heuristic Evaluation, UPA2010

  1. 1. Collaborative Heuristic Evaluation: improving the effectiveness of heuristic evaluation Helen Petrie, Lucy Buykx Human Computer Interaction Research Group Department of Computer Science University of York, UK
  2. 2. Standard heuristic evaluation (SHE) <ul><ul><li>Strengths </li></ul></ul><ul><ul><ul><li>Independent evaluator viewpoints </li></ul></ul></ul><ul><ul><ul><li>High volume of potential problems </li></ul></ul></ul><ul><ul><li>Weaknesses </li></ul></ul><ul><ul><ul><li>Laborious, boring, frustrating </li></ul></ul></ul><ul><ul><ul><li>Time consuming to combine problems from individual evaluators </li></ul></ul></ul><ul><ul><ul><li>High proportion of trivial problems </li></ul></ul></ul>
  3. 3. Collaborative Heuristic Evaluation (CHE) <ul><li>Method </li></ul><ul><ul><li>Evaluators work in a group </li></ul></ul><ul><ul><li>Propose potential usability problems </li></ul></ul><ul><ul><li>Create problem list in real time </li></ul></ul><ul><ul><li>Independently rate severity </li></ul></ul>
  4. 4. CHE Problem rating sheet 1 2 3 4 5  
  5. 5. Experiment method <ul><li>4 websites </li></ul><ul><ul><ul><li>airline, train, tourist information sites </li></ul></ul></ul><ul><li>2 groups of 5 usability professionals </li></ul><ul><li>2 methods: SHE and CHE </li></ul><ul><ul><li>each professional performed each method on two websites </li></ul></ul>
  6. 6. Experiment design Group 1 Group 2 CHE National Rail Visit Britain British Towns Easy Jet SHE Easy Jet British Towns Visit Britain National Rail
  7. 7. Analysis of data <ul><li>Made master problem list for SHE </li></ul><ul><ul><li>Only included problems found on pages visited by all evaluators </li></ul></ul><ul><ul><li>Relaxed matching of problems </li></ul></ul><ul><li>Master list for CHE already there </li></ul><ul><li>Analysed how many evaluators found each problem in each method </li></ul>
  8. 8. Results <ul><li>SHE: 22.9 usability problems per website </li></ul><ul><li>Pool of 250 problems on pages visited by all evaluators </li></ul><ul><li>CHE: 42.7 usability problems per website </li></ul><ul><li>Pool of 153 problems </li></ul><ul><li>(smaller pool, because of greater overlap between evaluators …) </li></ul>
  9. 9. Inter-evaluator agreement
  10. 10. National Rail website SHE CHE
  11. 11. Any-two agreement: measure of evaluator effect SHE CHE Agree it’s a problem 8.9% 91.6% Agree severity rating 36.0% n avg = 19 37.5% n avg = 38
  12. 12. Conclusion <ul><li>CHE …. </li></ul><ul><li>Exposes evaluators to each other ideas </li></ul><ul><li>Creates more agreement about problems </li></ul><ul><li>More efficient as a method </li></ul><ul><li> - creates master problem list in the session </li></ul><ul><ul><li>larger number of problems per hour </li></ul></ul><ul><li>Generates discussion/ideas for solving problems </li></ul><ul><li>More enjoyable for evaluators </li></ul>
  13. 13. <ul><li>Thank you! </li></ul><ul><li>Questions? </li></ul><ul><li>Contact us: [email_address] </li></ul><ul><li> [email_address] </li></ul>
  14. 14. Matching process <ul><li>“don’t know what overtaking trains means” </li></ul><ul><li>“don’t know what ‘include overtaking trains in journey results’ means” </li></ul><ul><li>“want to be able to select any time of day but drop down only offers 4 hour segments” </li></ul><ul><li>“specified times is well hidden and inflexible - want to go from any time” </li></ul>