Collaborative Heuristic Evaluation, UPA2010

1,796 views

Published on

Standard heuristic evaluation (SHE) continues to be very popular evaluation method in spite of serious criticisms of its validity and reliability. The paper proposes a new collaborative heuristic evaluation (CHE) and presents a study that shows the increased reliability and effectiveness of CHE in comparison to SHE.

Presentation given at 9:30 on Thursday 27 May by Helen Petrie and Lucy Buykx.

Published in: Technology, Education
  • Be the first to comment

Collaborative Heuristic Evaluation, UPA2010

  1. 1. Collaborative Heuristic Evaluation: improving the effectiveness of heuristic evaluation Helen Petrie, Lucy Buykx Human Computer Interaction Research Group Department of Computer Science University of York, UK
  2. 2. Standard heuristic evaluation (SHE) <ul><ul><li>Strengths </li></ul></ul><ul><ul><ul><li>Independent evaluator viewpoints </li></ul></ul></ul><ul><ul><ul><li>High volume of potential problems </li></ul></ul></ul><ul><ul><li>Weaknesses </li></ul></ul><ul><ul><ul><li>Laborious, boring, frustrating </li></ul></ul></ul><ul><ul><ul><li>Time consuming to combine problems from individual evaluators </li></ul></ul></ul><ul><ul><ul><li>High proportion of trivial problems </li></ul></ul></ul>
  3. 3. Collaborative Heuristic Evaluation (CHE) <ul><li>Method </li></ul><ul><ul><li>Evaluators work in a group </li></ul></ul><ul><ul><li>Propose potential usability problems </li></ul></ul><ul><ul><li>Create problem list in real time </li></ul></ul><ul><ul><li>Independently rate severity </li></ul></ul>
  4. 4. CHE Problem rating sheet 1 2 3 4 5  
  5. 5. Experiment method <ul><li>4 websites </li></ul><ul><ul><ul><li>airline, train, tourist information sites </li></ul></ul></ul><ul><li>2 groups of 5 usability professionals </li></ul><ul><li>2 methods: SHE and CHE </li></ul><ul><ul><li>each professional performed each method on two websites </li></ul></ul>
  6. 6. Experiment design Group 1 Group 2 CHE National Rail Visit Britain British Towns Easy Jet SHE Easy Jet British Towns Visit Britain National Rail
  7. 7. Analysis of data <ul><li>Made master problem list for SHE </li></ul><ul><ul><li>Only included problems found on pages visited by all evaluators </li></ul></ul><ul><ul><li>Relaxed matching of problems </li></ul></ul><ul><li>Master list for CHE already there </li></ul><ul><li>Analysed how many evaluators found each problem in each method </li></ul>
  8. 8. Results <ul><li>SHE: 22.9 usability problems per website </li></ul><ul><li>Pool of 250 problems on pages visited by all evaluators </li></ul><ul><li>CHE: 42.7 usability problems per website </li></ul><ul><li>Pool of 153 problems </li></ul><ul><li>(smaller pool, because of greater overlap between evaluators …) </li></ul>
  9. 9. Inter-evaluator agreement
  10. 10. National Rail website SHE CHE
  11. 11. Any-two agreement: measure of evaluator effect SHE CHE Agree it’s a problem 8.9% 91.6% Agree severity rating 36.0% n avg = 19 37.5% n avg = 38
  12. 12. Conclusion <ul><li>CHE …. </li></ul><ul><li>Exposes evaluators to each other ideas </li></ul><ul><li>Creates more agreement about problems </li></ul><ul><li>More efficient as a method </li></ul><ul><li> - creates master problem list in the session </li></ul><ul><ul><li>larger number of problems per hour </li></ul></ul><ul><li>Generates discussion/ideas for solving problems </li></ul><ul><li>More enjoyable for evaluators </li></ul>
  13. 13. <ul><li>Thank you! </li></ul><ul><li>Questions? </li></ul><ul><li>Contact us: [email_address] </li></ul><ul><li> [email_address] </li></ul>
  14. 14. Matching process <ul><li>“don’t know what overtaking trains means” </li></ul><ul><li>“don’t know what ‘include overtaking trains in journey results’ means” </li></ul><ul><li>“want to be able to select any time of day but drop down only offers 4 hour segments” </li></ul><ul><li>“specified times is well hidden and inflexible - want to go from any time” </li></ul>

×