Rapid Iterative Testing and Evaluation


Published on

Published in: Economy & Finance, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • I manage the Games User Research Group at Microsoft Michael is the inventor of RITE and is currently the chief usability for Internal HR tools Melissa user researcher in MGS and works with the FASA studio – Mechassault
  • Rapid Iterative Testing and Evaluation

    1. 1. Rapid Iterative Testing and Evaluation RITE Dennis Wixon Michael Medlock Melissa Federoff
    2. 2. Outline of Talk <ul><li>What’s wrong </li></ul><ul><li>What’s RITE </li></ul><ul><li>Creation of the RITE method </li></ul><ul><li>Running a RITE study </li></ul><ul><li>Conclusions </li></ul><ul><li>Q/A </li></ul>
    3. 3. Editorial: Research on Methods
    4. 4. Why
    5. 5. What’s wrong <ul><li>Purpose: the goal of a method is to find problems </li></ul><ul><li>Cost: determine the minimal number of participants </li></ul><ul><li>Evaluation: we should use traditional research methods to determine the “best” method </li></ul><ul><li>Assumption: there is a best method </li></ul>
    6. 6. What’s right? <ul><li>Purpose of an iterative test </li></ul><ul><ul><li>Rapid improvements </li></ul></ul><ul><ul><li>Create a collaborative environment </li></ul></ul><ul><li>Measure cost and benefit: </li></ul><ul><ul><li>Percentage of problems addressed </li></ul></ul><ul><ul><li>Percentage of effective fixes </li></ul></ul><ul><li>Evaluation: use case studies </li></ul><ul><li>Match method to purpose </li></ul><ul><li>Position user research as a business asset </li></ul>
    7. 7. What is a Rapid Iterative Test and Evaluation test (RITE)? <ul><li>Just like a “standard” usability test except: </li></ul><ul><ul><li>Changes made as soon as a problem is identified and a potential solution is clear </li></ul></ul><ul><ul><li>The changed interface is tested with subsequent users </li></ul></ul><ul><li>This is the Essence of RITE </li></ul>
    8. 8. “Standard” vs. RITE
    9. 9. What is the business case for RITE? <ul><li>Return on investment </li></ul><ul><ul><li>Faster detection of problems </li></ul></ul><ul><ul><li>More issues found </li></ul></ul><ul><ul><li>More issues fixed </li></ul></ul><ul><ul><li>Fixes are verified </li></ul></ul><ul><li>Better team dynamics </li></ul><ul><li>The method degrades gracefully </li></ul>
    10. 10. Case Study: Age of Empires II <ul><li>What is Age of Empires II (AOE II) </li></ul><ul><li>Historically themed war game </li></ul><ul><li>Progress through “Ages”: dark, feudal, castle, imperial Civilizations have different capabilities </li></ul><ul><li>Develop technologies </li></ul><ul><li>Rock/paper/scissors strategy </li></ul><ul><li>Real Time Strategy game (RTS) </li></ul><ul><li>Chess = turn based strategy game </li></ul><ul><li>Complex divided attention task, build an economy & an armed force </li></ul><ul><li>Defeat opponents with good combination of both </li></ul>
    11. 11. Interactive Tutorial <ul><li>Age of Empires (AOE) was a successful game – over 1 million users </li></ul><ul><li>AOE II should capture new users </li></ul><ul><li>Make it easy to learn without reducing the complexity which makes the game fun </li></ul><ul><li>The decision was made to create a tutorial </li></ul>
    12. 12. Record of errors & failures over time
    13. 13. Change made after 1 participant <ul><li>In AOE II users need to gather resources (wood, food, gold and stone). </li></ul><ul><li>1 st participant was instructed to gather wood by tutorial, but there were no trees in view. </li></ul><ul><li>Issue and solution were clear </li></ul><ul><ul><li>Place a tree within view & teach users to explore off-screen </li></ul></ul><ul><ul><li>Issue never reappeared with final 15 participants </li></ul></ul>
    14. 14. Likelihood that problems were fixed <ul><li>Final 6 participants with no previously fixed issue reoccurring (one issue that turned up never received a fix). </li></ul><ul><li>90% confident that between 61 – 100% likely to have fixed issues addressed (binary confidence interval). </li></ul><ul><li>Many issues had many more participants and a higher fix probability. </li></ul>
    15. 15. Re-fix example <ul><li>Re-fix = solution that did not work </li></ul><ul><li>In AOE II users need to make militia at the “Barracks” building. </li></ul><ul><li>Train vs. Create, first 6 of 8 participants </li></ul><ul><ul><li>Text inconsistency confused participants –sometimes “train”, sometimes “create” </li></ul></ul><ul><ul><li>Conceptually confusing as well –participants thought villagers would become militia </li></ul></ul><ul><li>Fixed text inconsistency by changing all labels to “train” </li></ul><ul><li>Conceptual issue still remained after next 2 participants, changed all labels to “create”. </li></ul>
    16. 16. AOE II: Reviews, Awards, Sales <ul><li>AOE II’s tutorial received “Excellence” award from Society of Technical Communication in 1999. </li></ul><ul><li>AOE II generated excellent reviews, many focusing on the tutorial. </li></ul><ul><li>Aug 1999 - Oct 2000 AOE II never left the top 10 in games sales. </li></ul><ul><ul><li>AOE I sold 456,779 units between Oct 97 – Dec 98. </li></ul></ul><ul><ul><li>AOE II sold almost double in similar time frame ( 916,812 copies between Oct 99 – Dec 00). </li></ul></ul>
    17. 17. Getting it RITE <ul><li>Key components to running a successful RITE study: </li></ul><ul><ul><li>UE familiar with product and target user </li></ul></ul><ul><ul><li>Product architected for iteration </li></ul></ul><ul><ul><li>Scheduling allows time for debriefing & iteration </li></ul></ul><ul><ul><li>Resources available to make changes </li></ul></ul><ul><ul><li>Decision makers present in the lab </li></ul></ul><ul><ul><li>Documentation of changes and fixes </li></ul></ul>
    18. 18. Scheduling RITE
    19. 19. Documenting RITE <ul><li>Key: </li></ul><ul><ul><li>X = issue observed with this participant </li></ul></ul><ul><ul><li>Yellow highlight = 1 st time fix attempted </li></ul></ul><ul><ul><li>Blue highlight = 2 nd time fix attempted </li></ul></ul><ul><ul><li>Heavy vertical lines = denote each time product changed </li></ul></ul><ul><ul><li>Black shading = Participant did not get to task where issue could have been experience </li></ul></ul>
    20. 20. Conclusions <ul><li>RITE WORKS </li></ul><ul><ul><li>Problems get fixed </li></ul></ul><ul><ul><li>Fixes get verified </li></ul></ul><ul><ul><li>Progress can be tracked </li></ul></ul><ul><ul><li>Teams feel a sense of accomplishment </li></ul></ul><ul><li>RITE has variants; e.g. Oracle example </li></ul>
    21. 21. Cautions <ul><li>RITE has limits </li></ul><ul><ul><li>Requires team to be present </li></ul></ul><ul><ul><li>Will not </li></ul></ul><ul><ul><ul><li>Discover a new product concept </li></ul></ul></ul><ul><ul><ul><li>Help you understand how users work in the real world </li></ul></ul></ul><ul><ul><ul><li>Generate product requirements </li></ul></ul></ul><ul><ul><ul><li>Prioritize usability issues </li></ul></ul></ul>
    22. 22. Conclusions about Methods <ul><li>The basics </li></ul><ul><ul><li>Study methods in context </li></ul></ul><ul><ul><li>Be clear about the goals of the method </li></ul></ul><ul><ul><li>Report method effectiveness as case studies </li></ul></ul><ul><li>The future </li></ul><ul><ul><li>Refine the case study approach </li></ul></ul><ul><ul><li>Institutionalize methods that have business impact </li></ul></ul><ul><ul><li>Build a new discourse community around methodology and design </li></ul></ul>
    23. 23. How do we see the future? With HOPE