Rapid Iterative Testing and Evaluation

Rapid Iterative Testing and Evaluation RITE Dennis Wixon Michael Medlock Melissa Federoff

Outline of Talk
What’s wrong
What’s RITE
Creation of the RITE method
Running a RITE study
Conclusions
Q/A

Editorial: Research on Methods

What’s wrong
Purpose: the goal of a method is to find problems
Cost: determine the minimal number of participants
Evaluation: we should use traditional research methods to determine the “best” method
Assumption: there is a best method

What’s right?
Purpose of an iterative test
Rapid improvements
Create a collaborative environment
Measure cost and benefit:
Percentage of problems addressed
Percentage of effective fixes
Evaluation: use case studies
Match method to purpose
Position user research as a business asset

What is a Rapid Iterative Test and Evaluation test (RITE)?
Just like a “standard” usability test except:
Changes made as soon as a problem is identified and a potential solution is clear
The changed interface is tested with subsequent users
This is the Essence of RITE

“Standard” vs. RITE

What is the business case for RITE?
Return on investment
Faster detection of problems
More issues found
More issues fixed
Fixes are verified
Better team dynamics
The method degrades gracefully

Case Study: Age of Empires II
What is Age of Empires II (AOE II)
Historically themed war game
Progress through “Ages”: dark, feudal, castle, imperial Civilizations have different capabilities
Develop technologies
Rock/paper/scissors strategy
Real Time Strategy game (RTS)
Chess = turn based strategy game
Complex divided attention task, build an economy & an armed force
Defeat opponents with good combination of both

Interactive Tutorial
Age of Empires (AOE) was a successful game – over 1 million users
AOE II should capture new users
Make it easy to learn without reducing the complexity which makes the game fun
The decision was made to create a tutorial

Record of errors & failures over time

Change made after 1 participant
In AOE II users need to gather resources (wood, food, gold and stone).
1 st participant was instructed to gather wood by tutorial, but there were no trees in view.
Issue and solution were clear
Place a tree within view & teach users to explore off-screen
Issue never reappeared with final 15 participants

Likelihood that problems were fixed
Final 6 participants with no previously fixed issue reoccurring (one issue that turned up never received a fix).
90% confident that between 61 – 100% likely to have fixed issues addressed (binary confidence interval).
Many issues had many more participants and a higher fix probability.

Re-fix example
Re-fix = solution that did not work
In AOE II users need to make militia at the “Barracks” building.
Train vs. Create, first 6 of 8 participants
Text inconsistency confused participants –sometimes “train”, sometimes “create”
Conceptually confusing as well –participants thought villagers would become militia
Fixed text inconsistency by changing all labels to “train”
Conceptual issue still remained after next 2 participants, changed all labels to “create”.

AOE II: Reviews, Awards, Sales
AOE II’s tutorial received “Excellence” award from Society of Technical Communication in 1999.
AOE II generated excellent reviews, many focusing on the tutorial.
Aug 1999 - Oct 2000 AOE II never left the top 10 in games sales.
AOE I sold 456,779 units between Oct 97 – Dec 98.
AOE II sold almost double in similar time frame ( 916,812 copies between Oct 99 – Dec 00).

Getting it RITE
Key components to running a successful RITE study:
UE familiar with product and target user
Product architected for iteration
Scheduling allows time for debriefing & iteration
Resources available to make changes
Decision makers present in the lab
Documentation of changes and fixes

Scheduling RITE

Documenting RITE
Key:
X = issue observed with this participant
Yellow highlight = 1 st time fix attempted
Blue highlight = 2 nd time fix attempted
Heavy vertical lines = denote each time product changed
Black shading = Participant did not get to task where issue could have been experience

Conclusions
RITE WORKS
Problems get fixed
Fixes get verified
Progress can be tracked
Teams feel a sense of accomplishment
RITE has variants; e.g. Oracle example

Cautions
RITE has limits
Requires team to be present
Will not
Discover a new product concept
Help you understand how users work in the real world
Generate product requirements
Prioritize usability issues

Conclusions about Methods
The basics
Study methods in context
Be clear about the goals of the method
Report method effectiveness as case studies
The future
Refine the case study approach
Institutionalize methods that have business impact
Build a new discourse community around methodology and design

How do we see the future? With HOPE

http://uxdesign.pl	181
http://www.slideshare.net	10
http://ergonomilk.com	7
http://ergonomie.posterous.com	5

Rapid Iterative Testing and Evaluation

by Maciej Lipiec

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 203

Statistics

Rapid Iterative Testing and Evaluation Presentation Transcript