Presented by: Jean Fox

1. 1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Evaluating Complex Systems:
Strategies for Testing Systems
You Can’t Understand
Jean E. Fox and Robin Kaplan
U.S. Bureau of Labor Statistics
UXPA 2017
June 7, 2017

2. 2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Background
 Challenge for usability testing at BLS
Users: Highly trained, highly experienced,
Ph.D. Economists, Mathematicians, and Statisticians
Tasks: Very complex tasks using brand new systems
Time frame: Not very long
 My experience is with
Internal System
Mostly brand new system, with no digital predecessor

3. 3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Background
 Reality
It was impossible for me to learn the systems
 Choice
Figure out how to do the tests, or let the complex
systems go without testing.

4. 4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Sample System
 Define Index Structure (DIS)
System to build the formula to calculate the
Consumer Price Index (measure of inflation)
Allows users to build a hierarchy of
– Geography
– Items

5. 5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

6. 6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

7. 7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

8. 8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Topics
 Learning the system
 Picking tasks
 Working out logistics
 Training participants
 Staffing the tests
 Moderating
 Recording results
 Analyzing data

9. 9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Learning the System
 How much does a UX professional really need
to know?
Ideal – know the whole system
Basic – know the part of the system you’re
working with
Minimal – basic understanding of tasks and goals

10. 10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Learning the System
 How:
Attend demos, meetings
Attend training
Work with knowledgeable staff (critical)
Observe users
Read manuals, reports, etc.
 Review the recordings after each participant
Will help you learn the system and refine the test
methodology

11. 11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Thoughts on
Learning the System

12. 12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Picking Tasks
 How do you choose tasks?
Use what you know about the system
Work with team members
– Have them suggest tasks with your guidance on what
makes a good task
Find focused tasks you can understand more easily
What seems like a simple task may be enough
 Consider task wording carefully

13. 13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Picking Tasks
 Be careful with vague, open-ended tasks
Easier to set up
But may lead to great variety in behaviors
Harder to evaluate performance
Consider this approach with a few participants to
identify more specific tasks to focus on.
 Participants may need their own tasks,
depending on how specialized their job is
Try to make them similar

14. 14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Thoughts on
Picking Tasks

15. 15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Working Out the Logistics
 May not always be an issue
 There could be complex connections to
multiple secure databases
 Determine how much of a task to set up
ahead of time
 Have a plan to reset the system after each
participant (and leave enough time for this)
 Record video if you can

16. 16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Thoughts on
Logistics

17. 17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Training the Participants
 How much training do you provide?
Ideally, none
But in reality, they won’t use the system without
training
Provide as little training as possible (it may
require some fine-tuning as you go)
– Group training session prior to testing
– Individual training during the test session
– Cheat sheet or other documents

18. 18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Thoughts on
Training

19. 19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Staffing the Test
 Have at least one knowledgeable team
member present for every test
Good practice anyway, but essential here
May also need IT staff present to fix problems
 Train the observers
When and how to provide help to participants
What language to use (or not)

20. 20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Moderating the Test
 Consider the responsibilities of observers
Training
Facilitating the test
Providing assistance when needed
 Encourage think aloud

21. 21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Moderating the Test
 Determine when to provide assistance
May be difficult for complex tasks
Look for cues other than task performance, such
as comments, body language, etc.
Rely on your experience moderating tests

22. 22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Thoughts on
Staffing the Tests
and Moderating

23. 23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Recording the Results
 Determining task success
Set up parameters ahead of time if possible
Work with observers (carefully, since they may be
biased to consider a task successful)
 Measuring task time
Did they only focus on task or get significantly
side-tracked by exploring a new feature?
When did they really finish?
Review video

24. 24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Thoughts on
Recording the Results

25. 25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Analyzing the Data
 Identifying problems
Some issues may be problems with training or
difficulties with the task itself.
Consider the kind of help you have to provide
Work with the observers to identify real problems
 Identifying solutions
Review the video
Rely on your experience
Work with the team to develop realistic solutions

26. 26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Thoughts on
Analyzing the Data

27. 27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Takeaways
 You CAN run usability tests with really
complex systems.
 It’s challenging, but…
 There are things you can do to make it easier.

28. Contact Information
28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov
Jean E. Fox
Bureau of Labor Statistics
Fox.Jean@bls.gov
Robin Kaplan
Bureau of Labor Statistics
Kaplan.Robin@bls.gov