Azlia Baker's presentation from the UXPA Boston 2014 Conference

1. Click here! Tap here!
Using Chalkmark for adaptive design
testing and Excel for analysis
UXPA Boston 2014
Azilah Baker

3. What you’ll learn
• A way to test adaptive designs
• How to set up a first-tap study using
Chalkmark
• A way to tell a more complete story with
numbers
• How to calculate statistical data in Excel
0%
20%
40%
60%
80%
100%
Task
1
Task
2
Task
3
Task
4
Task
5
Task
6
Task
7
Task
8
Task
9
Task
10
Accuracy

4. Agenda
• Introductions
• Chalkmark
• Set up & high-level Analysis
• Take the study
• Study results
• Analysis spreadsheet in detail
• Lessons learned

5. About Me: Azilah Baker
• User Experience Researcher, Fidelity
Investments, Boston
• Team of 30
• Spectrum of Quant & Qual studies
• In 2014 we’ve run studies with almost
15,000 participants to date

6. You’re here because…
• You’re interested in designing and/or testing
for mobile and want to compare it to
desktop
• You use Chalkmark and want to know what
else it can do
• You’re here to kill time before Beer o’clock

7. Adaptive vs Responsive
• Responsive design = fluidly
changes according to the
screen or device size
• Adaptive design = changes to
fit a set number of screen or
device sizes
Responsive design on Boston Globe

8. First-click testing
• Quantitative testing method
• Goal is to learn: Where do users begin
their task?
• Tools:
• Chalkmark
• UserZoom
• UsabilityHub
• Qualtrics

9. When to use First-click testing
• Early in the design cycle, after IA
• You have wireframes or flat images, but
not interactive prototypes
• Typically, you have multiple designs to
compare

10. Chalkmark yesterday

11. Chalkmark yesterday

12. Chalkmark today

13. Chalkmark today

14. Setting up a study
To test adaptive designs, you need:
1. Flat images
• Large screen: 1024px width
• Small screen: 320px width
• Tip: Use the same image format for all
(e.g. avoid mixing pngs and jpgs)
2. For each test condition, create its own
study in Chalkmark
3. Create a URL with javascript to randomize
study conditions

15. Setting up a study
Tips for the smartphone (320px wide)
• Check display on different phones
• Add 100px of blank space to the bottom of
the image canvas

16. Image size:
320 x 640
On some
phones, the
image is cut
off at the
bottom
Try it:
bit.ly/1nFH3Ap

17. Analysis: Chalkmark gives you…
• Accuracy data
• Time data (regardless of task success)
Try it:
bit.ly/1nFH3Ap

18. Define area of
correct answer:
Heatmaps >
Selection
What if you need
statistical data that
looks like this…
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Task
1
Task
2
Task
3
Task
4
Task
5
Task
6
Task
7
Task
8
Task
9
Task
10
Accuracy

19. Why use the Analysis spreadsheet?
1. Calculate accuracy, time & efficiency
(combination of accuracy & time) per
condition. Efficiency is a better measure of
a page’s usability.
2. Get statistics across all conditions
3. Save coordinates of the correct answers
because AOIs (areas of interest) in
Chalkmark are not “sticky” and cannot be
saved – has to be redefined each time

20. Preview: Analysis Spreadsheet
Use vlookup
Sample
Condition

21. Preview: Analysis Spreadsheet
Do both X and Y coordinates fall within the correct
answer range? 1 if yes; 0 if no
Efficiency = accuracy and time. If P was accurate, how
many correct answers per minute would they get?

22. Agenda
• Introductions
• Chalkmark
• Set up & high-level Analysis
• Take the study
• Study results
• Analysis spreadsheet in detail
• Lessons learned

23. Pilot Study
• Does the placement of action buttons affect
task accuracy and efficiency?
• Are there any differences on a large vs small
screen?
• Smartphone:
• http://WebUsabilityStudy.com/Adaptive1/
• Laptop or tablet:
• http://WebUsabilityStudy.com/Adaptive2/

24. Design A
Large screen
Design A
Small screen
Design B
Small screen
Design B
Large screen

25. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LargeA LargeB SmallA SmallB
Task 1 Accuracy
Results: Task 1
• Design A on large
screen more accurate
than small screen
• Higher efficiency on
large screens
0
1
2
3
4
5
6
7
8
9
10
LargeA LargeB
Task 1 Efficiency (# correct per min)
SmallA SmallB
0
5000
10000
15000
20000
25000
LargeA LargeB SmallA SmallB
Task 1 Time (ms)
*

27. 0
5000
10000
15000
20000
25000
LargeA LargeB SmallA SmallB
Task 2 Time (ms)
Results: Task 2
• Accuracy much lower
than Task 1
• Small screen A worst in
accuracy and efficiency
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LargeA LargeB SmallA SmallB
Task 2 Accuracy
0
1
2
3
4
5
6
7
8
9
10
LargeA LargeB SmallA SmallB
Task 2 Efficiency (# correct per min)

29. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
LargeA LargeB SmallA SmallB
Overall Accuracy
0
5000
10000
15000
20000
25000
LargeA LargeB SmallA SmallB
Overall Time (ms)
0
1
2
3
4
5
6
7
8
9
10
LargeA LargeB SmallA SmallB
Overall Efficiency (# correct per min)
Results: All Conditions
• Overall, participants
were less efficient on
the smartphone

30. Lessons Learned:
Chalkmark on mobile
Mobile experience similar to desktop
Easy to compare data across form factors
• Slightly higher potential for false taps
• Task bar eats up space
• Minimize pre-study questions
• Check display on different devices

31. Wrap up
• This is ONE way to test adaptive (and
responsive) design
• Calculating statistical data tells a more
complete story. Analysis is manual – but you
can do it!

32. THANK YOU!
Azilah Baker
azilah.baker@fmr.com
@azibaker
www.linkedin.com/in/azilahbaker
Session materials will be posted on LinkedIn