Click Here! Tap Here! Using Chalkmark for Adaptive Design Testing and Excel for Analysis

715 views
501 views

Published on

Azlia Baker's presentation from the UXPA Boston 2014 Conference

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
715
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Click Here! Tap Here! Using Chalkmark for Adaptive Design Testing and Excel for Analysis

  1. 1. Click here! Tap here! Using Chalkmark for adaptive design testing and Excel for analysis UXPA Boston 2014 Azilah Baker
  2. 2. What you’ll learn • A way to test adaptive designs • How to set up a first-tap study using Chalkmark • A way to tell a more complete story with numbers • How to calculate statistical data in Excel 0% 20% 40% 60% 80% 100% Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 Accuracy
  3. 3. Agenda • Introductions • Chalkmark • Set up & high-level Analysis • Take the study • Study results • Analysis spreadsheet in detail • Lessons learned
  4. 4. About Me: Azilah Baker • User Experience Researcher, Fidelity Investments, Boston • Team of 30 • Spectrum of Quant & Qual studies • In 2014 we’ve run studies with almost 15,000 participants to date
  5. 5. You’re here because… • You’re interested in designing and/or testing for mobile and want to compare it to desktop • You use Chalkmark and want to know what else it can do • You’re here to kill time before Beer o’clock
  6. 6. Adaptive vs Responsive • Responsive design = fluidly changes according to the screen or device size • Adaptive design = changes to fit a set number of screen or device sizes Responsive design on Boston Globe
  7. 7. First-click testing • Quantitative testing method • Goal is to learn: Where do users begin their task? • Tools: • Chalkmark • UserZoom • UsabilityHub • Qualtrics
  8. 8. When to use First-click testing • Early in the design cycle, after IA • You have wireframes or flat images, but not interactive prototypes • Typically, you have multiple designs to compare
  9. 9. Chalkmark yesterday
  10. 10. Chalkmark yesterday
  11. 11. Chalkmark today
  12. 12. Chalkmark today
  13. 13. Setting up a study To test adaptive designs, you need: 1. Flat images • Large screen: 1024px width • Small screen: 320px width • Tip: Use the same image format for all (e.g. avoid mixing pngs and jpgs) 2. For each test condition, create its own study in Chalkmark 3. Create a URL with javascript to randomize study conditions
  14. 14. Setting up a study Tips for the smartphone (320px wide) • Check display on different phones • Add 100px of blank space to the bottom of the image canvas
  15. 15. Image size: 320 x 640 On some phones, the image is cut off at the bottom Try it: bit.ly/1nFH3Ap
  16. 16. Analysis: Chalkmark gives you… • Accuracy data • Time data (regardless of task success) Try it: bit.ly/1nFH3Ap
  17. 17. Define area of correct answer: Heatmaps > Selection What if you need statistical data that looks like this… 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 Accuracy
  18. 18. Why use the Analysis spreadsheet? 1. Calculate accuracy, time & efficiency (combination of accuracy & time) per condition. Efficiency is a better measure of a page’s usability. 2. Get statistics across all conditions 3. Save coordinates of the correct answers because AOIs (areas of interest) in Chalkmark are not “sticky” and cannot be saved – has to be redefined each time
  19. 19. Preview: Analysis Spreadsheet Use vlookup Sample Condition
  20. 20. Preview: Analysis Spreadsheet Do both X and Y coordinates fall within the correct answer range? 1 if yes; 0 if no Efficiency = accuracy and time. If P was accurate, how many correct answers per minute would they get?
  21. 21. Agenda • Introductions • Chalkmark • Set up & high-level Analysis • Take the study • Study results • Analysis spreadsheet in detail • Lessons learned
  22. 22. Pilot Study • Does the placement of action buttons affect task accuracy and efficiency? • Are there any differences on a large vs small screen? • Smartphone: • http://WebUsabilityStudy.com/Adaptive1/ • Laptop or tablet: • http://WebUsabilityStudy.com/Adaptive2/
  23. 23. Design A Large screen Design A Small screen Design B Small screen Design B Large screen
  24. 24. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% LargeA LargeB SmallA SmallB Task 1 Accuracy Results: Task 1 • Design A on large screen more accurate than small screen • Higher efficiency on large screens 0 1 2 3 4 5 6 7 8 9 10 LargeA LargeB Task 1 Efficiency (# correct per min) SmallA SmallB 0 5000 10000 15000 20000 25000 LargeA LargeB SmallA SmallB Task 1 Time (ms) *
  25. 25. 0 5000 10000 15000 20000 25000 LargeA LargeB SmallA SmallB Task 2 Time (ms) Results: Task 2 • Accuracy much lower than Task 1 • Small screen A worst in accuracy and efficiency 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% LargeA LargeB SmallA SmallB Task 2 Accuracy 0 1 2 3 4 5 6 7 8 9 10 LargeA LargeB SmallA SmallB Task 2 Efficiency (# correct per min)
  26. 26. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% LargeA LargeB SmallA SmallB Overall Accuracy 0 5000 10000 15000 20000 25000 LargeA LargeB SmallA SmallB Overall Time (ms) 0 1 2 3 4 5 6 7 8 9 10 LargeA LargeB SmallA SmallB Overall Efficiency (# correct per min) Results: All Conditions • Overall, participants were less efficient on the smartphone
  27. 27. Lessons Learned: Chalkmark on mobile Mobile experience similar to desktop Easy to compare data across form factors • Slightly higher potential for false taps • Task bar eats up space • Minimize pre-study questions • Check display on different devices
  28. 28. Wrap up • This is ONE way to test adaptive (and responsive) design • Calculating statistical data tells a more complete story. Analysis is manual – but you can do it!
  29. 29. THANK YOU! Azilah Baker azilah.baker@fmr.com @azibaker www.linkedin.com/in/azilahbaker Session materials will be posted on LinkedIn

×