This document discusses website usability testing and user experience measurement. It outlines goals of creating an engaging user experience and compelling content to increase conversions. It then describes how to measure user experience through engagement and usability metrics, as well as qualitative feedback. The methodology section details creating test scenarios, identifying test subjects, running tests, analyzing data and improving based on findings. Specific tools are mentioned like visual analytics, engagement funnels, A/B testing and services like ClickTale, UserTesting.com and SilverBackApp for qualitative and quantitative user research.

1. Website Usability and User Experience

2. How do we create a great user
experience?

3. Brand Directional Goals
Amazing creative
Engaging brand stories
Compelling content
Increased conversions
Global awareness – Japan
market leads the way

4. How do we measure a great user
experience?

5. Engagement KPIs & Usability KPIs
In page engagement/usability = Click and Hover Heatmaps
Engagement funnels = Conversions, Bounce rates &
Abandonment rates
Usability funnels = Time to Convert & Abandonment profile
Qualitative metrics = What did you like or dislike? What made you
frustrated or was more complicated than necessary?

6. Collecting the Data

7. User Testing Methodology
Create
scenarios
or
A/B
tests
using
KPIs
and
Stakeholder
interests
(JP,
DE,
US)
Iden@fy
test
subjects
via
profiles
Run
test
scenarios
either
remote
or
hosted
on
site
Collect
data
with
A/V
tools
and
Visual
analy@cs
Analyze
data
and
iden@fy
paMerns
Design
and
develop
solu@ons
and/or
refine
test
scenarios

8. User Testing Methodology
Live session sampling – real customers, real data with usability
and engagement focused visual analytics
User scenarios and KPI-hitting user flows from all interested
stakeholders.
Hosted and moderated usability testing both internal and external
for US, DE and JP

9. Tools and Techniques
Visual analytics
Engagement funnels
Qualitative user testing
A/B & Multivariate testing

10. Visual Analytics
Mouse move heat maps
individual and
aggregated, show where
customers focus their
attention via the mouse
pointer (80% correlation
with eye gazing).
Attention maps show
how much time was
spent on a horizontal
band of content.
Mouse click heat maps
individual and
aggregated, show what
gets the most clicks or
doesn’t get any at all.
Scroll reach maps
measure how far down
the page customers go.

11. Engagement Funnels & Analytics

12. Qualitative User Testing
Moderated testing with an experienced host
• Profiled test customers using cross organization demographics data
• Place the test subject at ease and set the scene with a quick conversation about their
shopping habits and topical interests. eg if they are thinking about buying a handbag as a
gift for a friend or themselves – we can tailor the test to match their existing interest.
• Observations are recorded – audio/video and usability analytics in addition to the testing
host taking notes and marking cue points in the test session for followup review.
• Desktops, laptops, smartphones and tablets will be used for testing
• The test scenarios and KPIs being targeted are provided by direction from organizational,
business and brand stakeholders.

13. Qualitative User Testing

14. A/B & Multivariate Testing
Current A/B testing (Google Optimizer)
• limited to production sites
• requires final creative design and functionality
• limited to a few areas of high ROI due to effort involved
Visual Analytics and Rich Media Testing will support early A/B
testing for UI and Creative concepting with smaller test groups.
Each tool can be used on prototypes, development servers and
QA servers for previously unavailable testing processes.

15. User Testing Methodology
Frequency of testing/observation
• Real time session monitoring, analytics with sample set of users
• Annual in-depth user testing, analytics: 7-10 scenarios
• Project based user testing, analytics for new features, regional launches.
Duration for User Testing
• 1 hour sessions with 5 – 10 test participants

16. User Testing Roadmap
Visual Analytics – Now. We have begun the rollout this week
• First will be anonymous sessions, full transaction
• Next will be logged in user sessions, full transaction
Usability testing – Early November
• First will be remote un-moderated but profiled, to validate user scenarios
• Next will be on-site moderated testing as well as remote testing on a larger scale
International Testing – TBD
• We are working on vendor support for international testing using the same
scenarios as for the US market

17. Tool demos & examples

18. CLICKTALE
Visual analytics & Engagement funnels

19. CLICKTALE
Visual analytics & Engagement funnels

20. CLICKTALE
Visual analytics & Engagement funnels

21. UserTesting.com
External and Internal Qualitative Testing
Desktop and Mobile Devices

22. UserTesting.com
External and Internal Qualitative Testing
Desktop and Mobile Devices

23. SilverBackApp
Internal Qualitative User Testing on Multiple Devices

24. SilverBackApp
Windows and Android Too!

25. Thanks
for Stopping By!