Jessica Cameron and Mark Crowley discuss how metrics and benchmarking can improve your understanding of the user experience on your website.

1. You can’t manage what you can’t
measure
January 2016
1

2. CX metrics
2
https://www.experiencedynamics.com/blog/2015/03/
30-ux-statistics-you-should-not-ignore-infographic

3. Contents
3
 Why measure?
• The value of collecting metrics around user behaviour
 Methodology
• Analytics
• A/B testing
• True Intent studies
• User testing I
• User testing II: Remote quantitative
• One-minute survey
• Top task identification
• Tree testing
• Site search metrics
 Benchmarking
 Dashboards

4. Why measure?
The value of collecting metrics around user behaviour
4

5. The value of measuring behaviour
Measuring user behaviour can tell you:
What users are doing on your site
What problems they are having
Why they are having those problems
Whether users can accomplish what they came to your site to do
Which paths are most likely to lead to success
How satisfied users are with your site
These data are about the user experience
5

6. Methodology
How metrics are collected
6

7. Analytics
7
Key metrics:
► Conversion rates
► Bounce rates
► Time on site
Best for:
 Understanding what users are
doing
 Measuring the impact of changes
over time

8. A/B testing
8
Key metrics:
► Conversion rates
Best for:
 Testing specific hypotheses
 Comparing two (or more) options
 Testing changes before
implementing them

9. A/B testing
9
Which version did more testers click on – A or B?

A B
x
www.whichtestwon.com

10. A/B testing
10
www.usefulusability.com

11. True intent studies
11
Key metrics:
► User goals
► Path analysis
► Subjective ratings of success
► Subjective ratings of experience
(e.g., SEQ, SUS, NPS)
Best for:
 Understanding why users are
visiting your site
 Assessing the effectiveness of
different journeys/paths
 Collecting more reliable
satisfaction data

12. True intent studies
12
Thank you and goodbye!
[Additional questions as needed]
How easy or difficult did you find this site to use? (1-7 scale)
Were you able to accomplish what you came here to do today? (yes/no)
***Survey pops under until user is finished***
What have you come to this site to do today? (multiple choice)
Will you help us improve?

13. User testing
13
Key metrics:
► Completion rates
► Subjective ratings of experience
(e.g., SEQ, SUS)
Best for:
 Understanding why usability
problems exist

14. User testing
14
Oh, that’s great, that’s
all the information I
need. And I love the
colours they use here.
On this 1-7 scale, how easy
or difficult did you think it
was to find the information
you were looking for?
Let me see… 3 is slightly
difficult? I’d give it a 3. My
spelling isn’t very good,
but the search should still
have worked better than
it did.
(SEQ)

15. Remote quantitative user testing
15
Key metrics:
► Success / failure
► Confidence in answer
► Time on task
Best for:
 Adding a reliable, quantitative
metric to user testing
 Ideal for benchmarking

16. Remote quantitative user testing
16
Success, within
ideal timeframe
100
Scoring a task, examples
Success, but 2
minutes over ideal
time
75
Success, within
ideal timeframe,
but not sure of
answer
85
Wrong answer,
low confidence
40
Wrong answer,
high confidence
10
Timeout,
exceeded 5
minutes
40
Gave up 40

17. One-minute survey
17
Key metrics:
► Ranked order of site attributes – good and
bad
Best for:
 Giving direction to website
improvements
 Prioritizing projects
“Please choose the THREE factors from the list below that best describe your actual
experience with the [ ] website”

18. One-minute survey
18
“Please choose the THREE factors from the list below that best describe your actual
experience with the [ ] website”
26 statements in total

19. Top task identification
19
Key metrics:
► What customers most want to do
on a website or intranet – top
tasks
► What they definitely don’t want
to do – tiny tasks
Best for:
 Establishing user needs
 Complex, high traffic websites
with a lot of content and user
types

20. Top task identification
20

21. Top task identification
21
www.xkcd.com

22. Tree testing
22
Key metrics:
► First-click success rate
► Task success and failure rates
Best for:
 Evaluating information
architecture and category labels
after card sorting exercise
1 2 3

23. Tree testing
23

24. Site search metrics
24
Key metrics:
► Search results precision
► Search usability
► Content ‘searchability’
Best for:
 Addressing search issues
 Auditing content
 Comparing against search best practice

25. Site search metrics
25
Search on ‘weekend events’ on 17 December 2015

26. Metrics and the lifecycle of a website
26
Top task identification User testing
Analytics
Site search measurement
A/B testing
Site search measurement
User testing
1-minute survey
Analytics
True intent
A/B testing

27. Benchmarking
How metrics can be used to track site performance over time
27

28. The value of measuring behaviour over time
Benchmarking can tell you whether
changes to your site make users more
successful and satisfied
Benchmarking can tell you how best to
allocate development resources
to improve the overall user experience
28

29. Benchmarking
29
www.dilbert.com

30. All metrics
30
Completion rates
SEQ
SUS
NPS
User goals
Path analysis
Subjective ratings of success
Conversion rates
Time on task
Bounce rates
Time on site
Top tasks
Tiny tasks
Success / failure / disaster rates
Search results precision
Search usability
Content searchability
Take action priorities
First click success rate
ERQ

31. Example: informational website
31
Completion rates
SEQ
SUS
NPS
User goals
Path analysis
Subjective ratings of success
Conversion rates
Time on task
Bounce rates
Time on site
Top tasks
Tiny tasks
Success / failure rates
Search results precision
Search usability
Content searchability
Take action priorities
First click success rate
ERQ

32. Example: Benchmarking for an informational website
3232
ORIGINAL SITE
Benchmarking metrics
Top Task identification
Success/failure/
disaster rates
Time on task
Take action priorities

33. Example: Benchmarking for an informational website
3333
Benchmarking metrics
Success/failure rates
Time on task
SEQ
Take action priorities
Search results precision
Search usability
Content searchability

34. Tools for an informational website
3434
Benchmarking metrics
Success/failure rates
Time on task
SEQ
Take action priorities
Search results precision
Search usability
Content searchability
Remote quant
user testing
One-minute
survey
Site search
metrics

35. E-Commerce website: increase conversion
35
Completion rates
SEQ
SUS
NPS
User goals
Path analysis
Subjective ratings of success
Conversion rates
Time on task
Bounce rates
Time on site
Top tasks
Tiny tasks
Success / failure / disaster rates
Search results precision
Search usability
Content searchability
Take action priorities
First click success rate
ERQ

36. E-Commerce benchmarking: increase conversion
3636
Benchmarking metrics
User goals
Subjective success
SEQ
NPS
Conversion rates
Take action priorities

37. Tools for an E-Commerce website: increase conversion
3737
Benchmarking metrics
User goals
Subjective success
SEQ
NPS
Conversion rates
Take action priorities
True Intent study
Analytics
One minute
survey

38. 38

39. 39

40. 40

41. 41

42. 42