This document discusses a data-driven UX process that uses both qualitative and quantitative metrics. It presents the HEART framework for defining UX goals and metrics around Happiness, Engagement, Adoption, Retention, and Task Success. An example is provided of applying this process to an advanced search feature, where initial user interviews and testing revealed discrepancies between stated importance and actual usage. Prototypes were developed and tested, with metrics like SUS scores and error rates measured before and after to evaluate improvements. The process emphasizes continuously defining and refining goals, collecting both baseline and comparative metrics over time, and using data to inform iterative design changes.

1. Design by Numbers
A Data-Driven UX Process
Brian Rimel @brianrimel
UX Consultant, OpenSource Connections

2. User-Centered Design
Internal enterprise applications
Access to users

3. Why Data?
Balancing the qualitative and quantitative
You can’t always trust your users
Limited data doesn’t tell the whole story

4. The HEART Framework
Src: https://library.gv.com/how-to-choose-the-right-ux-metrics-for-your-product-5f46359ab5be

5. PULSE Metrics
Page views, Uptime, Latency,
Seven-day active users, Earnings

6. Unnecessary Data
Creates Noise

7. HEART Metrics
Happiness, Engagement, Adoption,
Retention, Task Success

8. Happiness
Satisfaction or Delight
System Usability Scale, Net Promoter Score

9. Engagement
Level of involvement
Number of visits per user per week

10. Adoption
New users/uses of a feature
Number of accounts created in the last 7 days

11. Retention
Rate at which existing users return
Percentage of seven-day active users that
are still active 30 days later

12. Task Success
Traditional behavior metrics for efficiency,
effectiveness, and error rate.
Percentage of completion errors for a given task

13. Goals
Signals
Metrics

14. Goals Signals Metrics
Happiness
The user feels the
welcome wizard is
easy to use
Level of user
satisfaction
Mean SUS Score
Engagement - - -
Adoption - - -
Retention - - -
Task Success
The welcome wizard
should be as
simple as possible
The number of
errors during the
process
Rate of error
per step
Example: Welcome Wizard

15. Goals should be SMART
Specific, Measurable, Attainable, Realistic, Time-Based

16. Normalize the Data
What does an increase in total active users tell us?

17. A Limited-Data Process

18. Initial Metrics Gathering
Existing metrics influence feature priority
Kano Survey for feature-level satisfaction

19. Kano Survey
src: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results
Feature Must-be
One-
Dimensional
Attractive
Unimportan
t
Undesired
Advanced Search 87% 8% 4% 1% 0%

20. Prioritizing of Features
1.
2.
3. Advanced Search
4.
5.
6.
7.
8.
9.
10.
From Kano Survey:
87% Must-be feature
From Usage Statistics:
22% Engagement/Week
Why the
discrepancy?

21. Goals Signals Metrics
Happiness
The user feels
comfortable using
advanced search
Level of
confidence
SUS Survey
Engagement
The features enable
consistent searching
Number of
advanced
searches
Searches per day
per user
Adoption - - -
Retention - - -
Task Success
The advanced
search process is
easily understood
User enters a
query, but does not
complete the
search
Percentage of
Abandoned
Searches
Advanced Search: Goals & Metrics

22. User Interview & Testing
Identify discrepancy between
stated importance and usage metrics
Establish baseline metrics
Measure satisfaction - SUS Survey

23. System Usability Scale (SUS)
src: https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

24. Review Findings
Metric Initial Testing
Mean SUS Score 56
Error Rate / Step 21%
Okay, so where is the problem?
Let’s map it!

25. Mapping the Journey

26. Develop Prototypes

27. User Testing of Prototype
Continue measuring baseline metrics
A/B Testing
Follow-up SUS Survey

28. Results & Recommendations
Great! But, what does this mean?
Context critical to interpretation
Metric Initial Testing Prototype Testing
Mean SUS Score 56 73
Error Rate / Step 21% 12%

29. The Customer Journey

30. Long-Term Metrics
Tracking Engagement, Adoption, Retention, and
Task Success over time
Periodic usability testing of full application

31. The Data-Driven Process

32. Tools

33. Kibana Dashboard

34. “Extremely satisfied is
like extremely edible.”
- Jared Spool

35. Key Takeaways
• Collaboratively define SMART goals
• Revisit and challenge goals
• Continuously monitor metrics over time
• Balance quantitative and qualitative measures

36. Questions

37. References
• Google HEART Metrics Study
http://static.googleusercontent.com/media/research.google.com/e
n//pubs/archive/36299.pdf
• Kano Survey
http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-
results
• SUS Survey
https://www.usability.gov/how-to-and-tools/methods/system-
usability-scale.html