This document discusses using key performance indicators (KPIs) and analytics to measure the success of product experiments and products. It provides examples of KPIs currently monitored, like logins and calls, and discusses whether these are the right metrics. The document advocates adopting a modern analytics toolset to track the right outcome-based metrics throughout the product development process, from planning to post-release. Resources and agreement on goals and success metrics for the next few months are needed to implement a more effective analytics approach.

1. kpi’s and you
product management with analytics

2. Who am I?
Daryl Hemeon - @darylhemeon
Tech Architect for iServices Claims Self Service
Projects
Web Dev for 15+ years
I still code and when I do it scares the crap out
of the CSS team

3. What are KPI’s?
Key Performance Indicators are really
Outcomes that we are interested in learning
about to see if our experiments (and our
products) are successful.

4. What do we monitor today?
1. Employee Reg
2. Logins
3. Contact Center Calls
4. Opt Out / In
5. Online claims filed
6. EFT Setup
7. Docs Uploaded
8. eSigned Auths
9. Claims Filed total
10. Open Claims

5. How do we monitor them?
1. Custom spreadsheet running macros to the
iServices database
2. My personal dashboard that reads 2 different
datasources
3. Product Owner(s) use magic and the phone
to get other stats

6. Are these the right stats?
Vanity Metrics
● Logins per month (did the user accomplish their goal?)
● Registrations per month (did the user then login?)
Learning Metrics - decision or actionable
metrics
● Messages Sent
● Documents Uploaded - after a request for information

7. How should this work?
1. Modern Toolset
2. Inject into the planning process
3. Inject into dev process
4. Monitor post-release
5. Product Owner now has the data to pivot priorities after
every sprint or iteration if they need to
6. Outstanding question - Who does this work?

8. Modern Toolset
We need to be using modern real-time analytics

9. the planning process
Before we created the map we should have voted the KPI’s
or Outcomes and after the map creation adjusted them.

10. How does Agile help?

11. How does it inform UX design?
Heatmaps and click charts can inform HOW your
application is actually being used and whether or
not it is meeting objectives.

12. What is needed for us to get there?
● Agile training for everyone - in progress - some people
already trained
● Cost Effective Google Analytics training for analysts and
developers (free on-line, Skill Soft)
● Continuous Integration for greater quality and delivery
with greater frequency - putting together a roadmap
● Resources are need to monitor post release
effectiveness and then work with product owners to
influence the backlog

13. What are we interested in for July?
● An agreement upon what stats will tell us if
we are successful
● An agreement upon the goals for the first 3
months of usage
● An understanding of the long term business
goals

14. References
● Lean UX by Jeff Gothelf
● Product Managers Guide to Analytics
● The Lean Startup by Eric Reis - discussion on Vanity Metrics
● The Marketer’s Guide to GA by the folks at Buffer
● Built to Last by Collins and Porras
● Training resources for GA

15. Questions?
We like them.