Backstage 2019 - Accelerating Product Insights at Intuit - John Humphrey

1. Accelerating Product
Insights at Intuit
John Humphrey
Head of Quickbooks Online
Platform Data Science and Analytics

2. Data are useful to the extent
that they lead to measurably
better decisions
Better decisions, faster

3. 3
• Answers are easy. . .questions are hard
• Don’t take the problem at face value
• Make sure the problem is worth solving and can be
solved
• Focus on delivering the MVP answer
The hardest part in solving any complex problem is
making sure you’ve identified the right problem to solve

4. 4
• Data scientists/analysts need business context
• We’ve embedded our teams to make context sharing easier
• Embedding also enables an agile approach to delivery
Our embedded org structure by design makes it easier
to understand customer needs

5. User segments and their
needs

6. 6
Needs of the Product Team
PM
Decisions
• Identify areas for improvement
• Assess impact of changes
Insights
• Visibility into user behavior
• Understand product usage
Metrics
• 90d Retention
• NPS

7. 7
Needs of the Marketing Team
Marketing
Decisions • Assess impact of campaigns
Insights
• Identify user segments
• Understand user segments’ usage
Metrics
• Subscriber growth
• LTV:CAC

8. 8
Needs of the Engineering Team
Engineering
Decisions • Identify performance issues
Insights • Understand friction points for users
Metrics
• Uptime
• Bugs
• TP99

9. 9
Needs of Executives
Executives
Decisions • Ensure teams hit their goals
Insights • Understand what drives growth
Metrics
• Revenue
• Subscribers

10. 10
Impact to the Business:
We moved slowly and often didn’t
have even an MVP answer, leading
to bad decisions
Even when we identified the right question, delivering
an MVP answer was difficult and expensive
PM
Needed us to build custom reports for every
feature or experiment
Marketing
Needed us to look up numbers for marketers
because tools were hard to use
Engineering
Needed us to pull user level event data for testing
and debugging
Executives
Needed insight into investment decisions (which
we seldom had time to provide so they flew blind)
Impact to the Analytics Team:
Time lost to low impact and
mundane work that didn’t drive
informed business decisions

11. Building the right solution
for multiple user segments

12. 12
Introducing Quickbooks Self-Employed
Categorize
Expenses
Track
Mileage
Capture
Receipts
Manage
Invoices
Complete
Financial Picture
Taxes
Handled

13. 13
The startup-like nature of Self-Employed made it the
right place to try a new approach to product insights
Resources to get things done
A business with a ton of questions… and just three analysts
A blank slate

14. 14
Getting Started (or “that time an intern’s summer project
changed the trajectory of product insights at Intuit”)
Use Segment as the
endpoint to collect events
and relay everywhere
Use Amplitude to quickly get
product insights into the
hands of decision-makers
Create a standard event
naming convention
(taxonomy) to clearly
describe events

15. 15
PMs Seamless view of product usage via funnel and path analyses
Marketers Segmented view of customer behavior via Custom Cohorts
Engineers Detailed realtime visibility via User and Account Lookup
Analysts Stakeholder self-service by providing custom attributes via Amplitude API
Executives Easily accessible insights via user-friendly dashboards
Knowing our users enabled us to solve for their needs

16. Validating Success

17. 17
It wasn’t long before we starting seeing results
📈Which led to better
results…
"
💡
Analysts get time to solve
our most difficult
problems!
PMs, Marketers, and
Engineers could easily
access product insights…

18. 18
Case Study: Our Original Permissions Experience (7 steps)

19. 19
We used Amplitude to figure out how to remove three steps
and increased the number of users allowing pushes by 25%

20. 20
PMs
Understand how customers are using the product to ship better
experiences, faster
Marketers
Understand how user segments engaged with product to improve
communication
Engineers
Deep dive into product data to debug issues and resolve customer
pain faster
Analysts Spend more time on more difficult questions
Executives See how product investments are paying off
Our first attempt largely hit the mark across the board

21. Learnings let you double
down on what is valuable

22. 22
Executives
Saw the value Self-Employed created and wanted this value for
other teams
PMs
Provided domain expertise on their product and spearheaded
the implementation
Analysts
Provided guidance to PMs to ensure the implementation gave
the intended result…including how to fix the taxonomy
Engineers Highlighted implementation issues and provided solutions
Providing product insights at scale is a team sport

23. 23
Scaling up investment = scaling up returns
Helped our Payments
business self-serve 90% of
its requests…
Which helped PMs discover
repetitive tasks in invoice
creation for recurring
transactions. . .
📈And gave analysts time to
build models that increased
automation by 15%

24. 24
PMs
Saw that many users were manually creating recurring invoices so
decided to automate the process
Customers
Shared that feature was hard to find and they wanted Quickbooks to
proactively make suggestions
Analyts
Built models to identify when a user is likely creating an invoice that
should be a recurring invoice
Engineers
Used the model to know when to point users to our recurring
invoice feature, thereby saving our users time
Case Study: Helping Users Automate Invoice Creation

25. What we learned

26. Be clear about the
problem you are
solving
Being close to the action
provides useful context

27. Democratize the
journey to product
insights
Collaboration increases buy-
in, and vice versa

28. Tools are easy,
content is hard
Technologies like Amplitude
make scaling easy, but
GIGO still applies

29. But don’t overthink
things: done > perfect
Your product is constantly
changing so mistakes
quickly cease to matter

30. Thank you!
John Humphrey
Head of Quickbooks Online
Platform Data Science and Analytics