The document outlines a presentation on implementing analytic and optimization tools, specifically focusing on A/B testing in product management. It highlights the importance of experimentation, providing a framework for planning and executing A/B tests, and includes best practices and considerations for successful outcomes. The session emphasizes that A/B testing is accessible to everyone and urges careful choice of tools and hypothesis formulation.

1. Brian Gardner - Director, Product Management
Ilya Izrailevsky - Engineering Manager
Implementing analytic
and optimization tools
on your website or app
WiFi: QBConnect Password: not required#QBConnect

2. 2
Today’s Speakers
Brian Gardner
Director, Product Management
@brianwgardner
Ilya Izrailevsky
Engineering Manager
@iizrailevsky
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

3. Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your
product is working (or not)
Agenda

4. Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your
product is working (or not)
Agenda

5. 5
“The sooner you run an
experiment, the sooner you
are likely to find a surprise,
and the surprise is the market
speaking to you, telling you
something you didn't know.
Several of our businesses
here came out of surprises.”
– Scott Cook, Intuit founder
& chairman
Intuit’s tradition of experimentation
Deep
Customer
Empathy
Go Broad to
Narrow
Rapid
Experiments
with
Customers
Customer
Delight
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

6. 6
What is an A/B test?
BA
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

7. Why A/B Test?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Discover which
experiences
delight your
customer
Accelerate your
learning loop
Make faster,
better decisions
with data

8. 8
Who We Serve

9. 9
Introducing:
WASABI
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

10. 10
First time use test: QuickBooks Self Employed
Test: Complete tax profile first Control: Connect account first
In the test cell, we see:
• Significant increase in tax profile completion
• Slight increase in categorizing a transaction from a card
• No change to connecting a bank
• No change to categorizing on the transactions page
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

11. 11
First time use test: QuickBooks Self Employed
Test: Complete tax profile first Control: Connect account first
In the test cell, we see:
• Significant increase in tax profile completion: +20%
• Slight increase in categorizing a transaction from a card: +6%
• No change to connecting a bank: Flat
• No change to categorizing on the transactions page: Flat
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

12. Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your
product is working (or not)
Agenda

13. 13
• Marketing vs Product (changing content & UI vs features or back-end)
• How much engineering is involved?
• Do you have sensitive data?
• What platforms and devices do you need to test across?
• Cost
• Analytics provided / options
• Efficient use of traffic
• Performance
• Ease of use
How do you choose a tool for A/B testing?
What are the dimensions you should think about when choosing A solution?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

14. 14
Landscape of Popular Options
• WYSIWIG
• Minimal Engineering
• Marketing focused
• Developer focused
• Flexible engineering effort
• Own your own data
• Customized
• Ongoing maintenance
• Own your own data
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
COMMERCIAL OPEN SOURCE IN-HOUSE
WASABI

15. Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your
product is working (or not)
Agenda

16. 16
The A/B Testing Lifecycle: Iterate to Awesomeness
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Step 1:
Plan an experiment
Step 2:
Develop a feature to test
Step 3:
Instrument your application with Wasabi
API Calls
Step 4:
Create and run the experiment
using the Wasabi UI
Step 5:
Deploy and track results
quickly

17. 17
• Is this user already assigned for this experiment?
• Does the user qualify for the experiment? (Segmentation Rules)
• Is the user already in a conflicting experiment? (Mutual Exclusion)
• Randomization
• Check vs. Sampling Percentage
• Assign to an experience (Bucket)
• Return assignment
How does user assignment actually work?
What should I show this user when a WASABI test is active?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

18. Demo

19. 19
Architecture: behind the scenes, here’s how it’s built
Web tier
Admin UI API API
Segmentation
Mutual Exclusion
Sampling
Bucket Dice Roll
Application tier
Cassandra MySQLData tier
Authentication
User Directory
Logging
Metrics
Pluggable
Components
Bucket
assignments
Events / analytics
Kafka
Stream
HDFS Spark
Vertica S3
Pluggable
Components
Wasabi A/B Testing Service
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky

20. Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your
product is working (or not)
Agenda

21. 21
Best practices and considerations for a good A/B test
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Get started! (Don’t let the perfect be the enemy of the good)1
Think about what you’ll learn from a “failed” test in advance.
Be skeptical – don’t take your results at face value
• Track all your key metrics – check for unintended consequences
• Think about sources of bias, assumptions: timing, infrastructure, etc.
Don’t stop at one round of experiments!
2
3
4
5
Don’t forget qualitative research as well

22. 22
What Should You Take Away?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Everyone can A/B test!
(It’s not just for marketers)
A lot of A/B testing tools in the
market. Choose wisely!
Take time to come up with your
hypothesis and how you’ll
measure it. It can make or break
your experiment.

23. 23
If you’re interested, here’s how you get Wasabi:
https://github.com/intuit/wasabi
Questions?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Contact us: data@intuit.com

24. Questions?

25. Thanks for coming!
We’d love your feedback.
Before you head out, take a minute to rate this session in the
QuickBooks Connect mobile app.
Keep the conversation going
developer.intuit.com/hub/quickbooks-connect
Access Session Slides
Engage with Speakers and Fellow Developers
Get Questions Answered