4. 2015
Success: an Ambiguous Word
Hmm, success …
Achieving scalable, available, efficient
solutions …
Millions of transactions per second …
A huge customer base …
6. 2015
What They Have In Common
Striving for a high success rate If failure, low impact
7. 2015
The Lean Experimentation Model
• Compare metrics to
hypotheses
• Get to root cause
• Savor the surprises
• Write down the Leap of
Faith Assumptions
• Select metric and test
method
• Declare the numeric
hypothesis you expect
to achieve
• Design it to be fast and frugal
• Collect behavioral data
13. 2015
Example: Lean Experiment on Mobile
Stage 0: Let’s build an inventory billing app
Stage 1: Bluetooth printing; get it working for one set of
customers (hardware stores)
Stage 2: Change direction – customer needs export of
transactions for accounting
Stage 3: Customer feedback – need email/SMS preferences
Stage 4: Let’s expand to other types of customers (food vendors)
Stage 5: Success for market segment
Inventory billing app for small business
14. 2015
Example: Lean Experiment in Cloud
Stage 0: Let’s create a A/B testing product
Stage 1: Get it working for one set of customers (TurboTax Online)
Stage 2: Customer feedback – works for them!
Stage 3: Customer feedback – needs advanced segmentation,
mutual exclusions on experiments, more features
Stage 4: Change direction – need scalability/reliability/availability
move to AWS
Stage 5: Let’s expand to other products (QuickBooks Online)
Stage 6: Let’s make it personalized with real-time A/B testing based
on decision models
Personalized A/B testing platform