This document discusses how the cost and impact of experimentation depends on technology, business model, and other factors. It argues that experimentation is easiest for web/consumer products using a free business model, but gets increasingly difficult and risky for native apps, enterprise software, and situations where customers rely heavily on stability and predictability. The best practices suggested include obtaining direct customer feedback, using beta programs to test new features or fixes before full release, and leveraging free consumer versions of products to enable more experimentation.

1. EXPERIMENTATION FOR
ANY CIRCUMSTANCES
JOCELYN GOLDFEIN / @JGOLDFEIN
WWW.JOCELYNGOLDFEIN.COM

2. “It might work for
Facebook, but constant
experimentation couldn’t
possibly work for my team.”

3. FORM FOLLOWS FUNCTION

4. TECHNOLOGY DRIVES COST OF EXPERIMENTING
OPERATING
SYSTEMS
DESKTOP
APPS
MOBILE
APPS
WEB
The more you control the deployment environment,
the less friction there is to experimentation.

5. BUSINESS MODEL DRIVES IMPACT OF EXPERIMENTING
ENTERPRISE:
$$$$
CONSUMER:
$
FREE /
AD-SUPPORTED
ENTERPRISE:
$$$
CONSUMER:
$$
The more money someone pays you, the more likely
your software is “mission critical” for them, and the
more entitled they feel to predictability and reliability.

6. OTHER DRIVERS OF COST AND BUSINESS IMPACT
• Growth vs. Profit
• New customers vs. existing
• Caliber/experience of engineering team
• Partner Ecosystem
• etc…
• …left as exercise for the reader

7. EXAMPLES
WEB
FREE/
AD-SUPPORTED
CONSUMER:
$
CONSUMER:
ENTERPRISE:
$$$
ENTERPRISE:
$$$$
OPERATING
SYSTEMS
DESKTOP APP
MOBILE
APPS
$$

8. EXAMPLES
WEB
FREE/
AD-SUPPORTED
CONSUMER:
$
CONSUMER:
ENTERPRISE:
$$$
ENTERPRISE:
$$$$
DESKTOP APP
MOBILE
APPS
$$
OPERATING
SYSTEMS

9. 1X
10X
COST OF A MISTAKE
10X
100X
WEB
FREE/
AD-SUPPORTED
ENTERPRISE:
$$$$
OPERATING
SYSTEMS

10. ZONE 1: WEB / CONSUMER
• Experimentation: Easy
• Impact to users: Low
• Fixing mistakes: fast & easy
• Code bloat: doesn’t matter, your hardware
• Worst case scenario: hotfix
• Best practice:
• Go ahead and A/B test in production
• Rich server-side controls and visualization

11. NATIVE CODE MAKES IT HARDER
• Experiments: more work (code bloat, multi-platform)
• Fixing mistakes: slow (app store approval)
• Worst case scenario: app unusable until users
update
• Best practice:
• Server-side code, server-side killswitches - test in
production w/ great caution
• Beta / “pre-release access program”

12. CUSTOMERS RELYING ON YOU MAKES IT HARDER
• User impact of experimentation: High
• They need your software to do their jobs
• Worst case scenario: customer abandons your product
• Best practice:
• Key customer input/feedback in design cycle
• Beta program - users expectations do a 180
• Free or cheap consumer version of your enterprise product
as experimentation platform

13. PRODUCTION A/B TEST
EVERYTHING:
• DESIGN FEEDBACK
• BUGS
• METRICS IMPACT
DIRECT CUSTOMER INPUT:
• DESIGN FEEDBACK
BETA PROGRAM
• BUGS, DESIGN FEEDBACK
SERVER-SIDE A/B TESTS:
• JUDICIOUSLY
TYPICAL CHOICES
CLIENT BETA :
• DESIGN FEEDBACK
• BUGS
SERVER-SIDE A/B TESTS:
• METRICS IMPACT
• BUGS
WEB
FREE/
AD-SUPPORTED
ENTERPRISE:
$$$$
OPERATING
SYSTEMS
DIRECT CUSTOMER INPUT:
• DESIGN FEEDBACK
BETA PROGRAM
• BUGS, DESIGN FEEDBACK
FREE PRODUCT CHANNEL
• BUGS, TECHNICAL FEASIBILITY

14. CULTURE WAR

15. WEB
MOBILE
APPS
FREE/
AD-SUPPORTED
CASE STUDY: FACEBOOK’S TRANSITION TO MOBILE

16. QUESTIONS?
JOCELYN GOLDFEIN / @JGOLDFEIN / WWW.JOCELYNGOLDFEIN.COM