Microsoft's Analysis and Experimentation team has enabled many different feature teams to run controlled experiments in order to make feature ship/no-ship decisions. In this talk, I will review several case studies of experiments run by teams at Microsoft. I will highlight both the value that running controlled experiments has provided these teams as well as the challenges encountered in order to get trustworthy results from the controlled experiments. Presented by Brian Frasca, Partner Data Scientist Manager at Microsoft.

1. #datapopupseattle
Running Effective Controlled Experiments
(aka A/B/n Tests)
Brian Frasca
Partner Data Scientist Manager,
Analysis & Experimentation at Microsoft
microsoft

2. #datapopupseattle
UNSTRUCTURED
Data Science POP-UP in Seattle
www.dominodatalab.com
D
Produced by Domino Data Lab
Domino’s enterprise data science platform is used
by leading analytical organizations to increase
productivity, enable collaboration, and publish
models into production faster.

3. Running&Effective&Controlled&
Experiments&(aka&A/B/n&Tests)
Brian&Frasca&
Partner&Data&Scientist&Manager,&Microsoft

4. Intro&to&Controlled&Experiments

5. We&have&lots&of&ideas&to&make&our&products&
better…

6. But&not&all&ideas&are&good!

7. Don’t&ship&all&ideas&–&just&the&good&ones
• Recipe&
• Come&up&with&lots&of&ideas&
• Test&your&ideas&to&see&which&are&good&and&which&aren’t&
• Ship&only&the&good&ideas&
• Good&idea,&right?&
• Surprisingly&hard&to&do&in&practice&
• How&do&you&tell&the&good&ideas&from&the&bad&ones?&(main&topic&of&this&talk)&
• People&tend&to&think&that&all&of&their&ideas&are&good&
• People&tend&to&distrust&the&data&when&it&tells&them&otherwise&
• People&think&it’s&wasteful&to&do&work&that&is&thrown&away&
• Strong&bias&toward&shipping:&if&you&build&it,&you&must&ship&it

8. “Ship&It”&mentality
Celebrate&success&after&shipping&product&changes

9. “Make&It&Better”&mentality
Celebrate&success&after&the&data&clearly&shows&the&product&is&better&

10. How&can&we&tell&which&ideas&are&good?
• Measure&user&behavior&before&and&after&ship&
• Misses&time&related&factors&such&as&external&event,&weekends&&&
holidays,&etc.&
• Measure&impact&to&user&behavior&using&a&Controlled&
Experiment&(aka&A/B&or&A/B/n&Test)&
• Controls&for&external&factors&such&as&time
Website&A Website&BWebsite&A Website&B
Oprah&calls&Kindle&
"her&new&favorite&
thing"&
Before&and&after&example

11. Running&a&Controlled&Experiment&
• Simple&concept:&
1. Randomly&split&traffic&between&two&(or&more)&
versions&
• Control:&Existing&System&&
• Treatment(s):&Feature(s)&being&tested&
2. Collect&metrics&of&interest&
3. Analyze&&
• Must&run&statistical&tests&to&confirm&differences&are&
not&due&to&chance&
• Best&scientific&way&to&prove&causality,&i.e.,&the&
changes&in&metrics&are&caused&by&changes&
introduced&in&the&treatment(s)
Control:&Existing&
System
Treatment:&
Existing&System&
with&Feature&X
50%&Users 50%&Users
Users&interactions&instrumented,&
analyzed&&&compared
Analyze&at&the&end&of&the&
experiment
Is&my&new&Feature&X&effective?

12. Keys&to&Making&Experiments&Successful
• Ability&to&easily&change&features&for&real&users&
• Great&in&services&world;&not&so&easy&for&hardware&
• Agile&development&cycles&
• Reliable&user&behavior&data&
• Clear&objective&or&“OEC”&(Overall&Evaluation&Criterion)&
“If&you&don't&know&where&you&are&going,&any&road&will&get&you&there.”&
&&&&&&&&&&&&bb&Lewis&Carroll,&Alice&in&Wonderland&
• Metric&driven&culture&
• Do&you&want&to&ship&features&or&move&metrics?&–&different&mindset

13. Safari&users&aren’t&clicking!
IE&users&are&the&most&engaged&and&Safari&users&are&barely&engaged&at&all.&
Should&we&trust&this&conclusion?&
Twyman’s)Law:)Any&figure&that&looks&interesting&or&different&is&usually&wrong.&
No.&The&reliability&of&click&data&differs&across&browsers.&
Observation:&People&tend&to&rationalize/explain&whatever&data&you&show&them.
Page&Click&Rate
0%
20%
40%
60%
80%
Chrome Firefox IE Safari

14. Overall&Evaluation&Criterion&(OEC)&for&Bing
• Business&goals:&increase&query&share&and&revenue&per&search&
• OEC&=>&combination&of&distinct&queries&per&user&and&revenue&per&user&
• Easy,&right?&
• Experiment&with&a&bug&showed&poor&results&to&users&
• Distinct&queries&per&user&increased&10%&
• Revenue&per&user&increased&30%&
• OEC&above&is&not&even&directionally&correct!&What&happened?&
• Poor&organic&search&results&impacted&user&behavior&
• Desired&result&missing&=>&try&a&different&query&
• No&good&search&result&available&=>&click&on&an&ad&result&instead

15. Our&Team

16. Microsoft&Analysis&&&Experimentation&(A&E)&
• Microsoft&Analysis&&&Experimentation&team&
• A&E&team&provides&a&platform&and&service&for&running&controlled&experiments&
• Team&focuses&on&experimentation&and&deep&dive&analysis&
• Our&mission&statement:&
! Accelerate!innovation!through!trustworthy!analysis!and!experimentation!
• Background&
• Got&Bing&to&scale:&run&over&10,000&experiments/year&
• Published&many&papers&on&experimentation:&http://www.expbplatform.com/&
• Now&bringing&experimentation&to&teams&across&Microsoft

17. Experimentation&Partners&&
We&have&taken&Bing&to&scale&(viz.,&thousands&of&experiments).&Our&new&frontier&is&scaling&experimentation&to&
new&partners&across&Microsoft.&Some&of&our&partners&include:

18. MSN&Case&Study
Infopane&Slides

19. MSN
• Portal&website&
• Team&wants&to&increase&user&
engagement&(more&clicks&
and&page&views)&
• “Infopane”&automatically&
rotates&through&set&of&slides

20. Experiment:&Add&More&Infopane&Slides
Control Treatment
12&Slides 16&Slides

21. User&Engagement&Drops!&Or&Does&It?
• Initial&Result:&The&opposite&of&hypothesis&
• Steep&drop&in&user&engagement&
• BUT,&the&result&came&with&warnings&in&data&quality

22. Sample&Ratio&Mismatch&(SRM)
Control Treatment
Users
50% 50%
50.2% 49.8%
Ideal:
Observed:
• Data&quality&warning&
• The&number&of&users&from&the&control&
and&treatment&were&too&unbalanced.&
• It&turned&out&that&too&aggressive&bot&
filtering&classified&more&users&in&the&
treatment&as&bots.

23. User&Engagement&Actually&Increased!
• Result&after&bot&filter&fix:&strongly&positive&
• Increase&in&user&engagement&(page&views/user,&clicks/user)&and&revenue&
• Lessons&from&this&experiment&
• Getting&trustworthy&data&is&very&important&but&it&can&be&difficult&
• Experimentation&provides&a&way&not&only&to&test&our&intuition&but&also&to&
discover&data&quality&issues

24. Xbox&Case&Study
Killer&Instinct

25. Xbox&–&Killer&Instinct
• Fighting&video&game&that&runs&on&
the&Xbox&One&console&
• Freemium&model:&can&play&one&
character&for&free&(Jago)&but&must&
pay&to&play&others&
• Team&hopes&to&increase&revenue&by&
getting&players&to&purchase&
additional&characters

26. Experiment:&Change&Free&Character
Is&revenue&affected&by&which&
character&is&offered&for&free?&
Guess&the&outcome:&
A. Jago&=>&more&revenue&
B. Glacius&=>&more&revenue&
C. No&significant&difference
GlaciusJago

27. Glacius&Wins!&Or&Does&He?
• Revenue&increased&for&Glacius&but&
engagement&(time&in&game)&decreased!&
• Mixed&outcomes&are&common&=>&teams&
must&decide&how&to&tradeoff&between&
various&metrics&
• Tradeoff&should&be&determined&abpriori&and&
built&into&the&OEC&(Overall&Evaluation&
Criterion)

28. WrapZup

29. Key&Takeaways
• Make&your&products&better&–&not&just&different&
• Not&all&ideas&are&good&
• Test&your&ideas&in&controlled&experiments&to&see&which&ones&are&good&
• Only&ship&the&good&ideas&
• Invest&in&trustworthy&data&
• Getting&data&is&easy;&getting&data&you&trust&is&hard&
• If&you&can’t&trust&the&results&then&doing&the&measurement&doesn’t&help&you

30. We’re&hiring!!!
• We’re&looking&for&Data&Scientists,&Software&Developers,&and&Product&
Managers&passionate&about&using&data&to&make&things&better&
• Interested?&Contact&us&at&aebjobs@microsoft.com&
• See&our&papers&here:&http://www.expbplatform.com/

31. #datapopupseattle
@datapopup
#datapopupseattle

32. #datapopupseattle
Thank You To Our Sponsors