Presentation for the 2018 Email Evolution Conference on using Machine Learning and AI to build smart segments and content optimization for email deliver.

1. Using AI-powered Dynamic Email
Composition to drive 1:1 Relationships
Ian Thomas, Microsoft

2. About me
1.6B1,300
20B 2.4B
• Run global Marketing Operations &
Analytics team at Microsoft
• Responsible for Windows, Surface, Xbox,
Bing, Microsoft Rewards, Windows Store
• 18 years in Web Analytics, Big Data &
Digital Marketing operations

4. We want to create
fans
Computing is more
personal
Customers expect us
to know them

5. Delight our customers
with highly relevant
communications
Make communications
a value-added part of
our customers’
Microsoft experience

7. The solution:
Think differently
about digital
marketing

8. Email Marketing
Traditional Model
Timescale:
Days/Weeks

9. Optimization-driven
Campaign Model
Machine Learning
Timescale:
Hours

10. What is a predictive model?
Red sky at night, sailor’s
delight.
Red sky in the morning,
sailor’s warning.
“
”

11. Goals & Outcomes

12. Key steps for outcome-driven marketing
Pick a
measurable
outcome and
stick with it
Use prediction
to identify the
best audience
Optimize to
match the right
creative &
messages to the
audience

13. Rethinking How We Send Email
From Lists, Templates and Mail Merge to Individuated Emails
Schedule
Generate Personalized
Content
Track Response
Optimize
Select Single
Customer Profile
Predict Optimal
Send Time
Send
Decision
Engine
Outcome

14. Creative BuildCampaign Setup
Modeling EngineData
Load to memory
Segment feature
selection
Build / update model
Azure
Data Lake
User
Profile
Model
Library
Product
data
User
Behavior
Merge Engine
Score & predict
HDInsight
Ruleset
Delivery
Modularized
templates
HTML list
Goal Audience Offers
Creative Library
Results

15. Dragon Ball FighterZ launch
Build traditional
target audience
based on genre
Deliver email to
fraction of
audience
Gather responses
(clicks)
Refine audience
with click-
propensity model
Deliver email to
remaining
(modeled)
audience
Lift vs. BAU Segments
Open Rate CTR
Treatment 1 329% 92%
Treatment 2 274% 72%
Results

16. Monster Hunter World launch
Build traditional
target audience
Within audience, build
propensity-based
segments for RPG &
Crafting genres
Measure results by
creative
Lift vs. BAU Segments
Game Play Rate
Crafter segment 336%
RPG segment 87%
Results
Send 2 creatives to
audience by genre
propensity

17. Windows 10 Onboarding

18. Windows Welcome – Janet’s Journey
Buys a Win10
PC, sets it up
Day 0
Generic
welcome email
Installs Office,
downloads Chrome
Day 4
Customized
engagement email
Sets up OneDrive,
tries Edge
Signs up for MS
Account, downloads
from Store
Day 7
Customized
engagement email

19. Games with Gold
Optimization/modularization of the GwG
monthly newsletter can drive better game
download & other engagement outcomes
Use multi-armed bandit experimentation to
test creative/offer variants across multiple
audience attributes, using ML to make
selections & improve performance over time
Drove lift in key metric, but cost and
complexity of setup was very high and some
other metrics suffered

20. The Big Question
“Will I be
replaced by a
robot?

22. What we’ve learned
It’s about people, not just tech
Take it one step at a time
Guardrails are important
We are learning as we go

23. What you should do
1. Focus on outcomes, not actions
2. Treat everything as an experiment
3. Set guardrails
4. Use data to generate new ideas
5. Hug a Data Scientist

24. Thank you
www.liesdamnedlies.com
@ian_thomas
Ian Thomas
ian.thomas@microsoft.com
linkedin.com/in/iandthomas/