This document discusses how organizations can harness data and experiments to build engaging individual outreach campaigns. It recommends incorporating holdout groups into testing, constantly iterating tests, and measuring efficacy at the individual level. BlueLabs uses techniques like uplift modeling and predictive analytics to identify those most likely to take action from outreach and engage in multistage processes. Relevant inputs like behavior, circumstances, community data, and creative modeling approaches are important to success.

1. BlueLabs | Data, Analytics, Technology.
Changing Behavior with
Experiments and Modeling
Dan Porter
Co-founder & Chief Analytics Officer, BlueLabs
Driving Strategy with Data

2. www.bluelabs.com Confidential and ProprietaryBlueLabs | Data, Analytics, Technology.
Media consumption is fragmenting
Digital is no longer a ubiquitous, single medium
Cord cutters are an increasingly large segment of the population
Companies and other organizations have more decisions than ever about
where to deploy resources to change human behavior
Current Environment

3. BlueLabs | Data, Analytics, Technology. www.bluelabs.com @blue_labs

4. Question: How can
organizations other than e-
commerce harness the power of
data to build engaging
individual level outreach
campaigns
BlueLabs | Data, Analytics, Technology. www.bluelabs.com @blue_labs

5. www.bluelabs.com Confidential and ProprietaryBlueLabs | Data, Analytics, Technology.
Always be Testing: Rather than have a “test” followed by a “program”,
holdout groups should be incorporated into any outreach campaign
Track outcomes over time: Rather than testing a given thing on a given
day, constantly iterate and track how outreach affects behavior long into the
future
Measure efficacy at individual level: Use uplift modeling to measure how
much more likely an individual is to take an action as a result of receiving
outreach
Answer:

6. BlueLabs | Data, Analytics, Technology.
We leverage data to nudge
citizens/patients/customers
towards taking actions and
completing multistage processes

7. BlueLabs | Data, Analytics, Technology.
What is Persuasion (uplift)
Modeling?

8. In persuasion modeling, we are
measuring and predicting:
The likelihood of someone to take an action as a result
of receiving specific outreach
We are not measuring:
Current
intent
Issue
importance
Interests

9. There are 4 types of people we
identify for persuasion modeling:
The key is to find the people who will benefit most from a treatment or message:
Takes no action with
treatment
Takes action with treatment
Takes no action without
treatment
Lost Causes Persuadables
Takes action without
treatment
Do Not Disturbs Sure Things

10. Some People React Negatively to
Campaign Contact

11. Some People React Negatively to
Campaign Contact

12. Some People React Negatively to
Campaign Contact

13. www.bluelabs.com Confidential and ProprietaryBlueLabs | Data, Analytics, Technology.
Real World Examples of “Sleeping Dogs”
• Message that undermines core brand to a
particular customer with the wrong message
• Inadvertently “reminding” somebody to cancel
an auto-renewal
• Sending too many emails and having the
recipient unsubscribe
• Trying to sell something to somebody too soon
after previous sale

14. Experimental Design

15. Persuasion Modeling
Goal: identify heterogeneous treatment effects
y=α + β’Xi + γTi+ σ’XiTi

16. Avoid
Persuasion scores target people
at the individual level:
Example Universe by Persuasion Score
Lift
Persuasion
Weak Targets Strong Targets

17. BlueLabs | Data, Analytics, Technology.
Digital Services Analytics

18. Many applications across industries
Healthcare Banking Automotive Many
Desired Outcome
Reduce Appointment
Flake Rate
Move Services
Online
Service car at
dealership
Opt into paperless
communication
Treatment
Reminder SMS,
Phone Calls
Reminder at
branch and on
phone
Online/offline
outreach, in car
alerts
Online outreach
with custom
message
Interesting Data
Input
Previous Flake Rates,
distance from office,
login to website
Failed attempts
using
website/app
Connected Car
Data
Lack of
responsiveness to
mail, online
engagement

19. iPhone released in 2007
$399
$80/month after fees
2 year contract
Good credit required.
Mobile broadband is now accessible
Android phone from
MetroPCS today.
$59
$30/month.
No contract.
+ Tethering allowed for computer
access.

20. BlueLabs | Data, Analytics, Technology.
By 2020, 41 million Americans
will be mobile only internet
users.

21. Confidential and Proprietary
Least Accessible Most Accessible
Mobile Web Desktop Web
Brick and mortar
locations
Brick and mortar
locations
Desktop Web Mobile Web
2007
2020
THE DIGITAL DIVIDE IS REVERSING

22. www.bluelabs.com Confidential and ProprietaryBlueLabs | Data, Analytics, Technology.
Awareness: People generally understand how to interact with entities via
legacy means, but need to learn about the digital services available to them.
Multiple Sessions: Processes are completed over multiple sessions.
Agencies must develop strategies to bring users back to complete
processes.
Feedback Loops: Over time, platforms must adapt to the populations that
use them. Experimentation and predictive modeling are keys.
Continuous Engagement: To make the best use of a platform, users may
return more frequently than they do for in-person services. This opens
opportunities to build programs that are efficient and effective.
FUNDEMENTAL DISTINCTIONS OF DIGITAL
SERVICES

23. Confidential and Proprietary
ENGAGEMENT MODELING
Identify individuals who are either
at retention risk or can benefit
from additional program use
Use randomized experiments
trials to identify individuals most
likely to take action if contacted
Operationalize contacts across
channels, including mail, phone,
SMS, in person, and online
Continually measure program
impact via longitudinal holdout
group
Replicate process to improve
additional area of program
engagement
We create a positive feedback loop of using individual outreach to drive
program engagement. This process has significantly moved the needle
on program success.

24. www.bluelabs.com Confidential and ProprietaryBlueLabs | Data, Analytics, Technology.
EXAMPLE MODELS
• Likelihood to complete a process if customer receives a particular piece of
outreach via a particular channel.
• Likelihood of opting out of an auto-renewal if a person receives a reminder,
vs receiving no reminder
• Preference for completing an application process online, on the phone, or
via digital service.
• Likelihood of unsubscribing from an email list if a person receives 3 emails,
but not if they receive one.

25. www.bluelabs.com Confidential and ProprietaryBlueLabs | Data, Analytics, Technology.
Behavior: Time since last engagement, number of validation errors, incorrect password attempts,
modes of engagement (online, in person, or call center), average number of engagements per
completed action.
Circumstances: Drive time trelevant brick and mortar locations, household level information,
purchasing power.
Community: Census variables, program uptake within community, health metrics, education.
CREATIVE MODELING INPUTS DRIVE SUCCESS
MORE THAN ALGORITHM SELECTION
EXAMPLE MODELING INPUTS
y=α + β’Xi + γTi+ σ’XiTi

26. www.bluelabs.com Confidential and ProprietaryBlueLabs | Data, Analytics, Technology.
TAKEAWAYS
• Organizations other than e-commerce can still target individually with uplift
modeling and experiments—generally high ROI compared with other types
of targeting
• Relevant content is important-–often driven much more by recent behavior
than demographics
• Creativity around what data is predictive is not strictly a statistical skill but
will make or break predictive models
• Multistage processes particularly conducive to such methods

27. BlueLabs | Data, Analytics, Technology.
Thank you!
Dan Porter
Co-founder & Chief Analytics Officer, BlueLabs
Driving Strategy with Data