MRS Speaker Evening- hosted by the ADA Network

supported by
MRS Speaker
Evening
hosted by the ADA
Network

•CPD and
recognition
•Mentoring
10 Speaker events
6 Webinars
50 e-books
100s of journals
CPD
•Accreditation
Specialist briefings
Bespoke training
Government
rosters
TPS exemption
•Annual Conference
(Impact)
•MRS
•Code of
•Conduct
•Sector
insight and
knowledge
•MRS Awards,
Excellence and
Oppies•Standards and
regulation
•Standards,
•Policy, thought
•leadership

For 60 years
the MRS
Code of
Conduct has
been the
foundation of
good
research.
MRS
Code of
Conduct
The Code
helps to
protect
providers,
buyers and
respondents
Research activities are
rooted into ethical
considerations.
Researchers look at two
major elements
• doing right and
• preventing harm,
through protecting
privacy, obtaining
consent.
Generally speaking, an
understanding of ethics is
rooted and should be seen
as an essential part of
acknowledging the
researcher's identity, their
own position in the
research and as a means
of justifying their research
approach
Ethics

Evidence-based resource allocation
27 June 2019
Alex Maksimov, Analytical Consultant, Google UK
alexmaksimov@google.com
Application in Paid Search marketing

Growing accountability in digital marketing

Growing accountability in digital marketing
Evidence-based resource allocation

How do I
define
return on
investment?
How do I
model
return on
investment?
How do I
allocate
investment
optimally?
Three key questions

Marketing return on investment
is poorly defined

ROI
Best practice from finance?
ROA
NPV IRR
ROS
EVA
GM OM
EBIT EBITDA NOPAT
ROE
Return on
Investment
Return on
Assets
Return on
Sales
Return on
Equity
Net Present
Value
Internal Rate
of Return
Gross
Margin
Operating
Margin
Economic Value
Added
Earnings Before
Interest and Taxes
Earnings Before
Interest, Taxes,
Depreciation and
Amortisation
Net Operating Profit
After Tax

Should marketing strive for this level of precision?

Probably not...

Probably not...
...but a healthy mix of metrics is essential
Absolute
vs
Relative
Revenue
vs
Profit
Average
vs
Marginal

$2K$10K
Absolute vs Relative
ROAS 5x
Investment
Revenue $50K
Campaign A
10x
Campaign B
$20K

$2K$10K
Absolute vs Relative
ROAS 5x
Investment
Revenue $20K
Campaign A
10x
$20K
Campaign B
Extra revenue
$30K

Revenue vs Profit
Campaign A Campaign B
$10K$10K
$5K $5K
$5K $5K
Revenue
Marketing
Investment
ROAS 2x 2x

Revenue vs Profit
$10K$10K
$5K $5K
$1K
$4K
$4K
$1K
Revenue
Profit
Product Cost
Marketing
Investment
ROAS 2x 2x

Revenue vs Profit
$10K$10K
$5K $5K
$1K
$4K
$4K
$1K
Revenue
Profit
Marketing
Investment
ROAS 2x 2x
Profit Difference
$3K
Product Cost

Average vs Marginal
Average ROAS
5x
4x
Average ROAS
3x
6x

Average vs Marginal
Average ROAS
5x
Marginal ROAS
4x
Average ROAS Marginal ROAS
3x
6x

1x
Average vs Marginal
Average ROAS
5x
Marginal ROAS
4x
Average ROAS Marginal ROAS
3x
6x
ROAS Difference

A healthy mix of investment metrics for better decision making
Common sense...
...but often not present in marketing reports
Absolute
vs
Relative
Revenue
vs
Profit
Average
vs
Marginal

We are used to seeing historical data in a time series format

Supplement time series with relationships
We are more interested in the relationship
between marketing investment and business
outcomes than trend over time

Start by modelling traffic as a function of marketing spend
Investment
Historical weekly observations
Observation

Investment
Observation
Model

Decreasing returns to scale
● Non-linear relationship
● Concave
● Often power or logarithmic functions
Investment
Observation
Model

Overlay business metrics
INVESTMENT

INVESTMENT INVESTMENT
x
CONVERSI
ON RATE

INVESTMENT INVESTMENT INVESTMENT
x
CONVERSI
ON RATE
x
ORDER
VALUE

Ability to anticipate business outcomes
A flexible model to estimate business outcomes at
● Different levels of spend
● Different resource allocations

Ability to make better decisions
Investment

Ability to make better decisions
Investment
+11
+5
+200 +200

Ability to allocate resources optimally
Investment
Campaign A
Campaign B

Investment
Campaign A
Campaign B
Current
Spend
Current
Spend

Investment
Campaign A
Campaign B
Current
Spend
Current
Spend
Marginal Return
Marginal Return

If allocation is even remotely optimised,
the next marketing dollar should result in fewer
sales than the previous
Optimally allocating a finite resource is at the heart of business strategy

Optimisation is achieved by making trade-offs between alternatives
Investment
Campaign A
Campaign B
Current
Spend
Current
Spend
Optimal
Spend
Optimal
Spend

But most advertisers are faced with too many choices
Response curves for different campaigns

Linear programming definition
Linear programming in its most basic form

A method to
achieve the best
outcome in a
model

Objective function
What do we want
the algorithm to
do?
A method to
achieve the best
outcome in a
model

Objective function Constraints
What do we want
the algorithm to
do?
Are there any
limits or threshold
that need to be
taken into
account?
A method to
achieve the best
outcome in a
model

Linear programming application in digital marketing
Maximise Conversions
Maximise Revenue
Maximise Profit
Objective function
Maximise Traffic
Allocate investment
to

Linear programming application in digital marketing
Maximise Conversions
Maximise Revenue
Maximise Profit
Budget < $XM
ROAS > X
CPC < $X.XX
Constraints
Objective function
CPA < $X.XX
Maximise Traffic
Subject to
Allocate investment
to

An example
Maximise
Traffic
Maximise
Conversions
Maximise
Revenue
Maximise
Profit
Objective
functions

Maximise
Traffic
Maximise
Conversions
Maximise
Revenue
Maximise
Profit
Investment ($M) 10.00 10.00 10.00 10.00
Objective
functions
An example
Input

Maximise
Traffic
Maximise
Conversions
Maximise
Revenue
Maximise
Profit
Traffic (M)
Conversions (K)
Revenue ($M)
Profit ($M)
Investment ($M) 10.00 10.00 10.00 10.00
ROAS (x)
CPC ($)
CPA ($)
ROI (x)
Objective
functions
An example
Constraints
Outcomes
Input

Maximise
Traffic
Maximise
Conversions
Maximise
Revenue
Maximise
Profit
Traffic (M) 28.25 27.63 25.83 26.47
Conversions (K) 56.01 63.42 57.36 56.66
Revenue ($M) 35.92 36.43 38.13 35.47
Profit ($M) 14.44 15.02 14.59 15.16
Investment ($M) 10.00 10.00 10.00 10.00
ROAS (x) 3.59 3.64 3.81 3.55
CPC ($) 0.35 0.36 0.39 0.38
CPA ($) 178.53 157.69 174.35 176.5
ROI (x) 1.44 1.50 1.46 1.52
Objective
functions
Constraints
Outcomes
Input
You’re already making trade offs, even if you don’t realise it

Maximise
Traffic
Maximise
Conversions
Maximise
Revenue
Maximise
Profit
Revenue ($M) 38.13 35.47
Profit ($M) 14.59 15.16
Investment ($M) 10.00 10.00
ROAS (x) 3.81 3.55
ROI (x) 1.46 1.52
Objective
functions
Constraints
Outcomes
Input
You’re already making trade offs, even if you don’t realise it

How do I define
return on investment?
Recap and recommended actions
❏ Audit your mix of metrics
❏ Include absolute, profitability and marginal
metrics

How do I define
How do I model
metrics
❏ Look beyond time series
❏ Identify relationships between variables
❏ Construct flexible models for decision making

How do I define
How do I model
How do I allocate
investment optimally?
metrics
❏ Look beyond time series
❏ Identify relationships between variables
❏ Construct flexible models for decision making
❏ Utilise linear programming
❏ Make the right trade-offs for your business

Ipsos MORI | November 2018 | Version 1 | Client-Internal Use Only
Using Propensity Score Matching to evaluate the effect
of advertising on our client’s brand image
Presentation to the MRS
Andrew Zelin, Ipsos MORI
27 June 2019

60
What we will cover
• Background to the Problem
• Propensity Scoring Methodology
• Using the scores to create a matched sample of
controls for the cases
• Other applications
• Conclusion

61
Existing Relationships II
A leading digital advertising company needed to assess whether seeing one of the their
advertisements is associated with having a positive image of the brand
Are those who have seen one of the advertisements, more likely to have a positive
image of the brand than those that have not?

62
Overview I
BUT…people who have higher levels of skills in digital areas and / or use this as part of
their careers (ie are more “tecky”)
• are more likely to see the ad
• and are also more positive towards the brand than people who are not

63
Existing Relationships I
Seen ad
+ve Image of
Brand
“Teckyness” of
respondent
Effect of interest
Confounder Confounder

64
Existing Relationships II
Seen ad
+ve Image of
Brand
“Teckyness” of
respondent
So from this, we cannot tell easily how much of the positive image is due to
seeing the ad v being a user of digital technology.

65
Overview II
“Propensity Score” Matching
A Matched Evaluation to compare the ad viewers (cases) with the non-viewers
(controls).
• through the creation of a “Propensity Score” (PS) for each respondent in terms
of how likely they are to have seen the ad,
• followed by appropriate matching of the cases and controls on this PS and hence
levels of “Teckyness”.
Ad viewers
(cases):
Non-viewers
(controls):
Brand Image

66
Overview III
• By doing this, we were able to delineate how much of the increase in perceived
brand image for those who viewed the ad over those that did not, was genuinely
down to viewing the ad, rather than simply being more “tecky”.
• ….other methods tested – eg at the analytical stage / by bringing these into the
model using covariates, but that has not been successful – eg variance inflation
Brand Image
Ad viewers
(cases):
?
Non-viewers
(controls):
?

67
Creating a Propensity Score: Overview
• Set up an “Evaluation” methodology using
Propensity Scores (PS)
• “Propensity Score Matching” - first
documented by Robin in 1983.
https://en.wikipedia.org/wiki/Propensity_score_matching
https://academic.oup.com/biomet/article/70/1/41/240879
STEPS:
1. Calculate a PS for each respondent (viewers and non-
viewers);
2. Find the closest match non-viewer(s) on PS for each
viewer;
3. Derive a weight for each non-viewer based on how well
it generally matches to all viewers;
4. Create a fresh sample of non-viewers from the original
viewers with selection probability prop. to this weight
this fresh sample will be matched to the viewers on the
basis of “teckyness”.

68
Creating a Propensity Score: Step 1
0.00
Propensity (to be an ad-viewer) Score
1.00
0.50
Definition of Propensity Score (PS):
• An overall “teckyness” score based on a composite of
related variables,
• These variables have different relative weights /
influences on this, such that this PS correlates as
strongly as possible with viewing the ad.
Logistic regression modelling was carried out:
• To determine which “techiness” questions are
correlated with being an ad viewer v not an ad
viewer
• to derive their coefficients / influences;
• and to calculate a PS for every respondent;
PS is driven by ones responses to:
• (L_T1) Digital skills;
• (GQ2) Attitudes to technology; Understanding
how to use technology is important for your
career;
PS = 1/(1+exp(3.294-1.05*L_T1-1.982*GQ2)

69
Ad viewers
(cases):
Non-viewers
(controls):
Creating a Propensity Score: Step 2 (Closest match)
• 500 cases
• 900 controls
• 1:1 match of 500:500 means losing 400 controls
• How can we still use these 400?

70
Ad viewers
(cases):
Non-viewers
(controls):
• What was actually done was:
• Take each case / ad viewer in turn;
• Create a selection weight for each control, based on how similar each non-
viewer Propensity Score (PSnv) is to the viewer PSv - ie (1/PSv-PSnv)^2)
• If the difference between PSv and PSnv exceeds 0.05, set weight to zero.
• Repeat for each viewer;
• After that, take each non-viewer, and summate all of the weights received from
all of its matches from each viewer;
….and this total is the “selection weight” needed for each non-viewer
0.05
Creating a Propensity Score: Step 3 (Weights)

71
• Control respondents (non-viewers) with high selection weights are
those which are well matched / were under-represented when
aligning the cases and controls for “teckyness”, and hence need to be
weighted up.
• Controls with low selection weights are those which are poorly
matched / were over-represented and need to be weighted down.
The final step involves a fresh
sample of controls being selected
through “Bootstrapping” from
the current pool of controls,
…matching the cases in terms of
“teckyness” / PS.
To achieve a balanced sample
such that comparisons on brand
image scores can be made by
directly comparing all of the
cases with the re-sampled
controls
To select a balanced sample of non-viewers to
match the viewers
Creating a Propensity Score: Step 4 (Re-selection)

72
Comparing the Propensity Scores of the samples
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
0.036 0.057 0.059 0.091 0.093 0.096 0.141 0.145 0.148 0.212 0.217 0.222 0.313 0.319 0.435
Percentofsample
Propensity score
Comparing the distributions of PS for the not-seen ad sample before and
after PW, vs PS of seen ad group
Not_seen_Raw_sample Not_seen_matched_sample Seen_ad

73
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
0.036 0.057 0.059 0.091 0.093 0.096 0.141 0.145 0.148 0.212 0.217 0.222 0.313 0.319 0.435
Percentofsample
Propensity score

74
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
0.036 0.057 0.059 0.091 0.093 0.096 0.141 0.145 0.148 0.212 0.217 0.222 0.313 0.319 0.435
Percentofsample
Propensity score
Note that in terms of PS, the profile of the matched
sample of controls (blue) aligns much better to that of
the cases (green), than does the unmatched controls
(red)

75
Comparing Brand images before vs after matching
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Mean PS Brand
favourability
<Google>
Brand Advocacy
<Google>
WOM Advocacy
<Google>
Closeness to
<Google>
How well does
Google offer P&S
Do you use
Google offer P&S
Percentofsample
Propensity score
Comparing Brand images of those who have seen the ad against those
who have not; before and after matching (n=2,000)
Not_seen_ad - Without_PW Not_seen_ad - With_PW Seen_ad

76
Comparing Brand images before vs after matching
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Mean PS Brand
favourability
<Google>
Brand Advocacy
<Google>
WOM Advocacy
<Google>
Closeness to
<Google>
How well does
Google offer P&S
Do you use
Google offer P&S
Percentofsample
Propensity score
Comparing Brand images of those who have seen the ad against those
who have not; before and after matching (n=2,000)
Not_seen_ad - Without_PW Not_seen_ad - With_PW Seen_ad
Note that the “matching” closes the gap in brand image between the views and the non-
viewers, but even adjusting for this – the ad still has a positive effect on image overall

77
Comments and learnings
What will be done in future analyses:
• Use and test a longer list of variables in the PS model – eg age,
gender, ethnicity;
• Validate to compare matches of PS, using single variables used in PS
(eg digital skils), as well as the overall PS;
• The value of the final “Bootstrap” re-sampling is questionable. Is it
wasteful on respondents? Are we better to use 900 weighted controls?

78
Other Applications
Data Fusion – Merging large databases
Clinical Trials / Evaluating
Patient Pathways
Mixed mode surveys

79
Conclusions and Impact on the end-Client
• Propensity Scoring and Weighting methods may be effective in evaluations to
eliminate confounding effects;
• These scores can be used to create matches, which in turn can generate selection
weights needed to produce balanced samples
• Propensity Scoring and Weighting constitute robust and objective methodologies
to be developed using evaluations to assess the true impact of advertisement
exposure on brand image.
• A number of standard Key Drivers analyses had been carried out on brand image
taking in all of the possible confounding factors, although none really got to the
heart of the matter as to whether seeing the ad is associated with a positive image,
adjusting for respondents’ levels of “teckyness
• The method is Relatively easy to implement;
• Has a number of other implications and possible uses.

Thank you for your attention
Andrew Zelin
Lead Statistician
Healthcare, Ipsos

QUANTIFYING THE
QUALITATIVE
MRS ADVANCED ANALYTICS

QUANTIFYING THE QUALITATIVE
WHO ARE WE
▸Rob Bramwell, Consultant from CDO
Partners
▸Brighton-based data and analytics
organisation
▸Specialise in helping companies improve
through the use of data
▸Work with organisations from startups to
global enterprises

WHY ARE WE HERE
▸Talk about our experiences working with customers to help them
programmatically analyse and process qualitative data
▸Share some frameworks and approaches we’ve used
▸Give some examples of customer projects
▸Hopefully help you see how you can quantify qualitative data

WHY
QUANTIFYING THE
QUALITATIVE

ANALYTICAL
PATTERNS
▸What
▸Usually a simple measure - what have
we sold
▸How
▸Adding dimensions to those measures
▸Trends, timelines
▸Why
▸Statistical analysis
▸Qualitative analysis

ANALYTICAL PATTERNS
▸Useful in a wide range of applications
▸HR and performance - why are people leaving
▸Research and policy - what should we be driving
▸Customer interaction - why are people complaining
▸Sales and marketing - why are customers churning

HO
QUANTIFYING THE
QUALITATIVE

MAKING DATA
WORK IN
HEALTHCARE▸Recently been working with a range of
government organisations and NGO using
qualitative and quantitative data
▸Specifically in healthcare where a lot of the
data is in a range of formats
▸Very difficult to automate the capture,
ingestion and analysis of this data
▸Makes it incredibly time consuming

WE DON’T HAVE ENOUGH
ANALYSTS, THE DATA IS TOO
MUCH TO ANALYSE AND
WE’RE BEHIND ON RESEARCH
Head of Intelligence - Healthcare NGO

ANALYTICAL PATTERNS
▸What is happening
▸We can see raw numbers but not in context easily
▸How is it happening
▸We can’t see trends or dimensionality as none of the data is joined together
and comes in snapshots
▸Why is it happening
▸Interviews, notes, personal opinion, diaries…not sustainable

WHAT
DID WE
QUANTIFYING THE
QUALITATIVE

QUANTITATIVE
WHAT AND HOW
FIRST▸ Provides significant value
▸ Quantitative metrics
▸ Effort is involved in the integration of data
sources and modelling of dimensions
▸ Data storytelling
▸ Gives you a baseline
▸ Points them at key areas to explore next

QUALITATIVE
▸ Identify data sources
▸ Patient reviews
▸ Published reports
▸ Government reports
▸ Regulatory
▸ News feeds
▸ Social media

QUALITATIVE
▸ Define your key performance questions
▸ What do we need to know
▸ Define your key analytical questions
▸ What would we like to know
▸ What hypotheses do we have
▸ Define a taxonomy
▸ Helps structure the filters and searches

QUALITATIVE
▸ We used a platform called Squirro
▸ Faceted search application
▸ Runs on Elastic and Python
▸ Considered other platforms, NLP and RPA style
processing we have used elsewhere for similar
use cases
▸ Healthwatch England
▸ EDF

BUILD
APPLICATION
S▸ Data pipelines
▸ Build automated pipelines
▸ Smart filters
▸ Build filters based on taxonomy to automate
searching
▸ Discovery dashboards
▸ Define user stories and key questions

DEM
QUANTIFYING THE
QUALITATIVE

OUTCOM
ES
QUANTIFYING THE
QUALITATIVE

OUTCOMES
▸30% analyst time saved through automation
▸Backlog of 50% cases reduced to 0%
▸Volumes of processing increased
▸Research and policy analysis can be done within days not
months

NEXT STEPS
▸Additional sources of data
▸More automation in data review and taxonomy
▸Predictive modelling on topics and clusters

QUESTIO
NS
QUANTIFYING THE
QUALITATIVE

GET IN
TOUCH!▸ rob@cdo-partners.com
▸ www.cdo-partners.com
▸ @cdopartners

MRS Speaker Evening- hosted by the ADA Network

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MRS Speaker Evening- hosted by the ADA Network

Similar to MRS Speaker Evening- hosted by the ADA Network (20)

More from MRS

More from MRS (20)

Recently uploaded

Recently uploaded (20)

MRS Speaker Evening- hosted by the ADA Network