Successfully reported this slideshow.
Your SlideShare is downloading. ×

Brand tracking with Bayesian Models and Metaflow

Ad

Brand Tracking with Bayesian
Models and Metaflow
Corrie Bartelheimer
Senior Data Scientist @ Latana
Berlin Bayesian, 10/11...

Ad

How many people have
heard of our brand?

Ad

How well known is the brand
in target group?
How do people perceive the
brand?
Have there been changes?

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Loading in …3
×

Check these out next

1 of 42 Ad
1 of 42 Ad

Brand tracking with Bayesian Models and Metaflow

Download to read offline

A common problem with brand tracking is that while online surveys are relatively cheap, the respondents are usually not representative of the general population but can be heavily biased. Additionally, clients are often interested in how their brand fares in specific target groups, for instance households with young children. It can be especially tricky to get reliable estimates for very small target groups.
Bayesian methods can help solve these problems but they are still not widely used in production: Bayesian models are commonly thought of as being too slow and too compute-intensive to use them at scale.
In this presentation, I talk about how a Bayesian approach addresses these problems and helps us to deliver more accurate results to our clients. By using AWS Batch together with the Netflix library Metaflow, we can run more than a thousand models in parallel per project thus allowing us to scale our operations.

Talk given at the Berlin Bayesians, 10/11/2021

A common problem with brand tracking is that while online surveys are relatively cheap, the respondents are usually not representative of the general population but can be heavily biased. Additionally, clients are often interested in how their brand fares in specific target groups, for instance households with young children. It can be especially tricky to get reliable estimates for very small target groups.
Bayesian methods can help solve these problems but they are still not widely used in production: Bayesian models are commonly thought of as being too slow and too compute-intensive to use them at scale.
In this presentation, I talk about how a Bayesian approach addresses these problems and helps us to deliver more accurate results to our clients. By using AWS Batch together with the Netflix library Metaflow, we can run more than a thousand models in parallel per project thus allowing us to scale our operations.

Talk given at the Berlin Bayesians, 10/11/2021

Advertisement
Advertisement

More Related Content

Advertisement

Brand tracking with Bayesian Models and Metaflow

  1. 1. Brand Tracking with Bayesian Models and Metaflow Corrie Bartelheimer Senior Data Scientist @ Latana Berlin Bayesian, 10/11/2021
  2. 2. How many people have heard of our brand?
  3. 3. How well known is the brand in target group? How do people perceive the brand? Have there been changes?
  4. 4. What makes Brand Tracking difficult?
  5. 5. Survey Problems Small target groups Signal or Noise? Representativeness of respondents
  6. 6. Traditional Approaches
  7. 7. Weighting
  8. 8. Weighting
  9. 9. Weighting
  10. 10. Quota Sampling
  11. 11. Quota Sampling Can take a while...
  12. 12. Introducing: Mr. P Multilevel Regression & Poststratification
  13. 13. Multilevel Regression
  14. 14. Multilevel Regression
  15. 15. Multilevel Regression
  16. 16. Multilevel Regression
  17. 17. Multilevel Regression
  18. 18. Multilevel Regression Multilevel
  19. 19. Multilevel Regression ♂ ♀
  20. 20. Multilevel Regression ♂ ♀
  21. 21. Multilevel Regression
  22. 22. Multilevel Regression
  23. 23. Poststratification
  24. 24. Poststratification
  25. 25. Poststratification Weight Prediction from our Model
  26. 26. Poststratification
  27. 27. Poststratification
  28. 28. On scale: Metaflow
  29. 29. One model per answer option
  30. 30. One model per answer option One question per brand, different competitor brands per market ~500 - 1500 models per project
  31. 31. One model per answer option One question per brand, different competitor brands per market ~500 - 1500 models per project ~20min per model = ~10 days compute time
  32. 32. Integrates with AWS Batch Easy to use for Data Scientists Supports reproducibility
  33. 33. MRP as Metaflow from metaflow import FlowSpec,step class MRPFlow(FlowSpec): @step def start(self): Self.data, self.questions = load_data() self.next(self.run_model, foreach="questions") @step def run_model(self): question = self.input self.result = run_mrp(question, self.data) self.next @step def join(self, inputs): for result in inputs: save(result) self.next(self.end) @step def end(self): pass
  34. 34. MRP as Metaflow from metaflow import FlowSpec,step class MRPFlow(FlowSpec): @step def start(self): Self.data, self.questions = load_data() self.next(self.run_model, foreach="questions") @step def run_model(self): question = self.input self.result = run_mrp(question, self.data) self.next @step def join(self, inputs): for result in inputs: save(result) self.next(self.end) @step def end(self):
  35. 35. MRP as Metaflow from metaflow import FlowSpec,step class MRPFlow(FlowSpec): @step def start(self): Self.data, self.questions = load_data() self.next(self.run_model, foreach="questions") @step def run_model(self): question = self.input self.result = run_mrp(question, self.data) self.next @step def join(self, inputs): for result in inputs: save(result) self.next(self.end) @step def end(self):
  36. 36. Parallelizing models @step def start(self): Self.data, self.questions = load_data() self.next(self.run_model, foreach="questions") @step def run_model(self): question = self.input self.result = run_mrp(question, self.data) self.next
  37. 37. Increasing resources @step def start(self): Self.data, self.questions = load_data() self.next(self.run_model, foreach="questions") @resources(cpu=8, memory=32000) @step def run_model(self): question = self.input self.result = run_mrp(question, self.data) self.next
  38. 38. Mr.P on AWS
  39. 39. Challenges Convergence How to monitor convergence of 1000+ models? More predictor variables Full joint distribution needed of all predictor variables.
  40. 40. Summary ● Multilevel-regression improves errors by using grouped structure ● Propagation of uncertainty improves weighting Corrie Bartelheimer Senior Data Scientist corrie.bartelheimer@latana.com
  41. 41. Introductory book on Bayesian Statistics: https://xcelab.net/rm/statistical-rethinking/ Stan: https://mc-stan.org/ Stan interface brms (R): https://paul-buerkner.github.io/brms/ MRP: Forecasting elections with non-representative polls https://www.sciencedirect.com/science/article/abs/pii/S0169207014000879 Metaflow https://metaflow.org/ MRP at Latana: - https://latana.com/whitepapers/mrp-vs-traditional-quota-sampling-brand-tracking/ - https://aws.amazon.com/blogs/startups/brand-tracking-with-bayesian-statistics-and-aws-batch/ Resources and Links

Editor's Notes

  • What is brand tracking? A company/brand is interested in how many people are aware of their brand (“brand awareness”).
  • Brands are usually interested in the following questions: How is their brand faring with specific target groups (e.g. they might only be interested in women, or only in people living in big cities).
    Another question is how people perceive their brand, what do they associate with it (is it fun, or intuitive, etc…).
    Of course, one important question is also if there have been changes over time. This is also relevant if the brand wants to know if their marketing campaign was successful and had an impact.
  • It’s easy to get a bunch of people to respond to an online survey but any target group (eg. women with kids) will most likely be small.
    This can make it hard (to impossible) to say if any change is due to a real change in how consumers see the brand or if it’s just a random change in the respondents composition.
    Also, online surveys are usually not very representative of the general population.
  • Traditional approaches (and their shortcomings)
  • One approach is to collect respondents data and weight the different groups afterwards. Let’s say we’re interested in both gender and the age.
  • We can then use census data to determine weights for each demographic cell. For each cell, we estimate the mean from the respondents data, multiply with the weight and sum up to get an estimate of the general population.
  • Imagine though we get data like this. Now the estimate based on only two respondents has the biggest weight and thus gets amplified. This means that small changes in this group can lead to big changes in the overall outcome. If we’re not interested in the overall estimate for the general population but for one of the subgroups, then (especially for the small groups) we get such large errors that the result is basically unusable.
  • Quota sampling goes the other way around and sets numbers of how many responses should be collected from each group.
  • The problem with quota sampling however, is that it can take a while to fill all cells with the number of responses needed. This also makes it more expensive.
    (Also possible to stop early with collecting data and then weight results).
  • Paper: see e.g. Forecasting elections with non-representative polls https://www.sciencedirect.com/science/article/abs/pii/S0169207014000879
  • First part: Multilevel Regression
  • The “Bayesian part” of the method, a hierarchical model. Our variable of interest “knows brand” is modelled by a Bernoulli likelihood
  • In our example from the previous slides, our predictor variables are gender and age.
  • We model gender and age as random intercepts/hierarchical.
  • Since it’s a Bayesian model, we also use priors of course.
  • Some intuition: for the gender parameter, this means that we estimate two parameters: male and female.
  • But since we group the two parameters, both come from a common distribution. This means that we allow both parameter to be different but we also think they should be similar.
    If there is little data for one group, then the model will estimate the parameter as close to the estimate for all related groups.
  • In our example this means that any demographic cell borrows strength from neighboring cells (similar groups).
  • The estimate for ♂, 30-60 is then the sum of the parameter for ♂ and the parameter 30-60. Both parameters also take into account the information from the other groups. Thus, each cell is not considered in isolation but we also use information from neighboring cells (similar groups)
  • Second part: Poststratification (weighting)
  • The poststratification part is basically the weighting method.
  • To get an estimate for the general popluation, we compute the prediction from our model for the different cells (= proportion of men/women that know brand) and weight this with the proportion of men/women in the general population.
  • Mathematically, the weight corresponds to the probability of gender and the prediction to the probability of knows brand conditioned on gender. Combining them gives the joint distribution pr(knows_brand, gender) according to the chain rule. Adding them up is equivalent to marginalizing out gender.
  • The difference to the approach outlines before, is that the predictions are actually samples from our posterior instead of point estimates. Using a Bayesian model thus allows us to propagate the uncertainty.
  • Back to our use case, on top of questions asking for “do you know this brand?” we also have questions like these.
  • These questions are binarized, thus resulting in one model per option (here 6)
  • The same question is also asked for competitor brands, there are different competitors for different markets which means we easily end up with around 1000 models even though just a handful of questions were asked.
  • Each model takes around 20min, so in total we have 10 days of computing time.

    So many models. So little time.
  • Metaflow Library: A library developed by Netflix https://metaflow.org/
    It integrates nicely with AWS Batch, is easy to use even for Data Scientists that don’t have strong background in cloud computing and additionally also supports reproducibility.
  • This is how MRP could look like in Metaflow.
  • It’s a DAG: Each step represents a job launched on Batch, where the model step is parallelized, so one job per model is launched.
  • The heavy lifting here is done using the “foreach” keyword. If questions is a list then with the foreach, metaflow will start one run_model job for each question. All the orchestration is being taken care of by metaflow.
  • To increase resources, we can add just line
  • Some remaining challenges: how to monitor convergence of more than 1000 models?
    How to add more predictor variables, especially custom variables specific to one brand for which no census data is available

×