Big Data often has demographic and behavioral data, but little to no attitudinal information. This presentation describes the many challenges in using survey data to add attitudinal information. Find out more https://goo.gl/qRg6IK

1. Jeroen Hardon
Bringing big data to life

2. 1.3
Exabytes
2.9
Million
Per second
375
Megabytes
Per day
24
Petabytes
Per day
50
Million
Per day
700
Billion
Minutes
per month
73
Items
Per second
Big data is everywhere
20
Hours
Per minute

3. A journey in segmentation with
data scientists and big data.

4. What was the
problem?
What was the
solution?
How well did
it work?

5. Needs-based
segmentation
7 segments
created
Classifier
tool build,
using 10
questions
Original segmentation study

6. This resulted in a
happy client.

7. “Let’s tag a segment to
each person in our
database of 40 million“

8. 12.000 people
from the database
answered the
classifier
questions
Those 12.000
were classified
in 1 of the 7
segments
Attitudinal
segments not
explained by
demographics
Attitudes ≠ Demographics

9. Revised
segments should
align better with
big data
Must predict
original
segments in
segmentation
study
Merging the
2 types of
data
New classification tool

10. The database
and survey
demographics
did not match
We build classifiers
by matching survey
data to resemble
the database
We generated many
samples of our
survey data and
built an ensemble
of classifiers
Ensembles

11. While building ensembles of
classifiers helped, it was still
inadequate.
We needed to strengthen the demographic / behavioral signal

12. Expectation Maximization
?

13. Expectation Maximization
5

14. Expectation Maximization
How do I
"assign" each of
the individual
fruits to a tree
type?
What are the
characteristics of
the fruit of each
tree type?

15. Expectation Maximization

16. Expectation Maximization

17. Expectation Maximization
Observed Data
Initial segmentation data
6500 respondents
Augment of 12000
from Big Data
Known
fixed
segment
Unknown
segment
+ Model 1

18. Expectation Maximization
Observed Data
Initial segmentation data
6500 respondents
Augment of 12000
from Big Data
Known
fixed
segment
Unknown
segment
+
Big data
variables
Model 2

19. We got classifiers that were slightly
less optimal in predicting survey
data, but much more aligned with
the big data.
We made sure to not let the predictive accuracy drop below 70%
(originally 80%)

20. How well did it work?
Seg 1 Seg 2 Seg 3 Seg 4 Seg 5 Seg 6 Seg 7
Seg 1 564 84 15 56 36 14 18
Seg 2 68 844 84 13 7 13 10
Seg 3 33 72 561 2 3 1 5
Seg 4 34 8 0 567 5 81 29
Seg 5 27 12 1 6 635 50 57
Seg 6 21 27 6 76 43 873 30
Seg 7 18 28 9 50 59 52 1193
Initialclassifier
segment
Revised classifier segment
Only 19% changed
Data Source: Survey Data of 6500

21. How well did it work?
Seg 1 Seg 2 Seg 3 Seg 4 Seg 5 Seg 6 Seg 7
Seg 1 135 102 18 66 207 157 45
Seg 2 119 545 171 58 174 203 101
Seg 3 55 113 316 44 240 219 72
Seg 4 90 67 4 283 233 287 69
Seg 5 303 169 41 216 1994 925 205
Seg 6 325 259 36 261 646 1591 127
Seg 7 52 26 3 90 193 191 156
Initialclassifier
segment
Revised classifier segment
Over 58% changed
Data Source: Augment of 12000

22. Conclusions
Big data cannot
predict
everything
No need to be
scared of big data.
Surveys and big
data can coexist
Expectation
maximization
provides a
framework for
joint modeling

23. So what?

24. So what?

25. So what?

26. So what?

27. Jeroen Hardon
Director Methodology and Innovation EU
Based in Rotterdam
j.hardon@skimgroup.com
Contact us
skimgroup.com
@SKIMgroup
SKIMgroup
SKIMgroup
27
Kevin Lattery
VP Methodology & Innovation
Based in New York
k.lattery@skimgroup.com