Slides from my Strata talk: http://strataconf.com/strata2014/public/schedule/speaker/163953 Abstract: Creating value from big, messy data sets can be a daunting task. The session introduces the Sidekick Pattern: using small, curated data to increase the value of Big Data. Drawing on lessons from data science for Jawbone’s UP fitness tracker, we will see how smart selection of data sidekicks can accelerate analysis, solve cold start problems, and simplify complicated data pipelines.

1. THE SIDEKICK PATTERN:
USING SMALL DATA TO MULTIPLY
THE VALUE OF BIG DATA
@AbeGong
Data Scientist, Jawbone
Strata - February 2014
Wednesday, February 12, 14

2. Wednesday, February 12, 14

3. Wednesday, February 12, 14

4. Wednesday, February 12, 14

5. Wednesday, February 12, 14

6. Wednesday, February 12, 14

7. Wednesday, February 12, 14

8. DATA SIDEKICKS
Wednesday, February 12, 14

9. EX: HIEROGLYPH
TRANSLATION
Wednesday, February 12, 14

10. EX: HIEROGLYPH
TRANSLATION
Wednesday, February 12, 14

11. EX: HIEROGLYPH
TRANSLATION
Wednesday, February 12, 14

12. EX: CAMPAIGN TARGETING
Wednesday, February 12, 14

13. EX: CAMPAIGN TARGETING
Wednesday, February 12, 14

14. EX: CAMPAIGN TARGETING
Wednesday, February 12, 14

15. EX: SLEEP CONTEXT
Wednesday, February 12, 14

16. EX: SLEEP CONTEXT
Wednesday, February 12, 14

17. EX: SLEEP CONTEXT
Wednesday, February 12, 14

18. [DATA ART EXAMPLE]
SUB-TITLE
Wednesday, February 12, 14

19. Wednesday, February 12, 14

20. EXAMPLES, PLEASE:
WHICH DATA STREAMS GET
BIG?
(...AND BESIDES SIZE, WHAT ELSE DO THEY HAVE IN COMMON?)
Wednesday, February 12, 14

21. BIG, RICH, MESSY
Wednesday, February 12, 14

22. BIG, RICH, MESSY CAREFULLY CURATED
Wednesday, February 12, 14

23. TRANSMUTATION!
Wednesday, February 12, 14

24. EX: HUFFPO MODERATION
Wednesday, February 12, 14

25. Wednesday, February 12, 14

26. Wednesday, February 12, 14

27. EX: HUFFPO MODERATION
Wednesday, February 12, 14

28. EX: HUFFPO MODERATION
Wednesday, February 12, 14

29. WHEN SHOULD I USE THE
SIDEKICK PATTERN?
Wednesday, February 12, 14

30. WHEN SHOULD I USE THE
SIDEKICK PATTERN?
• To
Wednesday, February 12, 14
separate munging and cleaning from scaling.

31. WHEN SHOULD I USE THE
SIDEKICK PATTERN?
• To
• To
Wednesday, February 12, 14
separate munging and cleaning from scaling.
bootstrap new data products.

32. WHEN SHOULD I USE THE
SIDEKICK PATTERN?
• To
• To
bootstrap new data products.
• To
Wednesday, February 12, 14
separate munging and cleaning from scaling.
leverage variety against volume.

33. EX: SLEEP RECOVERY
Wednesday, February 12, 14

34. EX: SLEEP RECOVERY
Wednesday, February 12, 14

35. EX: SLEEP RECOVERY
Wednesday, February 12, 14

36. EX: SLEEP RECOVERY
Wednesday, February 12, 14

37. Wednesday, February 12, 14

38. Wednesday, February 12, 14

39. LEVELS OF ABSTRACTION
Wednesday, February 12, 14

40. LEVELS OF ABSTRACTION
Wednesday, February 12, 14

41. LEVELS OF ABSTRACTION
Wednesday, February 12, 14

42. QUESTIONS? COMMENTS?
@AbeGong
Data Scientist, Jawbone
Strata - February 2014
Wednesday, February 12, 14

43. Wednesday, February 12, 14

44. Big
Rich
Messy
Sensory
User experience
External-facing
Abstract
Business logic
Internal-facing
“Qualitative”
Story-making
Wednesday, February 12, 14
Small
Focused
Curated
“Quantitative”
Science-making

45. TRANSMUTATION EXAMPLES
Example
Example
Property
Rosetta stone
Synonyms/Comparability
Bridge cases in IRT
scaling models
Relative ranking
Campaign targeting
Demographic categories
Sentiment analysis
Categories
Sleep context
Context
Pretty much all
supervised learning
Categories/Scales
Instrumental variables
Causality
...
HuffPo moderation
Credibility
Sleep recovery
Clean examples
Economic mobility
Continuity
Crowdflower gold
Wednesday, February 12, 14
Property
Credibility

46. RECOMMENDED READING
•
•
Paco Nathan: http://www.slideshare.net/pacoid/using-cascalog-to-build-an-appbased-on-city-of-palo-alto-open-data
•
Jay Kreps: http://engineering.linkedin.com/distributed-systems/log-what-everysoftware-engineer-should-know-about-real-time-datas-unifying
•
Joseph Turian: http://files.meetup.com/1542972/20120202-more-data-samemodels-STUDY-SLIDES.pdf
•
Wednesday, February 12, 14
Pete Skomoroch: http://www.slideshare.net/pskomoroch/strataendorsements-16939466
Me: http://blog.abegong.com/2014/02/wanted-good-examples-of-datasidekicks.html