![CONTEXT: WHAT’S BIG DATA?
6 TYPES
9
{
"created_at": "Thu Sep 15 16:29:08 +0000 2016",
"id": 776457834095644700,
"id_str": "776457834095644672",
"text": "I love @glip because it makes me more productive and
reliant on far fewer tools! #gliplove #goglip #gliptastic :)",
"truncated": false,
"entities": {
"hashtags": [
{
"text": "gliplove",
"indices": [
82,
91
]
},
{
"text": "goglip",
"indices": [
92,](https://image.slidesharecdn.com/understandingbigdata11-9-16-161110013229/75/Working-With-Big-Data-Nov-2016-9-2048.jpg)
Uploaded bySeth Familian
Working With Big Data - Nov 2016
This document summarizes Seth Familian's presentation on working with big data. The presentation covers the context and definition of big data, useful tools for working with big data like Splunk, building dashboards from big data using techniques like scheduled searches and indices, inferring customer segments from big data using RFM scoring and crossing segments with categories, and final thoughts on data as a new interface and starting points for working with big data. The presentation contains examples and screenshots to illustrate the concepts discussed.
More Related Content
![[AIIM16] How Regulatory Data Can Set the Narrative for an Analytics Opportunity](https://cdn.slidesharecdn.com/ss_thumbnails/regulatorydataanalytics-160502152048-thumbnail.jpg?width=640&height=640&fit=bounds)
![[AIIM16] Don’t Talk to Me about Content Management](https://cdn.slidesharecdn.com/ss_thumbnails/donttalktomeaboutcontentmanagement-160502152951-thumbnail.jpg?width=640&height=640&fit=bounds)
![[AIIM16] Change Management: Lessons Learnt in a Global Engineering and Produ...](https://cdn.slidesharecdn.com/ss_thumbnails/changemanagementlessonslearntglobelengineeringproductioncompany-160502154416-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Working With Big Data - Nov 2016
![[Big] Data For Marketers: Targeting the Right Market](https://cdn.slidesharecdn.com/ss_thumbnails/20171108bigdataformarketerstargetingtherightmarket-171109021834-thumbnail.jpg?width=640&height=640&fit=bounds)
Working With Big Data - Nov 2016
- 1. Seth Familian Founder +Principal, Familian&1 WORKING WITH BIG DATA FOLLOW ALONG! familian1.com/wwbd
- 2. INTRODUCTION SETH FAMILIAN FOUNDER +PRINCIPAL, FAMILIAN&1 2 Corporate Strategy User Experience Design Creative ProcraftinationTeaching + Education Growth Hacking
- 3. AGENDA ‣ Context: What’sbig data? ‣ Useful tools ‣ Building dashboards ‣ Inferring segments ‣ Final thoughts 3
- 4. WORKING WITH BIGDATA CONTEXT: WHAT’S BIG DATA? 4
- 5. CONTEXT: WHAT’S BIGDATA? WELCOME TO DATA OBESITY! 5 http://www.datasciencecentral.com/profiles/blogs/basic-understanding-of-big-data-what-is-this-and-how-it-is-going
- 6. CONTEXT: WHAT’S BIGDATA? HOW BIG IS BIG? 6 http://www.domo.com/blog/2013/05/the-physical-size-of-big-data/ in 1 year! creates enough data to fill
- 7. CONTEXT: WHAT’S BIGDATA? BIG IN GROWTH, TOO. 7 http://www.infosysblogs.com/brandedge/2013/04/20130419Infographc.html https://studentforce.wordpress.com/2013/09/21/umuc-big-data-revolution-is-here/
- 8. CONTEXT: WHAT’S BIGDATA? 9 SOURCES 8 https://studentforce.wordpress.com/2013/09/21/umuc-big-data-revolution-is-here/
- 9. CONTEXT: WHAT’S BIGDATA? 6 TYPES 9 { "created_at": "Thu Sep 15 16:29:08 +0000 2016", "id": 776457834095644700, "id_str": "776457834095644672", "text": "I love @glip because it makes me more productive and reliant on far fewer tools! #gliplove #goglip #gliptastic :)", "truncated": false, "entities": { "hashtags": [ { "text": "gliplove", "indices": [ 82, 91 ] }, { "text": "goglip", "indices": [ 92,
- 10. CONTEXT: WHAT’S BIGDATA? 6 TYPES 10
- 11. CONTEXT: WHAT’S BIGDATA? 6 TYPES 11
- 12. CONTEXT: WHAT’S BIGDATA? THE FOUR V’S 12 http://www.slideshare.net/gschmutz/ukoug2013-big-datafastdata 9 Data Sources 6 Data Types
- 13. CONTEXT: WHAT’S BIGDATA? COHESIVE ASSESSMENT 13 https://datafloq.com/read/understanding-sources-big-data-infographic/338
- 14. WORKING WITH BIGDATA USEFUL TOOLS FOR BIG DATA 14
- 15.
- 16.
- 17.
- 18. USEFUL TOOLS AND LET’S REFRAMEIT 18 EVENT-BASED ANALYTICS +TEXTUAL VISUAL ANALYTICS + INSIGHT PROCESSING + NORMALIZATION DATA TRANSFORMATION (ETL) ACTIVITY MODALITY DATA DISPLAY + DASHBOARDING STATISTICAL ANALYTICS VISUAL ANALYTICS
- 19. USEFUL TOOLS POWER PLAYERS 19 EVENT-BASED ANALYTICS +TEXTUAL VISUAL ANALYTICS+ INSIGHT PROCESSING + NORMALIZATION VISUAL ANALYTICSSTATISTICAL ANALYTICS DATA TRANSFORMATION (ETL) DATA DISPLAY + DASHBOARDING ACTIVITY MODALITY
- 20. USEFUL TOOLS SPLUNK 20 SPLUNK.COM FOR ANYMACHINE DATA
- 21. USEFUL TOOLS SPLUNK 21 SPLUNK.COM FOR ANYMACHINE DATA
- 22. USEFUL TOOLS CHARTED 22 CHARTED.CO FOR SUPERSIMPLE CHARTS
- 23.
- 24.
- 25. USEFUL TOOLS TAGUL 25 TAGUL.COM FOR GORGEOUSWORD CLOUDS
- 26.
- 27. USEFUL TOOLS GOOGLE ANALYTICS 27 ANALYTICS.GOOGLE.COM FORWEBSITE TRAFFIC
- 28.
- 29.
- 30. MIND MELTED YET?LET’S TAKE 15. BREAK TIME! 33 ‣ Stretch your legs ‣ Hydrate or grab a snack ‣ We’ll start again in 15 mins!
- 31. Seth Familian Founder +Principal, Familian&1 WORKING WITH BIG DATA FOLLOW ALONG! familian1.com/wwbd
- 32. WORKING WITH BIGDATA BUILDING DASHBOARDS FROM BIG DATA 35
- 33. BUILDING DASHBOARDS SMALL DATAEXAMPLE 36 via
- 34. Raw “Header” File BUILDING DASHBOARDS BIGDATA VISUALIZATION 37 26Mrows 250Kaffiliate IDs 28sub-channels
- 35. “Long” Data BUILDING DASHBOARDS THE“OLD SCHOOL” APPROACH 38 Raw “Header” File 2 1 Affiliates Lookup File Update Loop Summary Index saved searches scheduled searches TRANSFORM Instantly Generates EXTRACT LOAD Channel Dashboards
- 36. BUILDING DASHBOARDS ANATOMY OFA SPLUNK SEARCH 39
- 37. BUILDING DASHBOARDS SCHEDULED SEARCHES+ INDICES 40 by sourceoforder by af_type by af_source by af_name by af_name + ppc_s SUMMARY INDEX INDEX_MAIN_SOURCES INDEX_A.COM INDEX_AFFILS INDEX_PAID_SEARCH INDEX_SHOPPING_ENGINES 12M+ TRANSACTIONSFULL DB METRICS SAVED SEARCHES DATA DELTAS METRICS DATA DELTAS METRICS DATA DELTAS METRICS DATA DELTAS METRICS DATA DELTAS
- 38. BUILDING DASHBOARDS NESTED CHARTS+ SMALL MULTIPLES 41
- 39.
- 40.
- 41. BUILDING DASHBOARDS AUTOMATED BIGDATA FLOW (EXAMPLE 1) 45 RAW DATA EXTRACT LOADTRANSFORM Traffic Sources & Session Stats RAW DATA Behavioral Segments, Funnels, Retention & LTV EXTRACT Additional aggregation and data refinement Core Website Social Engagement Footprint Unified social footprint metrics Enrichment of email addresses CRM data store for easy segmentation + analysis Additional context on Twitter followers More flexible segments, funnels + retention metrics
- 42. BUILDING DASHBOARDS AUTOMATED BIGDATA FLOW (EXAMPLE 2) 46 LOAD Custom dashboards synced with 70+ APIs Traffic Sources & Session Stats Realtime (RT) TRANSFORMRAW DATA EXTRACT Core Website Social Engagement Footprint heavy-duty query tools already in place App Databases Custom aggregation scripts Postgres or Redshift DB Daily Pull Internally-reported metrics summarized for triangulation Daily CSV Behavioral segmentation + in-app messaging RT Behavioral Segments, Funnels, Retention & LTV RT Unified social footprint metrics Unified app downloads & ratings metrics App Store Activity email address enrichment
- 43. WORKING WITH BIGDATA INFERRING SEGMENTS FROM BIG DATA 47
- 44. INFERRING SEGMENTS FROMBIG DATA Frequency (F) Ranking Recency (R) Ranking Monetary (M) Ranking 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1. all customers are independently ranked into equal-sized “tiles” three times over 48 1,164,927 customers 2. M scores are multiplied by 100 and F scores are multipled by 10 to create unique ranking values 100 200 300 400 500 10 20 30 40 50 1 2 3 4 5 3. MFR scores are added up for each customer to yield 125 unique MFR segments: 111 121 131 141 151 112 122 132 142 152 155 211 221 231 241 251 212 222 232 242 252 255 511 521 531 541 551 512 522 532 542 552 555 Most recent, frequent, and highest-value customers Least recent, frequent, and lowest-value customers RFM SCORING
- 45. INFERRING SEGMENTS FROMBIG DATA 49 WHY QUANTILES? NORMAL DISTRIBUTION SKEWED-DISTRIBUTION
- 46. INFERRING SEGMENTS FROMBIG DATA 50 X11 X21 X31 X41 X12 X22 X32 X42 X13 X23 X33 X43 X14 X24 X34 X44 X15 X25 X35 X45 X51 X52 X53 X54 X55 High Frequency High Recency Low Frequency Low Recency Still Loyal Once Loyal New Old F + R = LOYALTY INSIGHTS
- 47. INFERRING SEGMENTS FROMBIG DATA $0 $1,500 $3,000 $4,500 $6,000 51 Average total spent ($) by new MFR quantiles rerun for non-outlier M1 + M2 customers M1 M2 M3 M4 M5 percent: top 20% of M1+2 2nd 20% 3rd 20% 4th 20% Bottom 20% segment size: 93,134 93,139 92,861 93,406 93,143 avg. $ spent: $3,337 $1,137 $642 $412 $276 total $ spent: $345,234,826 $105,573,528 $59,348,459 $38,398,553 $25,537,936 % of total revs: 53% 32% 18% 11% 8% High-Value Customers Low-Value Customers M = VALUE INSIGHTS
- 48. INFERRING SEGMENTS FROMBIG DATA 52 High Value Customers Low Value Customers Still Loyal Once Loyal New Old M1 M2 M3 M4 M5 212111 121 112 122 113 123 211 221 222 311 321 312 322 411 421 412 422 511 521 512 522 114 124 115 125 213 223 214 224 215 225 313 323 314 324 315 325 413 423 414 424 415 425 513 523 514 524 515 525 131 132 141 142 151 152 231 232 241 242 251 252 331 332 341 342 351 352 431 432 441 442 451 452 531 531 541 542 551 552 133 134 143 144 153 154 135 145 155 233 234 243 244 253 254 235 245 255 333 334 343 344 353 354 335 345 355 433 434 443 444 453 454 435 445 455 533 534 543 544 553 554 535 545 555 1 2 3 4 5 6 7 8 COMBINING INSIGHTS
- 49. INFERRING SEGMENTS FROMBIG DATA PRICE-POINT CUTOFFS 53 Best Camera/Lens Purchased DSLR Body DSLR Lens DSLR Body + Lens Point-and-Shoot Segment Name Relationship to Photography Memory Keepers Use cameras to record family memories and milestones less than $650 less than $300 less than $950 less than $450 Hobbyists Enjoy the picture-taking process; understand and use camera controls $650 - $1725 $300 - $750 $950 - $2300 $450 - $700 Prosumers Advanced skills, but do not make a living from photography $1725 - $2750 $750 - $3000 $2300 - $4200 $700 - $2500 Pros Rely on photography as a profession $2750+ $3000+ $4200+ $2500+
- 50. INFERRING SEGMENTS FROMBIG DATA CROSSING RFM W/ CATEGORIES 54 Low Value High Value Still Loyal Once Loyal New Old Still Loyal Once Loyal New Old Memory Keepers 1 2 3 4 5 6 7 8 Hobbyists 9 10 11 12 13 14 15 16 Prosumers 17 18 19 20 21 22 23 24 Professionals 25 26 27 28 29 30 31 32 3. Cross-Tabulate Top customers and categories to create behavioral and loyalty-based segments 9 key categories account for 81% of sales 2. Isolate the top customers and categories by total dollars spent, frequency, and recency (RFM) measures 465,683 top customers account for 88% of sales 1,164,927 customers 807 categories 1. Aggregate 72 months of Internet channel transaction data, organizing by key variables 2,246,094 Internet Channel transactions 4. Generate Segment-specific marketing recommendations which can be further targeted by brand YIELDS SOLID TARGETS FOR TACTICAL PLANNING
- 51. WORKING WITH BIGDATA FINAL THOUGHTS 57
- 52. FINAL THOUGHTS A NEWTYPE OF KNOWLEDGE WORKER 58 http://www.doclens.com/87922/think-issue-7-2014/
- 53. FINAL THOUGHTS AN INCREDIBLYVALUABLE SKILL 59 https://studentforce.wordpress.com/2013/09/21/umuc-big-data-revolution-is-here/
- 54. FINAL THOUGHTS THE CORNERSTONEOF A DAUNTING FUTURE? 60 https://studentforce.wordpress.com/2013/09/21/umuc-big-data-revolution-is-here/
- 55. FINAL THOUGHTS DATA ASINTERFACE 61 for using Made Visual BACKGROUND TITLES + BUTTONS TEXT + LINES Your data + brand up to 100,000 objects Anywhere on the Web using 1 line of code
- 56. FINAL THOUGHTS DATA ASINTERFACE 62 for using
- 57. FINAL THOUGHTS DATA ASINTERFACE 63
- 58.
- 59.
- 60.
- 61.
- 62.
- 63. DISCUSSION TIME WORKING WITHBIG DATA 69 QUESTIONS · FEEDBACK · IDEAS · INSIGHTS
- 64. THANK YOU KEEP INTOUCH! 70 SETH@FAMILIAN1.COM · @SETHFAM1