Big data Overview

Arnon Rotem-Gal-Oz
Arnon Rotem-Gal-OzDistinguished Engineer at Monday.com
BIG DATA
Arnon Rotem-Gal-Oz
Director of Technology Research, Amdocs
The blind men and the elephant. Poem by John Godfrey Saxe (Cartoon originally copyrighted by the authors; G.
Renee Guzlas, artists http://www.nature.com/ki/journal/v62/n5/fig_tab/4493262f1.html
1880 US
Census
Big data Overview
Hollerith
Tabulating
Machine
Hollerith photos by Martin Wichary :
http://www.flickr.com/photos/mwichary/4358926764/in/photostream/
Big data Overview
ource: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/
Big data happens when the data you
have to process is bigger than what
you can process in the given time with
current technologies
Myth: Big data = keep all data
Source: Big Data Public Private Forum : http://www.big-
project.eu/sites/default/files/D2.2.1_First%20draft%20of%20Technical%20white%20papers_FINAL_v1.01_
0.pdf
Source: Big Data Public Private Forum : http://www.big-
project.eu/sites/default/files/D2.2.1_First%20draft%20of%20Technical%20white%20papers_FINAL_v1.01_
0.pdf
Some Telco
Numbers
Source: Wikipedia
http://upload.wikimedia.org/wikipedia/commons/5/50/Telephone_operators,_1952.jpg
So, what do we do
with all this data?
Wikipedia http://upload.wikimedia.org/wikipedia/commons/0/06/UPS_Truck.jpg
It’s the insights, stupid*
* With apologies to Bill Clinton
ource: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/
Big data analytics is when sample = N
• Big data happens when the data you have to process
is bigger than what you can process in the given time
with current technologies
Big data Overview
“My daughter got this in the
mail!, She’s still in high school,
and you’re sending her coupons
for baby clothes and cribs? Are
you trying to encourage her to
get pregnant?”
Source: Forbes http://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-
father-did/
We need to
watch out that
Analytics won’t
get too creepy
When people hear
big data they think
fast data
Source: Steve Jones Cap Gemini
http://www.no.capgemini.com/node/778541
Subscribers
Collect
& Filter
Correlate
(simplified) Network proactive care flow
Account
Event Store
Identify &
Predict
Network
Failures
Reimburse
VIPs
Prioritize
technicians
Identify
impact on
high valued
Accounts
ource: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/
Big data is when we can handle data
fast enough to make a difference
• Big data happens when the data you have to process
is bigger than what you can process in the given time
with current technologies
• Big data analytics is when sample = N
Technology space
The Elephant in the room
Hadoop Stack
Map/R
educe
HDFS
HBase
Pig
Hive
Zoo
Keeper
Oozie Mahout
Giraph
Schema on read
Move data to computation
Maybe we should rethink
moving data to
computation…
Source : http://my-inner-voice.blogspot.co.il/2012/06/haddop-101-paper-by-miha-ahronovitz-and.html
Big data Overview
Map/reduce
Source: http://www.bodhtree.com/blog/2012/10/18/ever-wondered-what-happens-between-map-and-reduce/
Customer Segmentation
First
name
Last
name
ARPU Age Device Country …
Mr. Smith 100 22 iPhone 5s,White USA
John Doe 87 42 Samsung Galaxy S5,Gold France
Lady In Red 105 21 Samsung Note 3, White UK
…
Uluru, Australia by Stuart Edwards (cc) http://en.wikipedia.org/wiki/Uluru#mediaviewer/File:Uluru_Panorama.jpg
K-Means
ARPU
Age
Source : http://pypr.sourceforge.net/kmeans.html
K=3ARPU
Age
ARPU
Age
Source : http://pypr.sourceforge.net/kmeans.html
New paradigms
Map/R
educe
HDFS
HBase
Pig
Hive
Zoo
Keeper
Oozie Mahout
Giraph
New Paradigms
Map/R
educe
HDFS
HBase
Pig Hive
Zoo
Keeper
Oozie Mahout
YARN
Giraph
New Paradigms
Map/R
educe
HDFS
HBase
Pig Hive
Zoo
Keeper
Oozie Mahout
YARN
Giraph
Spark
Storm
Slider
Flink
Impala
Tez
Presto
Amdocs Analytics & Data Management
Heritage
2013
• Proactive Care
• TerraScale
• Network optimization
• Real time
analytics platform
• Single product catalog
• BSS–OSS
Integration
• CRM-Billing
Integration
OSS
Analytics Platform,
16 Analytics Patents
• aLDM logical data
model
• Policy control
Network Analytics
CRM
2000 2008
AcquisitionsPortfolio
34
Information Security Level 2 – Sensitive
© 2014 – Proprietary and Confidential Information of Amdocs
Touchpoints & Applications
CRM Self Service E-MailPCRF SMS OtherWi-Fi OffloadCampaign Mng. • • • • • • •
Operational
Envelope &
Platform
Administration
• Security
Management
• Configuration
Management
• Services
Inventory
• Performance
Management
• Fault
Management
• LoggerCollect &
Ingest
Transform
& Enrich
Aggregate
& Correlate
Drive
Insight
Close the
Loop
Machine
Learn &
Score
Application-Ready Data and Analytics/ML Insights
Entities and Profiles
Detailed Data
OSS
Probes SocialRAN Inventory Usage &
Charging
CRM
Real-Time & Batch Connectors
Insight Platform
Marketing
Analytical
Application
Framework:
Dashboards &
Visualisation
Decisioning
Engine
Dynamic Micro
Segmentation
Network Care Operations
ource: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/
• Big data happens when the data you have to process
is bigger than what you can process in the given time
with current technologies
• Big data analytics is when sample = N
• Big data is when we can handle data fast enough to
make a difference
Additional takeaways
• CSPs have always been in the big data
business – they just didn’t know it
• Big data is not a panacea
• Hadoop is shaping up as the big data OS
– Though there are alternatives arriving from the
cloud arena (mesos, kubernetes)
What we
covered here
is not even
the tip of the
iceberg
Source: wikimedia http://commons.wikimedia.org/wiki/File:Iceberg.jpg
Arnon Rotem-Gal-Oz
Director of Technology Research, Amdocs
arnonrot@amdocs.com / arnon@rgoarchitects.com
1 of 38

Recommended

Going Digital by
Going DigitalGoing Digital
Going DigitalRoderic Page
12.3K views159 slides
"What Have You Done Tomorrow" @ HR Vision Amsterdam 2015 by
"What Have You Done Tomorrow" @ HR Vision Amsterdam 2015"What Have You Done Tomorrow" @ HR Vision Amsterdam 2015
"What Have You Done Tomorrow" @ HR Vision Amsterdam 2015Volker Hirsch
19.4K views57 slides
Aliens by
AliensAliens
Aliensmenke13
523 views11 slides
AI & The Future of Work - Work & Life in the Age of Robots by
AI & The Future of Work - Work & Life in the Age of RobotsAI & The Future of Work - Work & Life in the Age of Robots
AI & The Future of Work - Work & Life in the Age of RobotsVolker Hirsch
4.1K views80 slides
Big Data by
Big DataBig Data
Big DataNGDATA
67.4K views61 slides
Big Data Decision-Making by
Big Data Decision-MakingBig Data Decision-Making
Big Data Decision-MakingTeradata Aster
3.9K views35 slides

More Related Content

Viewers also liked

introduction to data processing using Hadoop and Pig by
introduction to data processing using Hadoop and Pigintroduction to data processing using Hadoop and Pig
introduction to data processing using Hadoop and PigRicardo Varela
92.5K views32 slides
Pig, Making Hadoop Easy by
Pig, Making Hadoop EasyPig, Making Hadoop Easy
Pig, Making Hadoop EasyNick Dimiduk
84.7K views16 slides
Hive Quick Start Tutorial by
Hive Quick Start TutorialHive Quick Start Tutorial
Hive Quick Start TutorialCarl Steinbach
139.8K views36 slides
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop by
Facebooks Petabyte Scale Data Warehouse using Hive and HadoopFacebooks Petabyte Scale Data Warehouse using Hive and Hadoop
Facebooks Petabyte Scale Data Warehouse using Hive and Hadooproyans
82.7K views40 slides
Hadoop, Pig, and Twitter (NoSQL East 2009) by
Hadoop, Pig, and Twitter (NoSQL East 2009)Hadoop, Pig, and Twitter (NoSQL East 2009)
Hadoop, Pig, and Twitter (NoSQL East 2009)Kevin Weil
143.2K views58 slides
What is big data? by
What is big data?What is big data?
What is big data?David Wellman
184.4K views54 slides

Viewers also liked(15)

introduction to data processing using Hadoop and Pig by Ricardo Varela
introduction to data processing using Hadoop and Pigintroduction to data processing using Hadoop and Pig
introduction to data processing using Hadoop and Pig
Ricardo Varela92.5K views
Pig, Making Hadoop Easy by Nick Dimiduk
Pig, Making Hadoop EasyPig, Making Hadoop Easy
Pig, Making Hadoop Easy
Nick Dimiduk84.7K views
Hive Quick Start Tutorial by Carl Steinbach
Hive Quick Start TutorialHive Quick Start Tutorial
Hive Quick Start Tutorial
Carl Steinbach139.8K views
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop by royans
Facebooks Petabyte Scale Data Warehouse using Hive and HadoopFacebooks Petabyte Scale Data Warehouse using Hive and Hadoop
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
royans82.7K views
Hadoop, Pig, and Twitter (NoSQL East 2009) by Kevin Weil
Hadoop, Pig, and Twitter (NoSQL East 2009)Hadoop, Pig, and Twitter (NoSQL East 2009)
Hadoop, Pig, and Twitter (NoSQL East 2009)
Kevin Weil143.2K views
What is big data? by David Wellman
What is big data?What is big data?
What is big data?
David Wellman184.4K views
Introduction To Map Reduce by rantav
Introduction To Map ReduceIntroduction To Map Reduce
Introduction To Map Reduce
rantav106.2K views
HIVE: Data Warehousing & Analytics on Hadoop by Zheng Shao
HIVE: Data Warehousing & Analytics on HadoopHIVE: Data Warehousing & Analytics on Hadoop
HIVE: Data Warehousing & Analytics on Hadoop
Zheng Shao111.2K views
What is Big Data? by Bernard Marr
What is Big Data?What is Big Data?
What is Big Data?
Bernard Marr585.3K views
Big Data Analytics with Hadoop by Philippe Julio
Big Data Analytics with HadoopBig Data Analytics with Hadoop
Big Data Analytics with Hadoop
Philippe Julio441.9K views
Big Data - 25 Amazing Facts Everyone Should Know by Bernard Marr
Big Data - 25 Amazing Facts Everyone Should KnowBig Data - 25 Amazing Facts Everyone Should Know
Big Data - 25 Amazing Facts Everyone Should Know
Bernard Marr487K views

Similar to Big data Overview

Semantics, Deep Learning, and the Transformation of Business by
Semantics, Deep Learning, and the Transformation of BusinessSemantics, Deep Learning, and the Transformation of Business
Semantics, Deep Learning, and the Transformation of BusinessSteve Omohundro
1.9K views44 slides
Strata Conference NYC 2013 Full Version by
Strata Conference NYC 2013 Full VersionStrata Conference NYC 2013 Full Version
Strata Conference NYC 2013 Full VersionTaewook Eom
1.1K views89 slides
Minnesota ARLD Day 2011 by
Minnesota ARLD Day 2011Minnesota ARLD Day 2011
Minnesota ARLD Day 2011Jason Griffey
3.3K views93 slides
The Edge Group Quito Lima - july 2014 by
The Edge Group   Quito Lima - july 2014The Edge Group   Quito Lima - july 2014
The Edge Group Quito Lima - july 2014Jose A Torres
2.7K views251 slides
10fpresentation(bsj)final by
10fpresentation(bsj)final10fpresentation(bsj)final
10fpresentation(bsj)finalJohn Jung
17.1K views26 slides

Similar to Big data Overview(20)

Semantics, Deep Learning, and the Transformation of Business by Steve Omohundro
Semantics, Deep Learning, and the Transformation of BusinessSemantics, Deep Learning, and the Transformation of Business
Semantics, Deep Learning, and the Transformation of Business
Steve Omohundro1.9K views
Strata Conference NYC 2013 Full Version by Taewook Eom
Strata Conference NYC 2013 Full VersionStrata Conference NYC 2013 Full Version
Strata Conference NYC 2013 Full Version
Taewook Eom1.1K views
Minnesota ARLD Day 2011 by Jason Griffey
Minnesota ARLD Day 2011Minnesota ARLD Day 2011
Minnesota ARLD Day 2011
Jason Griffey3.3K views
The Edge Group Quito Lima - july 2014 by Jose A Torres
The Edge Group   Quito Lima - july 2014The Edge Group   Quito Lima - july 2014
The Edge Group Quito Lima - july 2014
Jose A Torres2.7K views
10fpresentation(bsj)final by John Jung
10fpresentation(bsj)final10fpresentation(bsj)final
10fpresentation(bsj)final
John Jung17.1K views
Safe use of cloud - alternative cloud by Tomppa Järvinen
Safe use of cloud - alternative cloudSafe use of cloud - alternative cloud
Safe use of cloud - alternative cloud
Tomppa Järvinen1.1K views
Evolution of AI - Why is my computer still so dumb? by Olivia Klose
Evolution of AI - Why is my computer still so dumb?Evolution of AI - Why is my computer still so dumb?
Evolution of AI - Why is my computer still so dumb?
Olivia Klose618 views
Open Data Utopia? (SciCAR 19) by Paul Bradshaw
Open Data Utopia? (SciCAR 19)Open Data Utopia? (SciCAR 19)
Open Data Utopia? (SciCAR 19)
Paul Bradshaw464 views
Georgia library association 2011 by Jason Griffey
Georgia library association 2011Georgia library association 2011
Georgia library association 2011
Jason Griffey3.1K views
OakX:Data+The Power of Visual Storytelling Anca Mosoiu/Techliminal by Oak X
OakX:Data+The Power of Visual Storytelling  Anca Mosoiu/TechliminalOakX:Data+The Power of Visual Storytelling  Anca Mosoiu/Techliminal
OakX:Data+The Power of Visual Storytelling Anca Mosoiu/Techliminal
Oak X477 views
Into the next dimension by Ed Charbeneau
Into the next dimensionInto the next dimension
Into the next dimension
Ed Charbeneau1.2K views
Towards Knowledge Graph based Representation, Augmentation and Exploration of... by Sören Auer
Towards Knowledge Graph based Representation, Augmentation and Exploration of...Towards Knowledge Graph based Representation, Augmentation and Exploration of...
Towards Knowledge Graph based Representation, Augmentation and Exploration of...
Sören Auer1.5K views
CONFidence 2014: Davi Ottenheimer Protecting big data at scale by PROIDEA
CONFidence 2014: Davi Ottenheimer Protecting big data at scaleCONFidence 2014: Davi Ottenheimer Protecting big data at scale
CONFidence 2014: Davi Ottenheimer Protecting big data at scale
PROIDEA522 views

More from Arnon Rotem-Gal-Oz

Taking ML to production - a journey by
Taking ML to production - a journeyTaking ML to production - a journey
Taking ML to production - a journeyArnon Rotem-Gal-Oz
192 views20 slides
Apache spark by
Apache sparkApache spark
Apache sparkArnon Rotem-Gal-Oz
1.2K views29 slides
Fallacies of Distributed Computing by
Fallacies of Distributed Computing Fallacies of Distributed Computing
Fallacies of Distributed Computing Arnon Rotem-Gal-Oz
772 views33 slides
Docker & Kubernetes intro by
Docker & Kubernetes introDocker & Kubernetes intro
Docker & Kubernetes introArnon Rotem-Gal-Oz
7.1K views28 slides
Docker Intro by
Docker IntroDocker Intro
Docker IntroArnon Rotem-Gal-Oz
664 views15 slides
Data security @ the personal level by
Data security @ the personal levelData security @ the personal level
Data security @ the personal levelArnon Rotem-Gal-Oz
493 views24 slides

More from Arnon Rotem-Gal-Oz(20)

Microservices - it's déjà vu all over again by Arnon Rotem-Gal-Oz
Microservices  - it's déjà vu all over againMicroservices  - it's déjà vu all over again
Microservices - it's déjà vu all over again
Arnon Rotem-Gal-Oz3.7K views
Big data in the cloud - welcome to cost oriented design by Arnon Rotem-Gal-Oz
Big data in the cloud - welcome to cost oriented designBig data in the cloud - welcome to cost oriented design
Big data in the cloud - welcome to cost oriented design
Arnon Rotem-Gal-Oz3.3K views
Distilling Insights @ Appsflyer (Data Architecture) by Arnon Rotem-Gal-Oz
Distilling Insights @ Appsflyer (Data Architecture)Distilling Insights @ Appsflyer (Data Architecture)
Distilling Insights @ Appsflyer (Data Architecture)
Arnon Rotem-Gal-Oz1.9K views
Building reliable systems from unreliable components by Arnon Rotem-Gal-Oz
Building reliable systems from unreliable componentsBuilding reliable systems from unreliable components
Building reliable systems from unreliable components
Arnon Rotem-Gal-Oz6.7K views
Things to think about while architecting azure solutions by Arnon Rotem-Gal-Oz
Things to think about while architecting azure solutionsThings to think about while architecting azure solutions
Things to think about while architecting azure solutions
Arnon Rotem-Gal-Oz1.3K views

Recently uploaded

2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptxanimuscrm
13 views19 slides
DSD-INT 2023 Dam break simulation in Derna (Libya) using HydroMT_SFINCS - Prida by
DSD-INT 2023 Dam break simulation in Derna (Libya) using HydroMT_SFINCS - PridaDSD-INT 2023 Dam break simulation in Derna (Libya) using HydroMT_SFINCS - Prida
DSD-INT 2023 Dam break simulation in Derna (Libya) using HydroMT_SFINCS - PridaDeltares
18 views9 slides
A first look at MariaDB 11.x features and ideas on how to use them by
A first look at MariaDB 11.x features and ideas on how to use themA first look at MariaDB 11.x features and ideas on how to use them
A first look at MariaDB 11.x features and ideas on how to use themFederico Razzoli
45 views36 slides
Unleash The Monkeys by
Unleash The MonkeysUnleash The Monkeys
Unleash The MonkeysJacob Duijzer
7 views28 slides
Citi TechTalk Session 2: Kafka Deep Dive by
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Diveconfluent
17 views60 slides
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker by
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - ParkerDSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - ParkerDeltares
9 views16 slides

Recently uploaded(20)

2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by animuscrm
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
animuscrm13 views
DSD-INT 2023 Dam break simulation in Derna (Libya) using HydroMT_SFINCS - Prida by Deltares
DSD-INT 2023 Dam break simulation in Derna (Libya) using HydroMT_SFINCS - PridaDSD-INT 2023 Dam break simulation in Derna (Libya) using HydroMT_SFINCS - Prida
DSD-INT 2023 Dam break simulation in Derna (Libya) using HydroMT_SFINCS - Prida
Deltares18 views
A first look at MariaDB 11.x features and ideas on how to use them by Federico Razzoli
A first look at MariaDB 11.x features and ideas on how to use themA first look at MariaDB 11.x features and ideas on how to use them
A first look at MariaDB 11.x features and ideas on how to use them
Federico Razzoli45 views
Citi TechTalk Session 2: Kafka Deep Dive by confluent
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
confluent17 views
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker by Deltares
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - ParkerDSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker
Deltares9 views
DSD-INT 2023 FloodAdapt - A decision-support tool for compound flood risk mit... by Deltares
DSD-INT 2023 FloodAdapt - A decision-support tool for compound flood risk mit...DSD-INT 2023 FloodAdapt - A decision-support tool for compound flood risk mit...
DSD-INT 2023 FloodAdapt - A decision-support tool for compound flood risk mit...
Deltares13 views
Copilot Prompting Toolkit_All Resources.pdf by Riccardo Zamana
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdf
Riccardo Zamana6 views
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -... by Deltares
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...
DSD-INT 2023 Simulating a falling apron in Delft3D 4 - Engineering Practice -...
Deltares6 views
Roadmap y Novedades de producto by Neo4j
Roadmap y Novedades de productoRoadmap y Novedades de producto
Roadmap y Novedades de producto
Neo4j50 views
DSD-INT 2023 Delft3D FM Suite 2024.01 1D2D - Beta testing programme - Geertsema by Deltares
DSD-INT 2023 Delft3D FM Suite 2024.01 1D2D - Beta testing programme - GeertsemaDSD-INT 2023 Delft3D FM Suite 2024.01 1D2D - Beta testing programme - Geertsema
DSD-INT 2023 Delft3D FM Suite 2024.01 1D2D - Beta testing programme - Geertsema
Deltares17 views
Fleet Management Software in India by Fleetable
Fleet Management Software in India Fleet Management Software in India
Fleet Management Software in India
Fleetable11 views
MariaDB stored procedures and why they should be improved by Federico Razzoli
MariaDB stored procedures and why they should be improvedMariaDB stored procedures and why they should be improved
MariaDB stored procedures and why they should be improved
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge... by Deltares
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
Deltares16 views
What Can Employee Monitoring Software Do?​ by wAnywhere
What Can Employee Monitoring Software Do?​What Can Employee Monitoring Software Do?​
What Can Employee Monitoring Software Do?​
wAnywhere21 views
Neo4j y GenAI by Neo4j
Neo4j y GenAI Neo4j y GenAI
Neo4j y GenAI
Neo4j42 views

Big data Overview

Editor's Notes

  1. Poem by John Godfrey Saxe It was six men of Indostan To learning much inclined, Who went to see the Elephant (Though all of them were blind), That each by observation Might satisfy his mind.
  2. 50 Mil. People 7+ years of manual summations read a blog post by Gil Press that stated that the first big data problem was in the 1880s (yes you read that right). In the late 1800s the processing of the US census was beginning to take so long that it was getting close to 10 years. Crossing this mark is meaningful as the census runs every 10 years and as birth rates are getting higher the outlook wasn’t very good. In 1886 Herman Hollerith started a business (that year later was merged with other companies to form IBM) to sell a tabulating machine that holds census data on punch cards. Indeed the 1890 census took less than 2 years to complete and handled both larger population (62 million people) and more data points than the 1880 census.
  3. https://www.census.gov/history/www/census_then_now/notable_alumni/herman_hollerith.html << year instead of almost 10 years 62 Million people 1890 census
  4. https://www.census.gov/history/www/census_then_now/notable_alumni/herman_hollerith.html << year instead of almost 10 years 62 Million people 1890 census
  5. Large Telco – 200M subscribers Orders data few GB Charge Events – 100TB per month Network 800TB - day
  6. So we pile up all this data – but what are we piling it for? 1992 Bill Clinton campaign – It’s the economy, stupid http://upload.wikimedia.org/wikipedia/commons/0/06/UPS_Truck.jpgz
  7. Now get this: In 2007 alone, this helped us: * shave nearly 30 million miles off already streamlined delivery routes. * save 3 million gallons of gas, and * reduce CO2 emissions by 32,000 metric tons¿the equivalent of removing 5,300 passenger cars from the road for an entire year.
  8. Now get this: In 2007 alone, this helped us: * shave nearly 30 million miles off already streamlined delivery routes. * save 3 million gallons of gas, and * reduce CO2 emissions by 32,000 metric tons¿the equivalent of removing 5,300 passenger cars from the road for an entire year.
  9. Retail are the leaders in using analytics Amazon is famous for that but they are not alone
  10. hat Target discovered fairly quickly is that it creeped people out that the company knew about their pregnancies in advance. “If we send someone a catalog and say, ‘Congratulations on your first child!’ and they’ve never told us they’re pregnant, that’s going to make some people uncomfortable,” Pole told me. “We are very conservative about compliance with all privacy laws. But even if you’re following the law, you can do things where people get queasy.” Bold is mine. That’s a quote for our times. So Target got sneakier about sending the coupons. The company can create personalized booklets; instead of sending people with high pregnancy scores books o’ coupons solely for diapers, rattles, strollers, and the “Go the F*** to Sleep” book, they more subtly spread them about: “Then we started mixing in all these ads for things we knew pregnant women would never buy, so the baby ads looked random. We’d put an ad for a lawn mower next to diapers. We’d put a coupon for wineglasses next to infant clothes. That way, it looked like all the products were chosen by chance. “And we found out that as long as a pregnant woman thinks she hasn’t been spied on, she’ll use the coupons. She just assumes that everyone else on her block got the same mailer for diapers and cribs. As long as we don’t spook her, it works.”
  11. hat Target discovered fairly quickly is that it creeped people out that the company knew about their pregnancies in advance. “If we send someone a catalog and say, ‘Congratulations on your first child!’ and they’ve never told us they’re pregnant, that’s going to make some people uncomfortable,” Pole told me. “We are very conservative about compliance with all privacy laws. But even if you’re following the law, you can do things where people get queasy.” Bold is mine. That’s a quote for our times. So Target got sneakier about sending the coupons. The company can create personalized booklets; instead of sending people with high pregnancy scores books o’ coupons solely for diapers, rattles, strollers, and the “Go the F*** to Sleep” book, they more subtly spread them about: “Then we started mixing in all these ads for things we knew pregnant women would never buy, so the baby ads looked random. We’d put an ad for a lawn mower next to diapers. We’d put a coupon for wineglasses next to infant clothes. That way, it looked like all the products were chosen by chance. “And we found out that as long as a pregnant woman thinks she hasn’t been spied on, she’ll use the coupons. She just assumes that everyone else on her block got the same mailer for diapers and cribs. As long as we don’t spook her, it works.” http://www.geektime.co.il/okcupid-experiments-on-users/ <-Facebook, okCupid
  12. Data from Actix (or other network sources) –20/30M subscribers would generate ~ 250K messages per second Monitor for anomalies like dropped calls Correlate with data from CRM (identify customer, account) Analyze for impact on VIPs Analyze for problems in the netwrok Automated action Change SLAs Notify customers (sorry note, small freebie etc) <1 -5 seconds away from the problem <-can have real time impact on satisfaction (should avoid falling into the creepiness problem mentioned with Target use case (we know what you’re doing!!)
  13. Fraud analysis at big telco – where insights arrive ong after the fraud ended Multiple connections with same IP from different locations Buying unlimited data and letting “reselling” it for Skype etc.
  14. Think of it as defining a view on a table but the underlying data can be Poly structured and unstructured data
  15. CRM data Map – identify subscriber, account Group by (account) Reduce update account profile
  16. Average revenue per user - ARPU
  17. SQL on Hadoop Streaming “Enterprise Grade”
  18. SQL on Hadoop Streaming “Enterprise Grade”
  19. Fraud analysis at big telco – where insights arrive ong after the fraud ended Multiple connections with same IP from different locations Buying unlimited data and letting “reselling” it for Skype etc.
  20. Volume Velocity (variety, ver