© 2014 IBM Corporation
Big Data & Analytics – beyond Hadoop
Ian Radmore, IBM UKI Big Data Specialist
June 18th, 2014
© 2014 IBM Corporation
Data: To have and to hold? Or to Analyse and Act!
Data in
Data at
2
© 2014 IBM Corporation
The auto industry is already the 2nd largest data generator AND 20% CAGR!
Ford Fusion: 145 actuators, 4700 relays
and 70 sensors, including radar, sonar,
accelerometer, camera, rain sensors.
Collectively, these devices generate
more than 25 gigabytes of data per
hour, which is processed by more than
70 on-board computers.
1 car year = 1TB
3
© 2014 IBM Corporation
A Big Data & Analytics approach helps provide a foundation for a Smarter Enterprise
Invest in aInvest in a
big data & analyticsbig data & analytics
platformplatform
Be proactive aboutBe proactive about
privacy, security andprivacy, security and
governancegovernance
Imagine It. Realise It. Trust It.
Build a culture thatBuild a culture that
infuses analyticsinfuses analytics
everywhereeverywhere
Confidence in Your
Data
Confidence in
Accelerating Value
Confidence in Your
Skills
4
© 2014 IBM Corporation
Deployed real-time CDR analysis solution to handle exploding data
volume growth and performance requirements
Analyzes call, internet usage, and text records in real-time to identify
and address poorly performing cells
Uses InfoSphere Streams and IBM Netezza
Significant Benefits:
Over 90% reduction in time to merge/load call record data
Over 90% reduction in storage
Increased network quality, improved customer satisfaction,
reduced churn
Sprint Increases Revenue & Improves
Customer Satisfaction
“Over 90+% reduction in
merge/load times and
storage requirements”
“Over 90+% reduction in
merge/load times and
storage requirements”
Capabilities Utilised:
• Stream processing
• Data Warehouse Analytics Appliance
5
© 2014 IBM Corporation
• Examines trends, volume, and content of millions of public Twitter
messages in real-time
• Analytic accelerators to understand sentiment (positive, negative,
neutral)
• Capabilities
• Stream Computing
• Visualization
• Benefits
• Real-time display of public sentiment as candidates respond
to questions
• Debate winner prediction based on public opinion instead of
solely political analysts
University of Southern California
Innovation Lab Monitors Political
Debates
Solution to measure public sentiment during key
primary & general presidential debates
6
© 2014 IBM Corporation
7
KTH Swedish Royal Institute of
Technology Reducing Traffic
Congestion
• Deployed real-time Smarter Traffic system to predict and
improve traffic flow.
• Analyzes streaming real-time data gathered from
cameras at entry/exit to city, GPS data from taxis and
trucks, and weather information.
• Predicts best time and method to travel such as when to
leave to catch a flight at the airport
Significant benefits:
• Enables ability to analyze and predict traffic faster and
more accurately than ever before
• Provides new insight into mechanisms that affect a
complex traffic system
• Smarter, more efficient, and more environmentally
friendly traffic
7
Capabilities Utilised:
Stream Computing
7
© 2014 IBM Corporation
Pacific Northwest Smart Grid
Demonstration Project
Capabilities:
Stream Computing – real-time control
system
Data Warehouse Appliance – analyze
massive data sets
Demonstrates scalability from 100 to
500K homes while retaining 10 years’
historical data
60k metered customers in 5 states
Accommodates ad hoc analysis of price
fluctuation, energy consumption profiles,
risk, fraud detection, grid health, etc.
8
© 2014 IBM Corporation
Information Integration & Governance
Systems Security
On premise, Cloud, As a service
Storage
IBM Watson Foundations
IBM Big Data & Analytics Infrastructure
New /
Enhanced
ApplicationsAll Data
What action
should I take?
Decision
management
Cognitive
What did
I learn?
Landing,
Exploration and
Archive data
zone
EDW and
data mart
zone
Operational
data zone
Real-time Data Processing & Analytics What is
happening?
Discovery and
exploration
Why did it
happen?
Reporting,
content and
analysisWhat could
happen?
Predictive
analytics and
modelling
Deep
Analytics
data zone
8
Realise It. Invest
© 2014 IBM Corporation
Realise It. In-Store Presence Zones
Intelligent location-based technology to gain deep insight
into customer in-store behaviour
Enable retailers to integrate the physical and digital experience to facilitate an ongoing
dialogue that creates loyalty via an exceptional in-store shopping experience
Presence Zones
Sensors
9
© 2014 IBM Corporation
IBM Internal Use Only
Realise It. The Customer Insight Appliance
10
© 2014 IBM Corporation
Realise It. A Multichannel Korean retailer
Reliable insight
provides decision support for senior
management
Targeted campaigns
can be developed for marketing
Precise measurement
of cross-channel campaigns
Business Challenge: As sales increased for this retailer’s online shopping mall,
management experienced increasing difficulty ensuring that an appropriate product mix
was being presented to its customers.
The Solution: The company adopted sophisticated analytics and marketing automation
to understand, predict and act on consumer buying behavior with confidence. Real-time
marketing automation delivers personalised content to each shopper, triggered by their
interaction history. Delivered at the right place and time, these offers can move the
shopper toward a sale and even increase the size of the purchase.
“We have greatly improved our understanding of our customers, which is helping us to
make smarter decisions that significantly improve business performance.”
—Spokesperson, multichannel Korean retailer
Combining marketing
automation with analytics
to personalise
communications and
optimise offerings
11
© 2014 IBM Corporation
Millions of
events per
second
Microsecond
Latency
Traditional / Non-traditional
data sources
Real time delivery
Powerful
Analytics
Algorithmic
Trading
Telco Churn
Prediction
Smart
GridCyber
Security Government /
Law enforcement
ICU
Monitoring
Environment
MonitoringValue
Clear business goals
Business change driven outcomes
Volume
Terabytes/second
Petabytes/day
Variety
All kinds of data
All kinds of analytics
Velocity
Decisions in microseconds
Massively scalable
Veracity
Screening, validation & certification of data
Example Streaming Data Sources:
Video, Audio, Networks, Social Media, Sensor, Weather
Realise It. IBM InfoSphere Streams:
Real-Time Adaptive Analytics for Big Data In-Motion
Connected
Car
11
3
© 2014 IBM Corporation
Create foundation
of trusted data
Understand usage and
monitor compliance
Model exposure and
understand variability
Trust the factsTrust the facts Ensure privacyEnsure privacy
and securityand security
Make riskMake risk
aware decisionsaware decisions
Trust It. Be proactive about privacy, security and governance.
14
© 2014 IBM Corporation
Big Data Uses Cases Delivered with Unique IBM Capabilities
Unique IBM Capabilities:
1. In-memory computing with BLU
Acceleration
2. Data privacy and security of big
data
3. Data Discovery and Exploration
4. Building Confidence in Big Data
with Information Governance
5. Stream computing
WATSON FOUNDATIONS
Decision
Management
Planning &
Forecasting
Discovery &
Exploration
Business Intelligence & Predictive Analytics
Content
Analytics
Information Integration & Governance
Data Mgmt &
Warehouse
Hadoop
System
Stream
Computing
Content
Management
WATSON FOUNDATIONS
Decision
Management
Planning &
Forecasting
Discovery &
Exploration
Business Intelligence & Predictive AnalyticsBusiness Intelligence & Predictive Analytics
Content
Analytics
Information Integration & Governance
Data Mgmt &
Warehouse
Hadoop
System
Stream
Computing
Content
Management
Real-time traffic flow
optimisation
Low-latency
network analysis
Fraud & risk
detection
Predictive asset
maintenance
Understand and act on
customer sentiment
Predict and act on
intent to purchase
15
© 2014 IBM Corporation
16
© 2014 IBM Corporation
http://www.youtube.com/watch?v=FGp-h-x0Hss
17
© 2014 IBM Corporation
Building a real-time enterprise is a journey, which depends on a solid Big Data & Analytics
foundation for success
Be proactive
about privacy,
security and
governance
Build a culture
that infuses
analytics
everywhere
Invest in a
big data &
analytics
platform
Imagine It. Realise It. Trust It.
18
© 2014 IBM Corporation
Ian Radmore
IBM Big Data Specialist,
UK & Ireland
IBM United Kingdom Limited
City Gate West
Toll House Hill
Nottingham
NG1 5FN
Mobile +44 7843 368078
Ian.radmore@uk.ibm.com
19

Big Data & Analytics – beyond Hadoop

  • 1.
    © 2014 IBMCorporation Big Data & Analytics – beyond Hadoop Ian Radmore, IBM UKI Big Data Specialist June 18th, 2014
  • 2.
    © 2014 IBMCorporation Data: To have and to hold? Or to Analyse and Act! Data in Data at 2
  • 3.
    © 2014 IBMCorporation The auto industry is already the 2nd largest data generator AND 20% CAGR! Ford Fusion: 145 actuators, 4700 relays and 70 sensors, including radar, sonar, accelerometer, camera, rain sensors. Collectively, these devices generate more than 25 gigabytes of data per hour, which is processed by more than 70 on-board computers. 1 car year = 1TB 3
  • 4.
    © 2014 IBMCorporation A Big Data & Analytics approach helps provide a foundation for a Smarter Enterprise Invest in aInvest in a big data & analyticsbig data & analytics platformplatform Be proactive aboutBe proactive about privacy, security andprivacy, security and governancegovernance Imagine It. Realise It. Trust It. Build a culture thatBuild a culture that infuses analyticsinfuses analytics everywhereeverywhere Confidence in Your Data Confidence in Accelerating Value Confidence in Your Skills 4
  • 5.
    © 2014 IBMCorporation Deployed real-time CDR analysis solution to handle exploding data volume growth and performance requirements Analyzes call, internet usage, and text records in real-time to identify and address poorly performing cells Uses InfoSphere Streams and IBM Netezza Significant Benefits: Over 90% reduction in time to merge/load call record data Over 90% reduction in storage Increased network quality, improved customer satisfaction, reduced churn Sprint Increases Revenue & Improves Customer Satisfaction “Over 90+% reduction in merge/load times and storage requirements” “Over 90+% reduction in merge/load times and storage requirements” Capabilities Utilised: • Stream processing • Data Warehouse Analytics Appliance 5
  • 6.
    © 2014 IBMCorporation • Examines trends, volume, and content of millions of public Twitter messages in real-time • Analytic accelerators to understand sentiment (positive, negative, neutral) • Capabilities • Stream Computing • Visualization • Benefits • Real-time display of public sentiment as candidates respond to questions • Debate winner prediction based on public opinion instead of solely political analysts University of Southern California Innovation Lab Monitors Political Debates Solution to measure public sentiment during key primary & general presidential debates 6
  • 7.
    © 2014 IBMCorporation 7 KTH Swedish Royal Institute of Technology Reducing Traffic Congestion • Deployed real-time Smarter Traffic system to predict and improve traffic flow. • Analyzes streaming real-time data gathered from cameras at entry/exit to city, GPS data from taxis and trucks, and weather information. • Predicts best time and method to travel such as when to leave to catch a flight at the airport Significant benefits: • Enables ability to analyze and predict traffic faster and more accurately than ever before • Provides new insight into mechanisms that affect a complex traffic system • Smarter, more efficient, and more environmentally friendly traffic 7 Capabilities Utilised: Stream Computing 7
  • 8.
    © 2014 IBMCorporation Pacific Northwest Smart Grid Demonstration Project Capabilities: Stream Computing – real-time control system Data Warehouse Appliance – analyze massive data sets Demonstrates scalability from 100 to 500K homes while retaining 10 years’ historical data 60k metered customers in 5 states Accommodates ad hoc analysis of price fluctuation, energy consumption profiles, risk, fraud detection, grid health, etc. 8
  • 9.
    © 2014 IBMCorporation Information Integration & Governance Systems Security On premise, Cloud, As a service Storage IBM Watson Foundations IBM Big Data & Analytics Infrastructure New / Enhanced ApplicationsAll Data What action should I take? Decision management Cognitive What did I learn? Landing, Exploration and Archive data zone EDW and data mart zone Operational data zone Real-time Data Processing & Analytics What is happening? Discovery and exploration Why did it happen? Reporting, content and analysisWhat could happen? Predictive analytics and modelling Deep Analytics data zone 8 Realise It. Invest
  • 10.
    © 2014 IBMCorporation Realise It. In-Store Presence Zones Intelligent location-based technology to gain deep insight into customer in-store behaviour Enable retailers to integrate the physical and digital experience to facilitate an ongoing dialogue that creates loyalty via an exceptional in-store shopping experience Presence Zones Sensors 9
  • 11.
    © 2014 IBMCorporation IBM Internal Use Only Realise It. The Customer Insight Appliance 10
  • 12.
    © 2014 IBMCorporation Realise It. A Multichannel Korean retailer Reliable insight provides decision support for senior management Targeted campaigns can be developed for marketing Precise measurement of cross-channel campaigns Business Challenge: As sales increased for this retailer’s online shopping mall, management experienced increasing difficulty ensuring that an appropriate product mix was being presented to its customers. The Solution: The company adopted sophisticated analytics and marketing automation to understand, predict and act on consumer buying behavior with confidence. Real-time marketing automation delivers personalised content to each shopper, triggered by their interaction history. Delivered at the right place and time, these offers can move the shopper toward a sale and even increase the size of the purchase. “We have greatly improved our understanding of our customers, which is helping us to make smarter decisions that significantly improve business performance.” —Spokesperson, multichannel Korean retailer Combining marketing automation with analytics to personalise communications and optimise offerings 11
  • 13.
    © 2014 IBMCorporation Millions of events per second Microsecond Latency Traditional / Non-traditional data sources Real time delivery Powerful Analytics Algorithmic Trading Telco Churn Prediction Smart GridCyber Security Government / Law enforcement ICU Monitoring Environment MonitoringValue Clear business goals Business change driven outcomes Volume Terabytes/second Petabytes/day Variety All kinds of data All kinds of analytics Velocity Decisions in microseconds Massively scalable Veracity Screening, validation & certification of data Example Streaming Data Sources: Video, Audio, Networks, Social Media, Sensor, Weather Realise It. IBM InfoSphere Streams: Real-Time Adaptive Analytics for Big Data In-Motion Connected Car 11 3
  • 14.
    © 2014 IBMCorporation Create foundation of trusted data Understand usage and monitor compliance Model exposure and understand variability Trust the factsTrust the facts Ensure privacyEnsure privacy and securityand security Make riskMake risk aware decisionsaware decisions Trust It. Be proactive about privacy, security and governance. 14
  • 15.
    © 2014 IBMCorporation Big Data Uses Cases Delivered with Unique IBM Capabilities Unique IBM Capabilities: 1. In-memory computing with BLU Acceleration 2. Data privacy and security of big data 3. Data Discovery and Exploration 4. Building Confidence in Big Data with Information Governance 5. Stream computing WATSON FOUNDATIONS Decision Management Planning & Forecasting Discovery & Exploration Business Intelligence & Predictive Analytics Content Analytics Information Integration & Governance Data Mgmt & Warehouse Hadoop System Stream Computing Content Management WATSON FOUNDATIONS Decision Management Planning & Forecasting Discovery & Exploration Business Intelligence & Predictive AnalyticsBusiness Intelligence & Predictive Analytics Content Analytics Information Integration & Governance Data Mgmt & Warehouse Hadoop System Stream Computing Content Management Real-time traffic flow optimisation Low-latency network analysis Fraud & risk detection Predictive asset maintenance Understand and act on customer sentiment Predict and act on intent to purchase 15
  • 16.
    © 2014 IBMCorporation 16
  • 17.
    © 2014 IBMCorporation http://www.youtube.com/watch?v=FGp-h-x0Hss 17
  • 18.
    © 2014 IBMCorporation Building a real-time enterprise is a journey, which depends on a solid Big Data & Analytics foundation for success Be proactive about privacy, security and governance Build a culture that infuses analytics everywhere Invest in a big data & analytics platform Imagine It. Realise It. Trust It. 18
  • 19.
    © 2014 IBMCorporation Ian Radmore IBM Big Data Specialist, UK & Ireland IBM United Kingdom Limited City Gate West Toll House Hill Nottingham NG1 5FN Mobile +44 7843 368078 Ian.radmore@uk.ibm.com 19