© 2014 IBM Corporation
Big Data Platform
Arild Kristensen
Nordic Sales Manager, Big Data Analytics
Tlf.: +47 90532591
Email: arild.kristensen@no.ibm.com
© 2014 IBM Corporation3
© 2014 IBM Corporation4
Welcome to the Big Data Opportunity
“The list of life's certainties has gotten longer.
Along with death and taxes we can now include
information overload.”
© 2014 IBM Corporation5
We have for the first time an economy based on a key resource
[Information] that is not only renewable, but self-generating.
Running out of it is not a problem, but drowning in it is.
– John Naisbitt
Source, Megatrends, Naisbitt, John, Grand Central Publishing 1988
We are not suffering from Information Overload. We
are suffering from Filter Failure.
– Clay Shirky
Sourcehttp://www.ted.com/talks/view/lang/en//id/575
© 2014 IBM Corporation6
Welcome to the Big Data Opportunity
Research firm IDC expects Big Data to grow from
$3.2 billion in 2010 to $16.9 billion in 2015
by 2015 we'll see 4.4 million jobs devoted to the
global support of Big Data
each IT job created by Big Data will generate
three more positions outside of IT.
© 2014 IBM Corporation11
Big Data Analytics And Natural Language
Cognitive: The Next Wave of Disruptive Technology
© 2014 IBM Corporation14
Understands
natural language and
human style
communication
Adapts and learns from
training, interaction,
and outcomes
Generates and
evaluates evidence-
based hypothesis
1 2
3
• Understands me
• Engages me
• Learns and improves over time
• Helps me discover
• Establishes trust
• Has endless capacity for insight
• Operates in a timely fashion
Watson combines transformational capabilities to deliver a
new world experience using cognitive computing
Watson:
© 2014 IBM Corporation15
IBM Watson
family
IBM Watson
Solutions
IBM Watson
Transformation
IBM Watson
Foundations
IBM Watson
Innovations
Provides the big data and analytics
capabilities that fuel Watson
Products based on
Watson’s core
attributes and
capabilities
APIs, tools, methodologies,
SDKs, and infrastructure that
fuels Watson
Bespoke solutions designed to
meet some of industries most
demanding needs leveraging
cognitive capabilities
IBM Watson
Ecosystems
The Watson Developer Cloud,
Watson Content Store and
Watson Talent Hub driving
innovation from partners
Introducing the IBM Watson family
© 2014 IBM Corporation16
How is Big Data transforming the way
organizations analyze information and
generate actionable insights?
© 2014 IBM Corporation17
Paradigm shifts enabled by big data
Leverage more of the data being captured
TRADITIONAL APPROACH BIG DATA APPROACH
Analyze small subsets
of information
Analyze
all information
Analyzed
information
All available
information
All available
information
analyzed
© 2014 IBM Corporation18
Paradigm shifts enabled by big data
Reduce effort required to leverage data
TRADITIONAL APPROACH BIG DATA APPROACH
Carefully cleanse information
before any analysis
Analyze information as is,
cleanse as needed
Small
amount of
carefully
organized
information
Large
amount of
messy
information
© 2014 IBM Corporation19
Paradigm shifts enabled by big data
Data leads the way—and sometimes correlations are good enough
TRADITIONAL APPROACH BIG DATA APPROACH
Start with hypothesis and
test against selected data
Explore all data and
identify correlations
Hypothesis Question
DataAnswer
Data Exploration
CorrelationInsight
© 2014 IBM Corporation20
Paradigm shifts enabled by big data
Leverage data as it is captured
TRADITIONAL APPROACH BIG DATA APPROACH
Analyze data after it’s been
processed and landed in a warehouse
or mart
Analyze data in motion as it’s
generated, in real-time
Repository InsightAnalysisData
Data
Insight
Analysis
© 2014 IBM Corporation21
Hadoop &
Streaming
Data
New
Sources
Unstructured
Exploratory
Iterative
Structured
Repeatable
Linear
Data
Warehouse
Traditional
Sources
Traditional Approach
Structured, analytical, logical
New Approach
Creative, holistic thought, intuition
Enterprise
Integration
Customer Data
Transaction Data
3rd Party Data
Core System Data
Web Logs, URLs
Social Data
Text Data: emails, chats
Log data
Analytics is expanding from enterprise data to big data,
creating new opportunities for competitive advantage
Contact Center notes
Geolocation data
© 2014 IBM Corporation22
Addressing Client Challenges through Big
Data Platform
© 2014 IBM Corporation23
A New Architectural Approach is Required
Information Integration & Governance
Systems Security
On premise, Cloud, As a service
Storage
New/Enhanced
Applications
All Data
What action
should I
take?
Decision
management
Landing,
Exploration
and Archive
data zone
EDW and
data mart
zone
Operational
data zone
Real-time Data Processing & Analytics What is
happening?
Discovery and
exploration
Why did it
happen?
Reporting and
analysis
What could
happen?
Predictive
analytics and
modeling
Deep
Analytics
data zone What did
I learn,
what’s best?
Cognitive
© 2014 IBM Corporation24
Information Integration & Governance
Actionable insight
Exploration,
landing and
archive
Trusted data
Reporting &
interactive
analysis
Deep
analytics &
modeling
Data types Real-time processing & analytics
Transaction and
application data
Machine and
sensor data
Enterprise
content
Social data
Image and video
Third-party data
Decision
management
Predictive analytics
and modeling
Reporting,
analysis, content
analytics
Discovery and
exploration
Operational
systems
Information
Integration
Data Matching
& MDM
Security &
Privacy
Lifecycle
Management
Metadata &
Lineage
IBM Big Data Analytics (Watson Foundations) - One architecture
that fits together
BigInsights
Streams
PureData
for
Analytics
DB2 Blu
Watson
Explorer
Cognos
Cognos
SPSSPureData
for
Analytics
PureData
Operational
Analytics
© 2014 IBM Corporation25
InfoSphere
DataStage
Automatically push transformational processing close to where the
data resides, both SQL for DBMS and MapReduce for Hadoop,
leveraging the same simple data flow design process and coordinate
workflow across all platforms
“Big Data Expert”
© 2014 IBM Corporation
IBM InfoSphere Streams:
Get real-time insights from data in-motion
© 2014 IBM Corporation27
27
Current fact finding
Analyze data in motion – before it is stored
Low latency paradigm, push model
Data driven – bring data to the analytics
Historical fact finding
Find and analyze information stored on disk
Batch paradigm, pull model
Query-driven: submits queries to static data
Traditional Computing Stream Computing
Stream Computing Represents a Paradigm Shift
Real-time
Analytics
© 2014 IBM Corporation28
28
Modify
Filter / Sample
Classify
Fuse
Annotate
Big Data in Real Time with InfoSphere Streams
Score
Windowed
Aggregates
Analyze
© 2014 IBM Corporation29
29
Streams Analyzes All Variety of Data
Mining in Microseconds
(included with Streams)
Image & Video
(Open Source)
Simple & Advanced Text
(included with Streams)
Text
(listen, verb),
(radio, noun)
Acoustic
(IBM Research)
(Open Source)
Geospatial
(Included with
Streams)
Predictive
(Included with
Streams)
Advanced
Mathematical
Models
(Included with
Streams)
Statistics
(included with
Streams)
∑population
tt asR ),(
Blue = included with the product
Red = built for Streams and used in
projects but not yet part of the product
© 2014 IBM Corporation30
30
How is Streams Being Used?
Stock market
Impact of weather on
securities prices
Analyze market data at
ultra-low latencies
Momentum Calculator
Fraud prevention
Detecting multi-party fraud
Real time fraud prevention
e-Science
Space weather prediction
Detection of transient events
Synchrotron atomic research
Genomic Research
Transportation
Intelligent traffic
management
Automotive Telematics
Energy & Utilities
Transactive control
Phasor Monitoring Unit
Down hole sensor monitoring
Natural Systems
Wildfire management
Water management
Other
Manufacturing
Text Analysis
ERP for Commodities
Real-time multimodal surveillance
Situational awareness
Cyber security detection
Law Enforcement,
Defense & Cyber Security
Health & Life
SciencesICU monitoring
Epidemic early
warning system
Remote healthcare
monitoring
Telephony
CDR processing
Social analysis
Churn prediction
Geomapping
© 2014 IBM Corporation
Watson (Data) Explorer
IBM Software Group
Information Management
Big Data
© 2014 IBM Corporation32
Watson Explorer solves #1 challenge customers face in Big Data:
Unlocking the value of information through a single interface
Create unified view of
ALL information for
real-time monitoring
Identify areas of information
risk & ensure data
compliance
Analyze customer analytics
& data to unlock true
customer value
Increase productivity &
leverage past work
increasing speed to market
Improve customer
service & reduce
call times
InfoSphere
Data Explorer
• Analyzes structured &
unstructured data—in place
• Unique positional indexing
• Unlimited scalability
• Advanced data asset navigation
• Pattern clustering
• Virtual documents
Contextual intelligence
• Text analytics
• Secure data integration
• Query transformation
• Easy-to-deploy big data applications
• User-friendly customisable interface
Providing unified, real-time
access and fusion of big
data unlocks greater
insight and ROI
Zoom in
Zoom out
12/05/201432
© 2014 IBM Corporation33
Watson Explorer Application Architecture
User Profiles
360O View
Applications
Information
Discovery
Applications
Big Data
Applications
Discovery &
navigation
applications
Web
Results
FeedsSubscriptions
Federated Query Routing
Application Framework
Authentication/Authorization
Query transformation
Personalization
Display
Meta-Data
User Profiles
Application layer
managing user
interactions, apps,
creating context,
routing queries
Thesauri
Clustering
Ontology Support
Semantic Processing
Entity Extraction
Relevancy
Text Analytics
Search Engine Metadata Extraction
Faceting
BI
Tagging
Taxonomy
Collaboration
Processing layer
for indexing,
analysis &
conversion
CM, RM, DM RDBMS Feeds Web 2.0 Email Web CRM, ERP File
Systems
Connector
Framework
Framework for
accessing data
sources
12/05/201433
© 2014 IBM Corporation34
Highly relevant, secure &
personalized results
Access all sources
or individual source
Refinements based
on metadata
Dynamic
categorization
Narrow down results set
Information Navigation, Discovery & Insight Through One Interface
Live link here
Setup alert to
notify change
Identify topical experts
Tag results
Rate results
Comment results
Store &
share results
© 2014 IBM Corporation35
Big Data Use cases
© 2014 IBM Corporation36
Top sources of information used as part of initial big data efforts –
typically start with data already being captured
Source: The real world use of Big Data, IBM
& University of Oxford
Big data sources
Respondents with active big data efforts were asked which data sources are
currently being collected and analyzed as part of active big data efforts within
their organization.
88%
73%
59%
57%
43%
42%
42%
41%
41%
40%
38%
34%
92%
81%
70%
65%
27%
19%
36%
47%
32%
0%
21%
22%
Transactions
LogData
Events
Emails
Social Media
Sensors
External Feeds
RFID Scans or POS Data
Free-formText
Geospatial
Audio
Still Images / Videos
Banking & Fin Mgmt
respondents
Global respondents
3
6
© 2014 IBM Corporation37
Big Data Exploration
Find, visualize, and understand
all big data for improved decision
making
Enhanced 360o View
of the Customer
View all internal and external
information sources to know
everything about your customers
Operations Analysis
Analyze a variety of machine data
for improved business results
Data Warehouse
Modernization
Modernize the data warehouse with
new technology: in-memory, stream
computing, Hadoop, appliances,
while building confidence in all data
Security Intelligence
Extension
Lower risk, detect fraud and
monitor cyber security in real-time
Big Data Use Cases
© 2014 IBM Corporation38
Arild Kristensen IBM Norway
Nordic Sales Manager Forusbeen 10
Big Data Analytics 4033 Stavanger
IBM Software Group Mobile: +47 90 53 25 91
Information Management arild.kristensen@no.ibm.com
linkedin.com/pub/arild-
kristensen/34/96b/184
twitter.com/ArildWK
www.ibmbigdatahub.com
www.analyticszone.com

Ibm big data-platform

  • 1.
    © 2014 IBMCorporation Big Data Platform Arild Kristensen Nordic Sales Manager, Big Data Analytics Tlf.: +47 90532591 Email: arild.kristensen@no.ibm.com
  • 2.
    © 2014 IBMCorporation3
  • 3.
    © 2014 IBMCorporation4 Welcome to the Big Data Opportunity “The list of life's certainties has gotten longer. Along with death and taxes we can now include information overload.”
  • 4.
    © 2014 IBMCorporation5 We have for the first time an economy based on a key resource [Information] that is not only renewable, but self-generating. Running out of it is not a problem, but drowning in it is. – John Naisbitt Source, Megatrends, Naisbitt, John, Grand Central Publishing 1988 We are not suffering from Information Overload. We are suffering from Filter Failure. – Clay Shirky Sourcehttp://www.ted.com/talks/view/lang/en//id/575
  • 5.
    © 2014 IBMCorporation6 Welcome to the Big Data Opportunity Research firm IDC expects Big Data to grow from $3.2 billion in 2010 to $16.9 billion in 2015 by 2015 we'll see 4.4 million jobs devoted to the global support of Big Data each IT job created by Big Data will generate three more positions outside of IT.
  • 6.
    © 2014 IBMCorporation11 Big Data Analytics And Natural Language Cognitive: The Next Wave of Disruptive Technology
  • 7.
    © 2014 IBMCorporation14 Understands natural language and human style communication Adapts and learns from training, interaction, and outcomes Generates and evaluates evidence- based hypothesis 1 2 3 • Understands me • Engages me • Learns and improves over time • Helps me discover • Establishes trust • Has endless capacity for insight • Operates in a timely fashion Watson combines transformational capabilities to deliver a new world experience using cognitive computing Watson:
  • 8.
    © 2014 IBMCorporation15 IBM Watson family IBM Watson Solutions IBM Watson Transformation IBM Watson Foundations IBM Watson Innovations Provides the big data and analytics capabilities that fuel Watson Products based on Watson’s core attributes and capabilities APIs, tools, methodologies, SDKs, and infrastructure that fuels Watson Bespoke solutions designed to meet some of industries most demanding needs leveraging cognitive capabilities IBM Watson Ecosystems The Watson Developer Cloud, Watson Content Store and Watson Talent Hub driving innovation from partners Introducing the IBM Watson family
  • 9.
    © 2014 IBMCorporation16 How is Big Data transforming the way organizations analyze information and generate actionable insights?
  • 10.
    © 2014 IBMCorporation17 Paradigm shifts enabled by big data Leverage more of the data being captured TRADITIONAL APPROACH BIG DATA APPROACH Analyze small subsets of information Analyze all information Analyzed information All available information All available information analyzed
  • 11.
    © 2014 IBMCorporation18 Paradigm shifts enabled by big data Reduce effort required to leverage data TRADITIONAL APPROACH BIG DATA APPROACH Carefully cleanse information before any analysis Analyze information as is, cleanse as needed Small amount of carefully organized information Large amount of messy information
  • 12.
    © 2014 IBMCorporation19 Paradigm shifts enabled by big data Data leads the way—and sometimes correlations are good enough TRADITIONAL APPROACH BIG DATA APPROACH Start with hypothesis and test against selected data Explore all data and identify correlations Hypothesis Question DataAnswer Data Exploration CorrelationInsight
  • 13.
    © 2014 IBMCorporation20 Paradigm shifts enabled by big data Leverage data as it is captured TRADITIONAL APPROACH BIG DATA APPROACH Analyze data after it’s been processed and landed in a warehouse or mart Analyze data in motion as it’s generated, in real-time Repository InsightAnalysisData Data Insight Analysis
  • 14.
    © 2014 IBMCorporation21 Hadoop & Streaming Data New Sources Unstructured Exploratory Iterative Structured Repeatable Linear Data Warehouse Traditional Sources Traditional Approach Structured, analytical, logical New Approach Creative, holistic thought, intuition Enterprise Integration Customer Data Transaction Data 3rd Party Data Core System Data Web Logs, URLs Social Data Text Data: emails, chats Log data Analytics is expanding from enterprise data to big data, creating new opportunities for competitive advantage Contact Center notes Geolocation data
  • 15.
    © 2014 IBMCorporation22 Addressing Client Challenges through Big Data Platform
  • 16.
    © 2014 IBMCorporation23 A New Architectural Approach is Required Information Integration & Governance Systems Security On premise, Cloud, As a service Storage New/Enhanced Applications All Data What action should I take? Decision management Landing, Exploration and Archive data zone EDW and data mart zone Operational data zone Real-time Data Processing & Analytics What is happening? Discovery and exploration Why did it happen? Reporting and analysis What could happen? Predictive analytics and modeling Deep Analytics data zone What did I learn, what’s best? Cognitive
  • 17.
    © 2014 IBMCorporation24 Information Integration & Governance Actionable insight Exploration, landing and archive Trusted data Reporting & interactive analysis Deep analytics & modeling Data types Real-time processing & analytics Transaction and application data Machine and sensor data Enterprise content Social data Image and video Third-party data Decision management Predictive analytics and modeling Reporting, analysis, content analytics Discovery and exploration Operational systems Information Integration Data Matching & MDM Security & Privacy Lifecycle Management Metadata & Lineage IBM Big Data Analytics (Watson Foundations) - One architecture that fits together BigInsights Streams PureData for Analytics DB2 Blu Watson Explorer Cognos Cognos SPSSPureData for Analytics PureData Operational Analytics
  • 18.
    © 2014 IBMCorporation25 InfoSphere DataStage Automatically push transformational processing close to where the data resides, both SQL for DBMS and MapReduce for Hadoop, leveraging the same simple data flow design process and coordinate workflow across all platforms “Big Data Expert”
  • 19.
    © 2014 IBMCorporation IBM InfoSphere Streams: Get real-time insights from data in-motion
  • 20.
    © 2014 IBMCorporation27 27 Current fact finding Analyze data in motion – before it is stored Low latency paradigm, push model Data driven – bring data to the analytics Historical fact finding Find and analyze information stored on disk Batch paradigm, pull model Query-driven: submits queries to static data Traditional Computing Stream Computing Stream Computing Represents a Paradigm Shift Real-time Analytics
  • 21.
    © 2014 IBMCorporation28 28 Modify Filter / Sample Classify Fuse Annotate Big Data in Real Time with InfoSphere Streams Score Windowed Aggregates Analyze
  • 22.
    © 2014 IBMCorporation29 29 Streams Analyzes All Variety of Data Mining in Microseconds (included with Streams) Image & Video (Open Source) Simple & Advanced Text (included with Streams) Text (listen, verb), (radio, noun) Acoustic (IBM Research) (Open Source) Geospatial (Included with Streams) Predictive (Included with Streams) Advanced Mathematical Models (Included with Streams) Statistics (included with Streams) ∑population tt asR ),( Blue = included with the product Red = built for Streams and used in projects but not yet part of the product
  • 23.
    © 2014 IBMCorporation30 30 How is Streams Being Used? Stock market Impact of weather on securities prices Analyze market data at ultra-low latencies Momentum Calculator Fraud prevention Detecting multi-party fraud Real time fraud prevention e-Science Space weather prediction Detection of transient events Synchrotron atomic research Genomic Research Transportation Intelligent traffic management Automotive Telematics Energy & Utilities Transactive control Phasor Monitoring Unit Down hole sensor monitoring Natural Systems Wildfire management Water management Other Manufacturing Text Analysis ERP for Commodities Real-time multimodal surveillance Situational awareness Cyber security detection Law Enforcement, Defense & Cyber Security Health & Life SciencesICU monitoring Epidemic early warning system Remote healthcare monitoring Telephony CDR processing Social analysis Churn prediction Geomapping
  • 24.
    © 2014 IBMCorporation Watson (Data) Explorer IBM Software Group Information Management Big Data
  • 25.
    © 2014 IBMCorporation32 Watson Explorer solves #1 challenge customers face in Big Data: Unlocking the value of information through a single interface Create unified view of ALL information for real-time monitoring Identify areas of information risk & ensure data compliance Analyze customer analytics & data to unlock true customer value Increase productivity & leverage past work increasing speed to market Improve customer service & reduce call times InfoSphere Data Explorer • Analyzes structured & unstructured data—in place • Unique positional indexing • Unlimited scalability • Advanced data asset navigation • Pattern clustering • Virtual documents Contextual intelligence • Text analytics • Secure data integration • Query transformation • Easy-to-deploy big data applications • User-friendly customisable interface Providing unified, real-time access and fusion of big data unlocks greater insight and ROI Zoom in Zoom out 12/05/201432
  • 26.
    © 2014 IBMCorporation33 Watson Explorer Application Architecture User Profiles 360O View Applications Information Discovery Applications Big Data Applications Discovery & navigation applications Web Results FeedsSubscriptions Federated Query Routing Application Framework Authentication/Authorization Query transformation Personalization Display Meta-Data User Profiles Application layer managing user interactions, apps, creating context, routing queries Thesauri Clustering Ontology Support Semantic Processing Entity Extraction Relevancy Text Analytics Search Engine Metadata Extraction Faceting BI Tagging Taxonomy Collaboration Processing layer for indexing, analysis & conversion CM, RM, DM RDBMS Feeds Web 2.0 Email Web CRM, ERP File Systems Connector Framework Framework for accessing data sources 12/05/201433
  • 27.
    © 2014 IBMCorporation34 Highly relevant, secure & personalized results Access all sources or individual source Refinements based on metadata Dynamic categorization Narrow down results set Information Navigation, Discovery & Insight Through One Interface Live link here Setup alert to notify change Identify topical experts Tag results Rate results Comment results Store & share results
  • 28.
    © 2014 IBMCorporation35 Big Data Use cases
  • 29.
    © 2014 IBMCorporation36 Top sources of information used as part of initial big data efforts – typically start with data already being captured Source: The real world use of Big Data, IBM & University of Oxford Big data sources Respondents with active big data efforts were asked which data sources are currently being collected and analyzed as part of active big data efforts within their organization. 88% 73% 59% 57% 43% 42% 42% 41% 41% 40% 38% 34% 92% 81% 70% 65% 27% 19% 36% 47% 32% 0% 21% 22% Transactions LogData Events Emails Social Media Sensors External Feeds RFID Scans or POS Data Free-formText Geospatial Audio Still Images / Videos Banking & Fin Mgmt respondents Global respondents 3 6
  • 30.
    © 2014 IBMCorporation37 Big Data Exploration Find, visualize, and understand all big data for improved decision making Enhanced 360o View of the Customer View all internal and external information sources to know everything about your customers Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Modernization Modernize the data warehouse with new technology: in-memory, stream computing, Hadoop, appliances, while building confidence in all data Security Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Big Data Use Cases
  • 31.
    © 2014 IBMCorporation38 Arild Kristensen IBM Norway Nordic Sales Manager Forusbeen 10 Big Data Analytics 4033 Stavanger IBM Software Group Mobile: +47 90 53 25 91 Information Management arild.kristensen@no.ibm.com linkedin.com/pub/arild- kristensen/34/96b/184 twitter.com/ArildWK www.ibmbigdatahub.com www.analyticszone.com