Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Mark Palmer, SVP of Analytics, TIBCO
https://about.me/mark.palmer
7 PREDICTIVE ANALYTICS, SPARK, STREAMING USE CASES
7 Predictive Analytics, Spark, Streaming Use Cases
1. Live Train Time Tables: 40% Reduction in Spread (Dutch Railways)
2. ...
Hadoop + Analytics: Discover, Automate, Act
AUTOMATE: Inject predictive
model into stream
DISCOVER: Data scientists use in...
SPARK
1: CAPTURE STREAMS,
NORMALIZE, PERSIST IN SPARK
Kafka
JMS
- HDFS
- Parquet
- HBase
A
A
A
Cleanse
Normalize
Bin
STREA...
STREAMING DATA PREP
Cleanse
Normalize
Bin
BIG DATA
STREAMING ANALYTICS
Model execution
Stream processing
ANALYTICS
Data di...
Prebuilt building blocks to
speed up Spark
implementations
Data
capture
Data
analysis
Model
scoring
Model
training
TIBCO
B...
IBM
HP
MSFT
Statistical
Correlation
“Buy HP @$92.97 and sell IBM @93.02 now?”
The First Step Forward Toward an Algorithmic...
IBM
Continuous, Not Real-Time
HP
IBM/ HP Spread
“Buy HP @$92.97 and sell IBM @93.02 now?”
Streaming Analytics
© Copyright 2000-2013 TIBCO Software Inc.
StreamBase: Act on What’s Happening NOW
AUTOMATION IS EVIL?
IT DEPENDS ON
YOUR
P...
1. Live Train Time Tables: 40% Reduction in Spread (Dutch Railways)
2. Intelligent Equipment: Saving $40M / year (Oil & Ga...
© Copyright 2000-2016 TIBCO Software Inc.
#1 Modern vehicles are mobile devices
#2
Existing transportation systems rely on...
© Copyright 2000-2016 TIBCO Software Inc.
© Copyright 2000-2016 TIBCO Software Inc.
© Copyright 2000-2016 TIBCO Software Inc.
External Data
Event-Driven Rules &
Predictive Analytics
Trip Optimization
Rules
Predictive
Maintenance
Rules
Alerts
Vehicl...
1,000 trains simultaneously transmit location,
capacity, “blocking.” Alerts on status are sent to
customers.
Bad weather d...
All systems go: timing estimates calculated in real-time
Visualize BE rule results in LiveView
Visualizing Events
CVA simulator creates a delay (e.g., weather, equipment problem)
CVA simulator creates a delay (e.g., weather, equipment problem)
Alerts appear in UI
Critical Business Moment
Trip 2202 will be 8 minutes
late, so trip 2211 is now
delayed
Alert can be sent via Kafka, BPM, signage update
tweet to the public…
Train now resumes normal speed -
although still delayed - trip 2202 no
longer impacts the next trip
1. Train Time Table Deviation: 40% Reduction in Spread (Dutch
Railways)
2. Intelligent Equipment: Saving $40M / year (Oil ...
© Copyright 2000-2016 TIBCO Software Inc.
© Copyright 2000-2016 TIBCO Software Inc.
© Copyright 2000-2016 TIBCO Software Inc.
© Copyright 2000-2016 TIBCO Software Inc.
© Copyright 2000-2016 TIBCO Software Inc.
© Copyright 2000-2016 TIBCO Software Inc.
Voltage
Temperature
Vibration
“When the temperature of any pump goes up more than 20 degrees for any 10
MINUTE window, SCH...
Streaming & Batch Analytics
Continuous
Predictive
Maintenance
Risk Management TIBCO Live Datamart
Integration
Geo-aware
an...
1. Train Time Table Deviation: 40% Reduction in Spread (Dutch
Railways)
2. Intelligent Equipment: Saving $40M / year (Oil ...
© Copyright 2000-2016 TIBCO Software Inc.
Streaming Data
Continuous Digital Loyalty
IoT Streaming
Analytics
Social Analytics
Live Datamart
Enterprise Data
Integrati...
1. Train Time Table Deviation: 40% Reduction in Spread (Dutch
Railways)
2. Intelligent Equipment: Saving $40M / year (Oil ...
“In December, 2012, Knight Capital lost $460M in under
40 minutes. That changed everything. Now, it’s no longer
acceptable...
Continuous Compliance
Market Data Stream
Streaming Analytics
Large Orders
Marking the tape
Layering
Ramping on
close
Alert...
Continuous Query
Continuous Query Processor Alerts
Rules
FTL
EMS
ActiveSpaces
Application Data
Social Media Data
Market Da...
1. Train Time Table Deviation: 40% Reduction in Spread (Dutch
Railways)
2. Intelligent Equipment: Saving $40M / year (Oil ...
© Copyright 2000-2016 TIBCO Software Inc.
Live Flight Operations & United Airlines
1. Train Time Table Deviation: 40% Reduction in Spread (Dutch
Railways)
2. Intelligent Equipment: Saving $40M / year (Oil ...
© Copyright 2000-2016 TIBCO Software Inc.
1. Train Time Table Deviation: 40% Reduction in Spread (Dutch
Railways)
2. Intelligent Equipment: Saving $40M / year (Oil ...
Location Automation
Rules
TIBCO BusinessEvents
Enterprise Integration BusTIBCO Enterprise Message Bus
Analytics Event
Aggr...
1. Live Train Time Tables: 40% Reduction in Spread (Dutch Railways)
2. Intelligent Equipment: Saving $40M / year (Oil & Ga...
Hadoop + Analytics: Discover, Automate, Act
AUTOMATE: Inject predictive
model into stream
DISCOVER: Data scientists use in...
7 Predictive Analytics, Spark , Streaming use cases
Upcoming SlideShare
Loading in …5
×

7 Predictive Analytics, Spark , Streaming use cases

5,892 views

Published on

7 Predictive Analytics, Spark , Streaming use cases

Published in: Technology
  • Be the first to comment

7 Predictive Analytics, Spark , Streaming use cases

  1. 1. Mark Palmer, SVP of Analytics, TIBCO https://about.me/mark.palmer 7 PREDICTIVE ANALYTICS, SPARK, STREAMING USE CASES
  2. 2. 7 Predictive Analytics, Spark, Streaming Use Cases 1. Live Train Time Tables: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. Continuous Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. IoT Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK)
  3. 3. Hadoop + Analytics: Discover, Automate, Act AUTOMATE: Inject predictive model into stream DISCOVER: Data scientists use interactive analytics to discover, score and model based on Hadoop / Spark data lakes Automation Case ManagementOperational Intelligence ACT: Automation, alerting and refinement, BPM
  4. 4. SPARK 1: CAPTURE STREAMS, NORMALIZE, PERSIST IN SPARK Kafka JMS - HDFS - Parquet - HBase A A A Cleanse Normalize Bin STREAMING DATA PREP StreamBase 2: DISCOVER MODEL Data scientists ANALYTICS REAL-TIME SPARK ACCELERATOR PATTERN POS Mobile Web Operations LIVE MONITORING & ANALYTICS Live Datamart, LiveView 6: LIVEVIEW 3: LOAD PREDICTIVE MODEL Model Stream Scoring STREAMING ANALYTICS StreamBase Real-Time Action 4: CONTINUOUS ALGORITHMIC ACTION Upsell Recommendation History 5: AUTO-RETRAINING MODEL TRACKING Real-time training StreamBase IMPALA
  5. 5. STREAMING DATA PREP Cleanse Normalize Bin BIG DATA STREAMING ANALYTICS Model execution Stream processing ANALYTICS Data discovery Model discovery Load model Messaging layer : - Kafka - HiveMQ - JMS - ActiveMQ - RabbitMQ - FTL - … Direct access : - Websocket - TCP/UDP - MQTT - HTTP - ... Public/private APIs : - Twitter, Faceboook,... - Google finance - ... HDFS, Hbase, Parquet, Avro SQL Data scientistsOperations The Spark Accelerator Pattern MODEL TRACKING Real-time model training Live monitoring CONNECTIVITY
  6. 6. Prebuilt building blocks to speed up Spark implementations Data capture Data analysis Model scoring Model training TIBCO Big Data Accelerator SIMPLIFYING SPARK
  7. 7. IBM HP MSFT Statistical Correlation “Buy HP @$92.97 and sell IBM @93.02 now?” The First Step Forward Toward an Algorithmic Computing
  8. 8. IBM Continuous, Not Real-Time HP IBM/ HP Spread “Buy HP @$92.97 and sell IBM @93.02 now?” Streaming Analytics
  9. 9. © Copyright 2000-2013 TIBCO Software Inc. StreamBase: Act on What’s Happening NOW AUTOMATION IS EVIL? IT DEPENDS ON YOUR PERSPECTIVE
  10. 10. 1. Live Train Time Tables: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. Continuous Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. IoT Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK 7 Predictive Analytics, Spark, Streaming Use Cases
  11. 11. © Copyright 2000-2016 TIBCO Software Inc. #1 Modern vehicles are mobile devices #2 Existing transportation systems rely on routes, schedules, work assignments: a “rear-view mirror” view approach #3 Millennials demand real-time insight into everything The Connected Vehicle Business Challenge
  12. 12. © Copyright 2000-2016 TIBCO Software Inc.
  13. 13. © Copyright 2000-2016 TIBCO Software Inc.
  14. 14. © Copyright 2000-2016 TIBCO Software Inc.
  15. 15. External Data Event-Driven Rules & Predictive Analytics Trip Optimization Rules Predictive Maintenance Rules Alerts Vehicle Clustering Rules Location Stream TIBCO Live Datamart AMX BPM Billions of events Traffic, Twitter, Weather Connected Vehicle Data Weather Case ManagementEnterprise Data BusinessWorks Real-Time Geo Fencing Rules Predictive Route Optimization Journey Disruption Rules Business Events, TERR, StreamBase Live Datamart Operational Command & Control App LiveView Analytics Hadoop / Spark Spotfire Scheduling, Maintenance, MDM, CRM ALERTS IoT Connected Vehicle Architecture
  16. 16. 1,000 trains simultaneously transmit location, capacity, “blocking.” Alerts on status are sent to customers. Bad weather delays a trip; streaming analytics continuously re-calculates the impact based on state in real-time Operators analyze the impact in real-time, re- calculates train “blocking,” and take action Operations returns to normal, customers alerted A Moment in the Life of a Connected Vehicle
  17. 17. All systems go: timing estimates calculated in real-time
  18. 18. Visualize BE rule results in LiveView Visualizing Events
  19. 19. CVA simulator creates a delay (e.g., weather, equipment problem)
  20. 20. CVA simulator creates a delay (e.g., weather, equipment problem) Alerts appear in UI
  21. 21. Critical Business Moment Trip 2202 will be 8 minutes late, so trip 2211 is now delayed
  22. 22. Alert can be sent via Kafka, BPM, signage update tweet to the public…
  23. 23. Train now resumes normal speed - although still delayed - trip 2202 no longer impacts the next trip
  24. 24. 1. Train Time Table Deviation: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. IT and Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK) 7 Predictive Analytics, Spark, Streaming Use Cases
  25. 25. © Copyright 2000-2016 TIBCO Software Inc.
  26. 26. © Copyright 2000-2016 TIBCO Software Inc.
  27. 27. © Copyright 2000-2016 TIBCO Software Inc.
  28. 28. © Copyright 2000-2016 TIBCO Software Inc.
  29. 29. © Copyright 2000-2016 TIBCO Software Inc.
  30. 30. © Copyright 2000-2016 TIBCO Software Inc.
  31. 31. Voltage Temperature Vibration “When the temperature of any pump goes up more than 20 degrees for any 10 MINUTE window, SCHEDULE MAINTENANCE” Algorithmic IoT
  32. 32. Streaming & Batch Analytics Continuous Predictive Maintenance Risk Management TIBCO Live Datamart Integration Geo-aware analytics Facility Management Alert Targeting TERR, PMML, StreamBase, BusinessEvents Digital Operations TIBCO LiveView Analytics Spotfire ALERTS Case Management Mobile Weather BusinessWorks,EMS,TIBCOMashery,eFTL TIBCO BPM Data Scientists Digital Operations (e.g., Drilling Operations) OSI PI Engineering Documents Financial WITSML In Memory Data Grid Open Spirit MDM Cloud Foundry Industrial Equipment Monitoring Industrial Equipment & Spark Spark
  33. 33. 1. Train Time Table Deviation: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. IT and Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK) 7 Predictive Analytics, Spark, Streaming Use Cases
  34. 34. © Copyright 2000-2016 TIBCO Software Inc.
  35. 35. Streaming Data Continuous Digital Loyalty IoT Streaming Analytics Social Analytics Live Datamart Enterprise Data Integration In memory data grid Segment & Target Offers & Points Digital Operations Analytics ALERTS Algorithmic Loyalty Case Management API Management Supply Chain Partners Mobile Vehicles Mobile Loyalty Wearables Data Scientists Digital Operations Call Centers Mobile Rewards Generic EventsReport & Analyze Operations
  36. 36. 1. Train Time Table Deviation: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. IT and Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK) 7 Predictive Analytics, Spark, Streaming Use Cases
  37. 37. “In December, 2012, Knight Capital lost $460M in under 40 minutes. That changed everything. Now, it’s no longer acceptable to run our business based on end-of-day reports.” - Head of Risk Management, top 3 bank
  38. 38. Continuous Compliance Market Data Stream Streaming Analytics Large Orders Marking the tape Layering Ramping on close Alerts Ramping on open Spiking Spoofing (1) Spoofing (2) Spoofing (3) Wash Trades Sensitivity Adjustments Wall Street Continuous Compliance Architecture Audit Trail Logging Compliance Alerting Audit Order Stream Live Datamart In aggregate, peak event rates of 600,000 events a second, or a rate of 51 billion events a day Compliance staff +100M orders a day, 90% cancel rate 500,000 EPS peak Continuous compliance analytics answer every interesting surveillance question, (at the peak rate of) 51 billion times a day Orders Market Data Contextual Case Management
  39. 39. Continuous Query Continuous Query Processor Alerts Rules FTL EMS ActiveSpaces Application Data Social Media Data Market Data Sensor Data Spark In memory data grid Enterprise data Market Data IoT Mobile Social Command & Control ACTION The Birth of the Live Datamart Live Datamart
  40. 40. 1. Train Time Table Deviation: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. IT and Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK) 7 Predictive Analytics, Spark, Streaming Use Cases
  41. 41. © Copyright 2000-2016 TIBCO Software Inc. Live Flight Operations & United Airlines
  42. 42. 1. Train Time Table Deviation: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. IT and Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK) 7 Predictive Analytics, Spark, Streaming Use Cases
  43. 43. © Copyright 2000-2016 TIBCO Software Inc.
  44. 44. 1. Train Time Table Deviation: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. IT and Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK) 7 Predictive Analytics, Spark, Streaming Use Cases
  45. 45. Location Automation Rules TIBCO BusinessEvents Enterprise Integration BusTIBCO Enterprise Message Bus Analytics Event Aggregator Hadoop TIBCO BusinessWorks Enterprise Application Web In-Memory Operational Data Store TIBCO BusinessWorks, Activespaces SMS Email PDA API Management TIBCO API Exchange Mobile Apps Operational Control TIBCO Live Datamart & LiveView Partners Enterprise Application Enterprise Application Enterprise Apps Sensor Data The Postal Service Internet of Things
  46. 46. 1. Live Train Time Tables: 40% Reduction in Spread (Dutch Railways) 2. Intelligent Equipment: Saving $40M / year (Oil & Gas – Many) 3. Algorithmic Loyalty: Finding the Jacket You Didn’t Know You Needed (North Face) 4. Predictive Risk & Compliance: Avoiding $440M in 40 Minutes of Loss (ConvergEx) 5. Live Flight Optimization: Get You Home on Time (United Airlines) 6. IT and Transaction Optimization: Watch 20,000 Systems at Once (Morgan Stanley) 7. IoT Parcel Tracking: From 20% to 100% Real-Time (Royal Mail, UK 7 Predictive Analytics, Spark, Streaming Use Cases
  47. 47. Hadoop + Analytics: Discover, Automate, Act AUTOMATE: Inject predictive model into stream DISCOVER: Data scientists use interactive analytics to discover, score and model based on Hadoop / Spark data lakes Automation Case ManagementOperational Intelligence ACT: Automation, alerting and refinement, BPM

×