ADVANCED ANALYTICS
FOR ANY DATA
AT REAL-TIME SPEED
Dan Potter
Chief Marketing Officer, Datawatch Corporation
How Many Sensors
Are In Your Pocket?
NASDAQ: DWCH
Pioneer in real-time visual data discovery and self-service data
preparation
Global operations and support
 US, UK, Germany, France, Australia, Singapore, Philippines
Extensive global customer base
 93 of the Fortune 100
 12 of the 15 largest financial institutions
Embedded and resold by leading vendors
About Datawatch
IoT and Industrial Analytics
50 billion devices will be interconnected by 2020
- Cisco
By 2017, over 50% of analytics implementations will make use of
event data streams generated from instrumented machines,
applications and/or individuals
- Gartner
Real-time visualization leads to more opportunities, greater
output, and lower costs
- Aberdeen Group
IoT companies attracted more than $1 billion in venture capital
- Forbes
What is Industrial Big Data?
IoT Platform Requirements
Stream
Processing
Visual Data
Discovery
Real-Time
Transport
Sensor
Data
OT
Data
IT
Data
Visualization for IoT
Data
Access
Visual
Discovery
Monitor
and Alerts
Data
Preparation Predictive
Analytics
Take
Action
6 Requirements for IoT Visualization
Data Discovery
Streaming Data Visualization
Time Series Data
Predictive & Advanced Analytics
Data Preparation
Real-time Geospatial & Location
#1 Data Discovery, not just Dashboards
• Easy for users to author,
customize and share
• Interactive exploration &
visually filter results
• Quickly identify
anomalies and outliers
with large or in-motion
datasets
• Rich palette of
visualizations for static
and time series data
#2 Streaming Data Visualization
Database Distributed or
Hybrid Database
In-Memory
Database
Streaming Analytics
Faster Speed, Faster Insights
Data at Rest
Limitations of Traditional BI
Database Distributed or
Hybrid Database
In-Memory
Database
Streaming Analytics
Data at Rest
Streaming Data Discovery
Database Distributed or
Hybrid Database
In-Memory
Database
Streaming Analytics
Streaming Data Discovery
Database Distributed or
Hybrid Database
In-Memory
Database
Streaming Data
Alert! Steam turbine stress
level over threshold
How does this compare to
intra-day? What is mean
time to failure?
What is likely to occur?
Act! Schedule shut down
Streaming Data Visualization
• Connect directly to data in motion
• Hosted IoT platforms
• Complex Event Processing & MQ
• Data model optimized for both caching and persistence
• High density visuals with rendering in milliseconds
Monitor Analyze Take Action
#3 Time Series Data
• Traditional BI only looks at buckets of
time
• Day, week, month, year
• Sensor data is a continuous and has
different requirements
• Second, millisecond, nanosecond
• Time windows
• Time slices
• Playback
• Complete situational awareness
• Now (streaming)
• Intra-day
• Historic
#4 Predictive & Advanced Analytics
• Enrich streaming OT data
with “what is likely to occur”
• Predictive models based on
historic data patterns
• Many use cases in IoT (e.g.
predictive maintenance,
smart logistics, clinical
pattern detection etc.)
• Leverage commercial and
open source solutions
We Want to Predict the Future for
Equipment
Modeled and
transformed
for analysis
#5 Data Preparation
• Sensor and machine data often in multi-structured format
• Need to transform, enrich and prepare data
• Almost no metadata
• For example, wave form visualization from JSON arrays
stored in MongoDB and streaming
Log Files
HTML,
XML JSON
PDFs
#6 Real-Time Geospatial & Location
• Real-time (stream) plotting
• Street-level geo maps or
custom SVG files
• Time-series playback
HealthcareRetail Logistics
Utilities
6 Requirements for IoT Visualization
Visual Data Discovery
Streaming Data Visualization
Time Series Data
Predictive & Advanced Analytics
Data Preparation
Real-time Geospatial & Location
New
Analytic
Approach
Required
The Next Wave of Business
Transformation
Source: Industrial Analytics: The Next Wave of Business Transformation
Gartner, March 2014
ROI for Real-Time Data Visualization
Growth in New Pipeline
Increase in Cash Generated
Greater Operational Cost Reduction
“Real-Time Data Visualization” October 2013
Customer Use Case
• Fortune 500 oil & gas exploration and production company
• Moving to real-time streaming visualization
• From 24 hour latency moving data overnight from OSIsoft Pi to Oracle Warehouse
feeding dashboards
• To real-time, streaming data discovery connecting directly to OSIsoft Pi server
• Initial goal is to reduce steam cost by 3-5% ($M) in year 1
Pi Server
Smart Meter Dashboard
Streaming data from electric smart meters
Hospital Emergency Ward – Bird’s Eye View
Streaming data from patient monitoring system
Workbook

Advanced Analytics for Any Data at Real-Time Speed

  • 1.
    ADVANCED ANALYTICS FOR ANYDATA AT REAL-TIME SPEED Dan Potter Chief Marketing Officer, Datawatch Corporation
  • 2.
    How Many Sensors AreIn Your Pocket?
  • 3.
    NASDAQ: DWCH Pioneer inreal-time visual data discovery and self-service data preparation Global operations and support  US, UK, Germany, France, Australia, Singapore, Philippines Extensive global customer base  93 of the Fortune 100  12 of the 15 largest financial institutions Embedded and resold by leading vendors About Datawatch
  • 4.
    IoT and IndustrialAnalytics 50 billion devices will be interconnected by 2020 - Cisco By 2017, over 50% of analytics implementations will make use of event data streams generated from instrumented machines, applications and/or individuals - Gartner Real-time visualization leads to more opportunities, greater output, and lower costs - Aberdeen Group IoT companies attracted more than $1 billion in venture capital - Forbes
  • 5.
  • 6.
    IoT Platform Requirements Stream Processing VisualData Discovery Real-Time Transport Sensor Data OT Data IT Data
  • 7.
    Visualization for IoT Data Access Visual Discovery Monitor andAlerts Data Preparation Predictive Analytics Take Action
  • 8.
    6 Requirements forIoT Visualization Data Discovery Streaming Data Visualization Time Series Data Predictive & Advanced Analytics Data Preparation Real-time Geospatial & Location
  • 9.
    #1 Data Discovery,not just Dashboards • Easy for users to author, customize and share • Interactive exploration & visually filter results • Quickly identify anomalies and outliers with large or in-motion datasets • Rich palette of visualizations for static and time series data
  • 10.
    #2 Streaming DataVisualization Database Distributed or Hybrid Database In-Memory Database Streaming Analytics Faster Speed, Faster Insights
  • 11.
    Data at Rest Limitationsof Traditional BI Database Distributed or Hybrid Database In-Memory Database Streaming Analytics
  • 12.
    Data at Rest StreamingData Discovery Database Distributed or Hybrid Database In-Memory Database Streaming Analytics
  • 13.
    Streaming Data Discovery DatabaseDistributed or Hybrid Database In-Memory Database Streaming Data Alert! Steam turbine stress level over threshold How does this compare to intra-day? What is mean time to failure? What is likely to occur? Act! Schedule shut down
  • 14.
    Streaming Data Visualization •Connect directly to data in motion • Hosted IoT platforms • Complex Event Processing & MQ • Data model optimized for both caching and persistence • High density visuals with rendering in milliseconds Monitor Analyze Take Action
  • 15.
    #3 Time SeriesData • Traditional BI only looks at buckets of time • Day, week, month, year • Sensor data is a continuous and has different requirements • Second, millisecond, nanosecond • Time windows • Time slices • Playback • Complete situational awareness • Now (streaming) • Intra-day • Historic
  • 16.
    #4 Predictive &Advanced Analytics • Enrich streaming OT data with “what is likely to occur” • Predictive models based on historic data patterns • Many use cases in IoT (e.g. predictive maintenance, smart logistics, clinical pattern detection etc.) • Leverage commercial and open source solutions
  • 17.
    We Want toPredict the Future for Equipment
  • 18.
    Modeled and transformed for analysis #5Data Preparation • Sensor and machine data often in multi-structured format • Need to transform, enrich and prepare data • Almost no metadata • For example, wave form visualization from JSON arrays stored in MongoDB and streaming Log Files HTML, XML JSON PDFs
  • 19.
    #6 Real-Time Geospatial& Location • Real-time (stream) plotting • Street-level geo maps or custom SVG files • Time-series playback HealthcareRetail Logistics Utilities
  • 20.
    6 Requirements forIoT Visualization Visual Data Discovery Streaming Data Visualization Time Series Data Predictive & Advanced Analytics Data Preparation Real-time Geospatial & Location New Analytic Approach Required
  • 21.
    The Next Waveof Business Transformation Source: Industrial Analytics: The Next Wave of Business Transformation Gartner, March 2014
  • 22.
    ROI for Real-TimeData Visualization Growth in New Pipeline Increase in Cash Generated Greater Operational Cost Reduction “Real-Time Data Visualization” October 2013
  • 23.
    Customer Use Case •Fortune 500 oil & gas exploration and production company • Moving to real-time streaming visualization • From 24 hour latency moving data overnight from OSIsoft Pi to Oracle Warehouse feeding dashboards • To real-time, streaming data discovery connecting directly to OSIsoft Pi server • Initial goal is to reduce steam cost by 3-5% ($M) in year 1 Pi Server
  • 24.
    Smart Meter Dashboard Streamingdata from electric smart meters
  • 26.
    Hospital Emergency Ward– Bird’s Eye View Streaming data from patient monitoring system
  • 27.

Editor's Notes

  • #5 IoT companies attracted more than $1 billion in venture capital in 2013
  • #9 Time Series Conflation Streaming visualization Alerts Integration with CEP and event processing JSON file formats MQTT Scada and Pi Server – OSIsoft Predictive – R and Pytohn
  • #14 Would you cross the street based on yesterday’s news
  • #15 An in-memory, OLAP-based StreamCube is associated with each graphical display object. The system processes new data as it arrives, selects the subset of important data, recalculates the relevant sections of the model and refreshes the associated parts of the display immediately. The parts of the model and the display that are not affected by the new data are not touched. This is faster and more efficient than conventional data visualization tools that operate on batch-loaded snapshots of data, run less frequently, and then recalculate the model and rebuild the display for each iteration.
  • #17 Can be used in combination with time series transforms
  • #21 Time Series Conflation Streaming visualization Alerts Integration with CEP and event processing JSON file formats MQTT Scada and Pi Server – OSIsoft Predictive – R and Pytohn
  • #23 Source: Aberdeen Group, “Real-Time Data Visualization,” October 2013.