• Share
  • Email
  • Embed
  • Like
  • Private Content
Real-Time Sensors Data Webinar | SQLstream | July 2013 Series
 

Real-Time Sensors Data Webinar | SQLstream | July 2013 Series

on

  • 376 views

The Internet of Things relies on networks of connected devices at unprecedented scale and complexity. Maximizing the opportunity requires organizations to collect and process high velocity sensor data ...

The Internet of Things relies on networks of connected devices at unprecedented scale and complexity. Maximizing the opportunity requires organizations to collect and process high velocity sensor data in real-time. This Webinar presentation shows how real-time operational intelligence is shaping the Internet of Everything, transforming log file and sensor machine data streams into actionable, real-time intelligence. Use cases from Intelligent Transportation, Telematics and M2M are presented.

Statistics

Views

Total Views
376
Views on SlideShare
376
Embed Views
0

Actions

Likes
0
Downloads
20
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Presentation flow:#1 – What is operational intelligence – the bridging the chasm#2 – Analytics -> prescriptive analytics -> action#3 – Drivers (analyst validation)#4 – Machine Data where and what is it?#5 – And it’s growing#6 – What industries is it growing in most?
  • Many customers don’t even realize that they have an application that requires the capabilities of a real-time analytic platform like SQLstream. Since streaming, continuous, real-time information is such a new concept for customers, they don’t realize they need a different technology to address it. Most companies are doing significant retrofits of existing database technologies (like Oracle, SQL-Server, Hadoop, in-memory databases like SAP HANA, column-store databases, etc.) to try to handle:Much more frequent updatesQueries that the continually run over-and-over again to see if anything has changedSLA’s and or narrow time-action windows where they need to respond to an event that has happenedThe massive inflows of real-time informationThe reality is that they need a real-time data management platform to handle this vs. trying to retrofit existing technologies that were never meant for real-time. A perfect analogy is a mechanic trying to use a wrench to remove a spark plug. The spark plug has a hex head, and wrenches work fantastically on hex-head applications. However, removing/installing a spark plug requires a socket, as it has specific access requirements that just can’t be addressed by a wrench. A mechanic needs both tools, and one tool is NOT a replacement for the other; they’re complementary. In the same way, SQLstream does NOT replace existing historical database systems (Hadoop, Oracle, etc.), rather, we complement them.** If there are customer questions on this slide and/or additional information is needed, please refer to the “SQLstream vs. Traditional Databases” slide in the competitive section of this presentation.
  • SQLstream enables the powerful capability of replaying historical events saved to the data warehouse or Hadoop. These events are replayed through SQLstream at high speed through the same SQL algorithms and business logic used on the real-time feeds. It enables the ability to test predictive algorithms with past events to understand behavior and validate that actions are properly triggered. It also enables powerful measurement and reporting of real-time exceptions on any historical data set.This is a capability unique to SQLstream. SQLstream enables this capability by using our standard database adapter to “read” rows of data from the warehouse in timestamp order over a defined time period. It enables businesses to back-test the inferred rules for predicting fraud, identifying a potential buyer, predicting a traffic jam or understanding when a machine will failby writing SQL queries to run against the historical data. The results of the queries show whether A, B and C are correlated with the predicted outcome D with a confidence factor of say >95%. The beauty of the SQLstream approach is that we can reuse those back-testing SQL queries to perform the same analysis, but now predictively instead of forensically, running them against the live streaming data. When they detect A, B and C they then generate an output on D with a 95% confidence factor. The predictive algorithm can now It might represent fraud, a good prospect for a product or a potential traffic jam ahead. You get the idea.
  • Streaming data:#1 Streaming Data Analytics#2 Architecture#3 Platform Architecture#4 SQL#5 Product Portfolio

Real-Time Sensors Data Webinar | SQLstream | July 2013 Series Real-Time Sensors Data Webinar | SQLstream | July 2013 Series Presentation Transcript

  • If you haven’t dialed into the audio portion, please do so now: U.S.A +1 (646) 307-1721| 789-157-692 TH ANK YOU FOR J OINING! TH E WEBIN AR IS A BOUT TO START Listen to Your Sensors: a Tale of Managing Large-scale, Intelligent Sensor Networks in Real-time
  • | 2Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤  Explain real-time Big Data and Operational Intelligence ¤  The principles of streaming data management ¤  Share our thoughts, experiences and use cases ¤  Audience Q&A PROGRAM MISSION
  • | 3Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤  Introduction (5 min) ¤  Presentation (35 min) o  An Introduction to Operational Intelligence o  Solution Architecture for Real-time Sensor Analytics o  Streaming Big Data management o  Industry Use Cases o  Demonstration (Remote sensor monitoring) o  The Total Cost of Performance ¤  Q | A (20 min) JULY 16 AGENDA
  • | 4Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤  July 9 2013 |10:00am PST- recording available upon request Analytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence ¤  July 16 2013 |11:00am PST Listen to your Sensors: A Tale of Managing Large Scale Sensor Networks in Real-time ¤  July 23 2013 |11:00am PST Predict and Avert: Using Log File Data to Prevent Cybersecurity and Fraud Attacks in Real-time ¤  July 30 2013 |10:00am PST No more CPR for your CDRs: Meet Real-time Traffic Utilization, Billing and Fraud Detection The Operational Intelligence Series
  • | 5Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤  25 years of experience in database and software applications ¤  Senior positions with Oracle, Hyperion Solutions, Information Builders and Algebraix Data ¤  Recent guest speaker at 2013 Sensors Expo Today’s Presenter: Glenn Hout SQLstream VP for the Americas
  • | 6Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com THE INTERNET OF EVERYTHING Managing Large-scale Connected Sensor Networ ks TECHNOLOGY ¤  OPEX & CAPEX ¤  Experience ¤  Loyalty BUSINESS CASE ¤  Velocity ¤  Volume ¤  Unstructured REAL-TIME BIG DATATHE NEXT DIGITAL AGE REAL-TIME OPERATIONAL INTELLIGENCE 50 Billion Connected Devices and Sensors by 2020 (Cisco IBSG) ¤  Wireless ¤  IPv6 Networks ¤  Smart Energy
  • | 7Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ABOUT SQLSTREAM facts o  Launched 2009 o  Over 1.5M lines of code o  Multiple deployments across many industries capabilities o  Unstructured and structured data o  Accelerates and extends Hadoop & RDBMS o  Not limited to SQL innovations o  Only true streaming data management platform o  Only true standard SQL streaming engine o  Five patents for stream processing ü  Real-time Operational Intelligence ü  High-velocity machine data. ü  Streaming Big Data Management Platform
  • ¤  TWEET: during and after the webinar, please use #RTSensorData for live discussions ¤  DIRECT QUESTIONS: please use the box to the right of your screen ¤  RECORDINGS: an edited version of the webinar recording will be emailed upon request
  • Operational Intelligence
  • | 10Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com The Internet-of-Things Explosion | SOURCES E ve r y t h i n g i s b e c o m i n g i n s t r u m e n t e d ; va r i e t y ENVIRONMENTAL TRANSPORTATION NETWORKS Environmental Monitoring Location-based services Machine-to-Machine Smart Grid Cars as Sensors Logistics
  • | 11Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com OPERATIONAL INTELLIGENCE Integrating Operations and Analytics in Real-time As we move toward a real-time business environment, the capability to process data flows swiftly and flexibly will become increasingly important. SQLstream leads the industry in this kind of capability. ”Robin Bloor Chief Analyst for Bloor Group Business Intelligence Operations Real-time Operational Intelligence Continuous monitoring and analytics Improve decision-making Automate operational processes Environment Smart Grid Transportation Telematics M2M Remote Monitoring ”
  • | 12Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com The Information Value Chain What is happening? What might happen? What just happened? Make stuff happen!
  • | 13Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com COLLECTION: Massive numbers of sensors o  Thousands, tens-of-thousands, hundreds-of-thousands, millions+ of sources o  Creation of multiple siloes SCALE: Continuous “fire-hose” o  Massive data volumes and velocities o  Information overload PROCESSING: Actionable intelligence o  Data “decision” value can drop exponentially with passing seconds o  Data velocity and rate-of-change critical for decisions o  Lack of real-time processing Big Data Explosion| CHALLENGES A n a l y t i c p ro c e s s i n g a t m a s s i ve s c a l e BIG DATA MACHINE DATA HIGH-VELOCITY, LOW-LATENCY INTELLIGENCE
  • | 14Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤ Capabilities o  Health monitoring of key vehicle systems o  Frequent monitoring when “exception” system events occur o  Real-time “panic” alerts ¤ Objectives o  Reduce vehicle “walkaway” events by 50% o  Improve customer satisfaction o  Reduce warranty costs o  Future: Potential “paid for” service REAL-TIME INTEGRATED VEHICLE HEALTH MANAGEMENT Warning! Battery draining fast. Please check for lights left on.
  • | 15Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤ Real-time vehicle diagnostics & alerting o  Pre-emptive vehicle service o  Critical event monitoring and alerting ¤ Asset tracking for fleet management o  Track vehicles, shipments and transit ETA ¤ Real-time driving log o  Safety, compliance and alerting ¤ Dynamic road tolling o  Assess tolls based on weight, miles, roads used VEHICLE TELEMATICS & HEALTH MONITORING Log
  • | 16Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤ Insurance Programs o  Breaks on insurance for good drivers •  Immediate “journey” reports to smart phones o  Real-time analytics and alerting •  Safety, compliance and real-time monitoring ¤ Geofencing o  Alerts to smartphones when drivers travel outside prescribed boundaries INSURANCE & SENSOR MONITORING Driving Analysis: ……… ………………. ………….
  • | 17Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤  Real-time health diagnostics, monitoring and alerting ¤  Asset tracking and location-based services ¤  Location- and activity-based advertising ¤  Safety, regulatory and compliance ¤  Customer loyalty solutions ¤  Fleet management solutions ¤  Usage-based applications: insurance, tolling, traffic, etc. ¤  Emergency services and responder alerting REAL-TIME SENSOR APPLICATIONS
  • | 18Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Information coming continuously or in irregular intervals o  Sensor data, log file data, health information (from cars, machinery, hardware, people), etc. Anything that requires immediate attention o  Alerts that must be acted on now, not in minutes, hours or days Any process that struggles with batch window latencies o  Service level agreements (SLA’s) with narrow time-action windows o  Batch queries that are continually re-run o  Data cleansing, transformation and loading (ETL) Massive scale o  Information inflows so large that they must be continuously processed REAL-TIME APPLICATIONS | INDICATORS
  • | 19Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Capability Real-Time System Traditional Database / Hadoop Analytic Processing •  Continuous queries •  Incremental calculations •  Batch queries •  Expensive, batch recalculations Latency from event to action •  Milliseconds to seconds •  Tens-of-minutes, hours, days •  Significant latency Rate of Change •  Monitor data velocity and complex rates-of-change •  Static snapshot of point-in-time Scalability & Hardware Requirements •  Millions to tens-of-millions of events/second •  Minimal hardware reqt’s •  Massive data inflow and continual recalculation limits analytic depth •  Significant hardware reqt’s Iterative Analytics •  Cascading analytics within and across events - pipelining •  Requires myriad views, temporary tables and custom application code QUERY-THEN-STORE VS. STORE-THEN-QUERY
  • | 20Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Category Operational Intelligence Business Intelligence Vehicle Diagnostics Is my vehicle about to fail? How many vehicles failed in the last 6 months in the state of New York? Sensors ALERT - The airbag just deployed! What is the current location, vehicle and driver information? How many vehicles had airbags that deployed in the past 2 weeks by vehicle type? Telematics & Asset Tracking Where is the vehicle right now? Where has it been and how is it moving on current journey? What was the average speed of all vehicles on various road segments on prior journeys? Advertising What is a relevant ad to place based on current location, current activity and historical patterns? What is a relevant ad to place based on historical patterns? Security Is an intrusion attempt underway? Where have intrusions happened in the past and from what sources? Fraud/risk Is the current transaction fraudulent? How many fraudulent transactions occurred at online electronics retailers during the past 3 months? REAL-TIME ANALYTICS VS. HISTORICAL ANALYTICS
  • | 21Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com REAL-TIME, CONTINUOUS OPERATIONAL INTELLIGENCE Real-time alerts, action and visualizations Enhance real-time data by joining it with historical information Persist both detail and aggregate data to historical archives
  • | 22Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com “REPLAY” EVENTS TO DEVELOP PREDICTIVE ALGORITHMS Data Warehouse DATABASEADAPTER Replay events at high-speed Test & refine predictive algorithms Measure & report results and real-time exceptions
  • | 23Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com REAL-TIME OPERATIONAL INTELLIGENCE C o m p l i m e n t a r y w i t h E x i s t i n g S o l u t i o n s Sensor Feeds Data Transmission Data Collection Lightweight Agents (listeners) Existing Archival Database(s)
  • | 24Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com ¤  Dashboards o  Simple drag-and-drop interface o  Multiple key performance indicators on a single panel o  Real-time, continuous charting capability on any metric ¤  Visualizations o  100% web-based o  Variety of charting types and customizable alerts o  Smart-phone access o  Stunning graphics capabilities POWERFUL DASHBOARDING & VISUALIZATIONS
  • | 25Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Complementary o  Doesn’t replace business intelligence solutions, but provides powerful, complementary operational intelligence capabilities Massive scalability o  Ability to provide analytics and alerting on massive volumes of sensor feeds with minimal infrastructure Low Total Cost of Ownership (TCO) o  Minimal infrastructure requirements and rapid time-to-value o  Utilize existing business intelligence skill sets for real-time data SUMMARY
  • Streaming Data Management
  • | 27Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com High-velocity Big Data Analytics Historical queries and data enrichment Storing valuable derived streams for future access OperationalIntelligence Logs Sensors GPS Networks Social media RFIDs Servers Telecom Smart grid Oil & Gas Manufacturin g Logistics M2M Telematics Retail Internet Banking Data centers Automotive ¤  Continuous Queries over Sliding Time Windows ¤  Analysis and Integration of Unstructured and Structured data ¤  Prescriptive Analytics drives Automated Actions
  • | 28Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Real-time Arc hitecture Streaming Analysis and Integration for Infinite Flows of Unstructured Data in Real Time Streaming Agent & Adapter Layer + JDBC API Hadoop Streaming Query Planner & Optimizer for MPP Execution SQL Developer Tools Platform Administration Streaming SQL Real-time Applications Real-time Dashboards & Visualization Logs Sensors GPS Networks Social MediaServers M2M Telematics Impala SQL HBase HDFS / MR Hadoop for Stream Persistence, Enrichment & Replay (Optional) Any external data warehouse, operational system and enterprise platform
  • | 29Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com CLEANING & FILTERING STREAMING ANALYTICS STREAMING AGGREGATION CONTINUOUS INTEGRATION Geospatial ApplicationsSecurity Monitoring Real-time Dashboards Health Monitoring QoS and QoE AN OPERATIONAL INTELLIGENCE PLATFORM Logs SensorsGPS Networks Social MediaServersM2M Telematics
  • | 30Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com The SQLstream s-Streaming Product Portfolio s-Server Data Management Platform for Streaming Big Data s-Analyzer Real-Time Visualization for Streaming Operational Intelligence s-Transport Geo-Analytics for Location-based Applications s-Visualizer Advanced Visualization s-Cloud s-Server EC2 AMI Deployment s-Studio Developer&AdminConsole
  • Total Cost of Performance
  • | 32Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com R E C O R D S P E R S E C O N D Total Cost Of Performance (total COP) The H igh -Velocity, Low -L atenc y Tipping Point for Big Data Patterns Trends MiningConnections Searches Inventory ReportsStatistics Billing SOCIAL E-COMM SECURITY TELEMATICS TELECOM Trading Advertising AlertsDetection Signal Intelligence TOTALCOST
  • | 33Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Intelligence TELECOM Patterns Trends MiningConnections Searches Inventory ReportsStatistics Billing Trading Advertising AlertsDetection Signal SOCIAL E-COMM SECURITY TELEMATICS R E C O R D S P E R S E C O N D TOTALCOST Total Cost Of Performance (total COP) The H igh -Velocity, Low -L atenc y Tipping Point for Big Data
  • Glenn Hout Email | glenn.hout@sqlstream.com Phone | 650.343.0864 Website | www.sqlstream.com Upcoming events | www.sqlstream.com/webinars/ Q | A
  • | 35Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com MACHINE-TO-MACHINE (M2M) APPLICATIONS Perfor mance, Analytics and Predictive Health Monitoring ¤  Transportation Asset tracking and logistics and consumer Journey Time from GPS data ¤  Telematics Vehicle health and driver insurance from vehicle sensor data ¤  Healthcare Remote health monitoring applications from low power micro-sensors ¤  M2M Data Feed Monetization Streaming data aggregation with subscription-based analytics ü Improve uptime ü Avoid failures ü One platform
  • | 36Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com STREAMING SQL Example: Compute Average vehicle speed across any subset of network over rolling time windows from GPS events In the example shown below, sensor data arriving in the “RoadPositionInfo” stream is used to calculate average speed per zone over rolling one, five and ten minute windows: CREATE OR REPLACE VIEW "SpeedZoneStats" DESCRIPTION ‘Rolling averages for multiple windows partitioned by zone' AS SELECT STREAM "zone", -- zone id "segmentid", -- parent road segment "speedlimit", -- speed limit for zone AVG("Speed")OVER last1Min AS "avgSpeed1", -- 1-min running average AVG("Speed")OVER last5Min AS "avgSpeed5", -- 5-min running average AVG("Speed")OVER last10Min AS "avgSpeed10" -- 10-mn running average FROM "RoadPositionInfo" WINDOW last1Min AS (PARTITION BY "zone" RANGE INTERVAL '1' MINUTE PRECEDING), last5Min AS (PARTITION BY "zone" RANGE INTERVAL '5' MINUTE PRECEDING), last10Min AS (PARTITION BY "zone" RANGE INTERVAL '10' MINUTE PRECEDING); The output of the application is a continuous stream of exception alerts based on pre-defined conditions and rolling averages of speed information per road segment over various time windows Multiple analytics queries can execute in parallel over any number of different data streams, even36Copyright © 2012 Proprietary information of SQLstream Inc. All rights reserved