Operational Intelligence Webinar | SQLstream | July 2013 Series
 

Like this? Share it with your network

Share

Operational Intelligence Webinar | SQLstream | July 2013 Series

on

  • 1,074 views

As the speed of business accelerates, and more data is created every second, gaining a competitive edge relies on transforming massive volumes of log file and other machine-generated data into ...

As the speed of business accelerates, and more data is created every second, gaining a competitive edge relies on transforming massive volumes of log file and other machine-generated data into actionable intelligence in real-time.

Leading organizations are leveraging SQLstream's real-time operational intelligence to make smarter decisions faster, and at price they can afford.

Statistics

Views

Total Views
1,074
Views on SlideShare
1,074
Embed Views
0

Actions

Likes
1
Downloads
31
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • SQLstream was founded in 2003 with a vision of transforming the way streaming service, system and sensor data are processed. We have a very experienced team, headquartered in San Francisco with people also in Chicago, San Diego and in the UK. We recently accepted institutional money from Fontinalis Partners, a Venture Capital firm with Private Equity roots and limited partners comprising some of the largest industrial families around the world. Their General Partners include Bill Ford, Executive Chairman of Ford Motor Company. The company left stealth mode in 2008 after 1.2M lines of code to deliver the world’s first SQL standards compliant stream-to-business platform. More on that later. We have 7 patents of which 3 are already granted, and 4 are pending. It has taken over 100 Engineering years of effort to design and develop our platform, and we have secured some marquee customers along the way, most of which are leaders in their industries.
  • Presentation flow: #1 – What is operational intelligence – the bridging the chasm #2 – Analytics -> prescriptive analytics -> action #3 – Drivers (analyst validation) #4 – Machine Data where and what is it? #5 – And it’s growing #6 – What industries is it growing in most?
  • ----- Meeting Notes (4/8/13 09:57) ----- BI is all rear mirror based. State examples. Changing speed limits before gridlock. Detecting dropped calls before they drop. Detecting authentication issues before business is lost. ----- Meeting Notes (4/9/13 15:04) ----- cloud traffic customer loyalty
  • GPS, RFID, Web Servers, Email, Messaging, Mobile, Telephony, Databases, Telematics, Security, Storage Machine- data is one of the fastest growing, most complex and most valuable segments of big data
  • Industries: Telecommunications, Smart Grid, Oil & Natural Gas, Manufacturing, Logistics, Machine-to-Machine, Telematics, Automotive Retail, Banking, Internet, Data Center Data Sources: Logs, Sensors, GPS, Networks, Servers, RFIDs, Social Media
  • The Big Data heat map has been compiled from the following sources: Gartner: “Market Trends: Big Data Opportunities in Vertical Industries” Wikibon Forrester IDC
  • Streaming data: #1 Streaming Data Analytics #2 Architecture #3 Platform Architecture #4 SQL #5 Product Portfolio
  • There are an ever growing number of data sources that an enterprise needs to harness in order to create deliver the integrated information and analyses it needs. There are many vertical silo solutions that allow a particular source of information to be processed and shard with other applications in real-time. The problem is that the same sources of data need to feed any number of applications and each application needs to combine data from many separate sources – all in real-time. So for N source and M destinations you have an N-times-M complexity problem, in terms of transforming data from each source to each destination. This becomes a major headache to manage and maintain. Every new source of data added or application added increases this pain greatly. Eran Wagner
  • Put simply, there is a new real-time data challenge facing many enterprises today. The data volumes are exploding exponentially – making it too costly to analyze with conventional technologies where you have to store all of the data, even if most of the data might have a very limited ‘shelf life’. The costs come from needing specialized data warehousing systems to handle such large and ever growing data volumes, and those license fees are almost always based on the volume of data stored. So why store everything if you are only really concerned with the results of analyses? At the same time, businesses are having to become nimbler and more agile. They need to consume and analyze data faster than their competitors, and fast enough to hold the attention of their customers while they are interacting with their product, service, systems or personnel. Finally, automating such real-time processes is hard to do in a scalable and effective manner.

Operational Intelligence Webinar | SQLstream | July 2013 Series Presentation Transcript

  • 1. Copyright © SQLstream Inc. Analytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence The SQLstream July 2013 Webinar series
  • 2. | 2Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com • Deliver webcasts that explain real-time Big Data and the world of Operational Intelligence in real terms • Introduction to streaming data management for real-time, high velocity, low latency applications • Share our thoughts, experiences, industry use cases and examples MISSION
  • 3. | 3Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com • July 9 2013 |10:00am PST Analytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence • July 16 2013 |11:00am PST Listen to your Sensors: A Tale of Managing Large Scale Sensor Networks in Real-time • July 23 2013 |11:00am PST Predict and Avert: Using Log File Data to Prevent Cybersecurity and Fraud Attacks in Real-time • July 30 2013 |10:00am PST No more CPR for your CDRs: Meet Real-time Traffic Utilization, Billing and Fraud Detection The Operational Intelligence Series
  • 4. | 4Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Analytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence  As the speed of business accelerates, more data is created every second.  Gaining a competitive edge relies on transforming massive volumes of log file and other machine-generated data into actionable intelligence in real-time.  Leading organizations are leveraging real-time operational intelligence to make smarter decisions faster, and at price they can afford.
  • 5. | 5Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com SQLstream, Inc. History  Launched 2009  Over 1.5M lines of code  Multiple deployments across many industries, with top real-world benchmarks Features  Supports all forms of unstructured and structured data  Accelerates and extends Hadoop & RDBMS  Not limited to SQL Key innovations  Only true streaming data management platform  Only true standard SQL streaming engine  Covered by 5 broad patents for stream processing A streaming data management platform for real-time Operational Intelligence from high-velocity Big Data.
  • 6. Operational Intelligence
  • 7. | 7Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Bridging The Chasm As we move toward a real-time business environment, the capability to process data flows swiftly and flexibly will become increasingly important. SQLstream leads the industry in this kind of capability. “ ”Robin Bloor Chief Analyst for Bloor Group Aberdeen’s research has shown that Best-in-Class organizations are demanding access to actionable intelligence faster than ever. This is precisely the growing demand that SQLstream is meeting with their Streaming Big Data Engine, while continuing to bring other attractive features like full Hadoop integration. “ ” Nathaniel Rowe Leading Analyst for Aberdeen Group Business Intelligence Post-hoc Analysis Data Warehousing Strategic insights Operations Transaction Processing Machine Data Everyday business Operational Intelligence integrates Operations and BI Operational Intelligence Optimizes tactical decisions from real-time actionable insights Combines operations data with BI data continuously Provides Real-time integrated view of the business and operations Security Compliance Fraud Quality Promotion Advertising Cross-selling Security Compliance Fraud Quality Promotion Advertising Cross-selling
  • 8. | 8Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com The Information Value Chain What is happening? What might happen? What just happened? Make stuff happen!
  • 9. | 9Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Drivers For Operational Intelligence* Source: Ventana Research Operational Intelligence Benchmark Research, 2013  Analyze complex relationships “Most organizations do not have technologies that can analyze complex relationships across multiple source systems in real- time, but most are planning to deploy”  Low latency is essential “No latency can be tolerated for security and network management for example”  Integration is the missing link “Most Business Intelligence systems are not integrated with operational intelligence data”  Specialized tools perform better “When organizations use specialized tools they are much more satisfied with the Operational Intelligence initiatives” Manage riskManage risk Manage performance Manage performance Comply with regulations Comply with regulations Identify opps. for improvement Identify opps. for improvement 59% 59% 58% 58% 54% *Importance of goals in current or planned deployment of technology
  • 10. | 10Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Machine-generated Big Data Explosion High volume, high velocity, structured and unstructured data from software platforms, applications and systems GPSGPS TelematicsTelematics IP Networks, VideoIP Networks, Video Servers, Social Media, SecurityServers, Social Media, Security Servers, Applications, Storage NetworksServers, Applications, Storage Networks Machine-generated data will increaseMachine-generated data will increase to 42% of all datato 42% of all data by 2020, up fromby 2020, up from 11% in 2005.11% in 2005. ““The Digital Universe in 2020”The Digital Universe in 2020” IDCIDC
  • 11. | 11Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Connecting Machine Data, Connecting The World DATA SOURCESDATA SOURCES INDUSTRIESINDUSTRIES
  • 12. | 12Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Machine Data Where is the intelligence? TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342Transaction Log Details Web Server Logs CDR Records Smartphone GPS Updates Twitter {"created_at:Thu Feb 17 15:30:55 +0000 2013,id:304612775055998976,id_str:304612775055998976,text:@MyServiceProvider today sucks, keeps dropped!,source:u006ca href=http:www.url.com rel=nofollow,followers_count:147,friends_count:10142, location: San Francisco, time_zone: Pacific, geo_enabled:true, location:u00dcT: -6.1987552,106.8661953, screen_name:APerson <id>1597831220</id><deviceid>0198873465</deviceid><lat>lat=47.643957</lat><lon>lon= -122.3269</lon><time>2013-02-17T15:37:26Z</time><bearing>223.4535</bearing> <id>1597865781</id><deviceid>0198873465</deviceid><lat>lat=47.645982</lat><lon>lon=- 122.327500</lon><time>2013-02-17T15:37:26Z</time><bearing>200.6138</bearing> <id>1597940125</id><deviceid>0198873465</deviceid><lat>lat=47.647381</lat><lon>lon=- 122.326501</lon><time>2013-02-17T15:37:26Z</time><bearing>87.4357</bearing> [Sun Feb 17 15:30:49 2013] [notice] srv-sfo-08 caught SIGTERM, shutting down [Sun Feb 17 15:30:49 2013] [notice] Apache/2.2.21 -- resuming normal operations TERMINATE,ctl09gsx,01299796304,GMT-08:00,02-17-13,15:21:00,9,387,64ms,02-17-13,15:30:55,0005, IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60, 234.234.60.75,65678,411,399,SIP,SANFRANCISCO,0x4B1698,0x0005E,0x49768,4157588342,0198873465 TimestampTimestamp TimestampTimestamp TimestampTimestamp TimestampTimestamp TimestampTimestamp Mobile #Mobile #CustomerCustomer Mobile #Mobile # Device IDDevice IDTerm Reason Term Reason Device IDDevice ID LocationLocation LocationLocation Service Provider Service Provider Fail CodeFail Code ServerServer
  • 13. | 13Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Real-time Big Data Heat Map Hot right now Hot, growing fast (< 2 years) Growing fast (2 – 5 years) 5 – 10 years > 10 years
  • 14. Total Cost of Performance
  • 15. | 15Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com RECORDS PER SECOND LATEN CY Total Cost Of Performance (total COP) The High-Velocity, Low-Latency Tipping Point for Big Data Patterns Trends MiningConnections Searches Inventory ReportsStatistics Billing SOCIAL E-COMM SECURIT Y TELEMATI CS TELECOM Trading Advertising AlertsDetection Signal Intelligence TOTALCOST
  • 16. | 16Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Intelligence TELECOM Patterns Trends MiningConnections Searches Inventory ReportsStatistics Billing Trading Advertising AlertsDetection Signal SOCIAL E-COMM SECURIT Y TELEMATI CS RECORDS PER SECOND TOTALCOST LATEN C Y Total Cost Of Performance (total COP) The High-Velocity, Low-Latency Tipping Point for Big Data
  • 17. Streaming Data Management
  • 18. | 18Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com High-velocity Big Data Analytics Historical queries and data enrichment Storing valuable derived streams for future access OperationalIntelligence  Continuous Queries over Sliding Time Windows  Analysis and Integration of Unstructured and Structured data  Prescriptive Analytics drives Automated Actions
  • 19. | 19Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Real-time Architecture Streaming Analysis and Integration for Infinite Flows of Unstructured Data in Real Time Streaming Agent & Adapter Layer + JDBC API Hadoop Streaming Query Planner & Optimizer for MPP Execution SQL Developer Tools Platform Administration Streaming SQL Real-time Applications Real-time Dashboards & Visualization Impala SQL HBase HDFS / MR Hadoop for Stream Persistence, Enrichment & Replay (Optional) Any external data warehouse, operational system and enterprise platform
  • 20. | 20Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com CLEANING & FILTERINGCLEANING & FILTERING STREAMING ANALYTICSSTREAMING ANALYTICS STREAMINGSTREAMING AGGREGATIONAGGREGATION CONTINUOUSCONTINUOUS INTEGRATIONINTEGRATION Internet ofInternet of EverythingEverythingFraudFraud PreventionPrevention NetworkNetwork MonitoringMonitoring CybersecurityCybersecurity QoS andQoS and QoEQoE An Operational Intelligence Platform
  • 21. | 21Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com SELECT STREAM ROWTIME, url, numErrorsLastMinute FROM ( SELECT STREAM ROWTIME, url, numErrorsLastMinute, AVG(numErrorsLastMinute) OVER lastMinute AS avgErrorsPerMinute, STDDEV(numErrorsLastMinute) OVER lastMinute AS stdDevErrorsPerMinute FROM ServiceRequestsPerMinute WINDOW lastMinute AS (PARTITION BY url RANGE INTERVAL ‘1’ MINUTE PRECEDING) ) AS S WHERE S.numErrorsLastMinute > S.avgErrorsPerMinute + 2 * S.stdDevErrorsPerMinute; The Power Of Streaming SQL BUSINESS NEED: Predicting run-away applications before resource consumption becomes an issue. BLAZING SPEED: Processing millions of records per second on low- end servers.
  • 22. | 22Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com The SQLstream s-Streaming Product Portfolio s-Server Data Management Platform for Streaming Big Data s-Analyzer Real-Time Visualization for Streaming Operational Intelligence s-Transport Geo-Analytics for Location-based Applications s-Visualizer Advanced Visualization s-Cloud s-Server EC2 AMI Deployment
  • 23. Industry Use Cases
  • 24. | 24Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Real-time Operations – IT And Beyond High Velocity: The Next Frontier For Big Data INTERNET OFINTERNET OF THINGSTHINGS NETWORKS &NETWORKS & SERVICESSERVICES SECURITYSECURITY Machine-to-MachineMachine-to-Machine Cars as SensorsCars as Sensors Web SecurityWeb Security Banking & FinanceBanking & FinanceTelecommunicationsTelecommunications Customer LoyaltyCustomer Loyalty
  • 25. | 25Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Log Analytics For Security Intelligence  Identity Theft Monitoring » Real-time detection and prevention » Real-time log and web feed analysis on a massive scale  Fraud prevention » Identify and block suspicious account activity » Monitor transaction and activity logs in real-time  Cybersecurity Attacks » Identify patterns of activity for Advanced Persistent Threats » Log file and security device monitoring, with geospatial analytics
  • 26. | 26Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Internet Of Everything – Use Case Examples  Intelligent Transportation » Reduce congestion and travel times; improve traveller experience » Real-time flow and congestion prediction from GPS data  Telematics (V2V Infrastructure) » Reduce ‘walkaway’ events and warranty costs » Real-time vehicle health monitoring every 10 ignition cycles  M2M » Monetization of M2M data feeds » Real-time wireless sensor analytics and aggregation
  • 27. | 27Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com Telecommunications – Use Case Examples  CDR Analytics » Real-time charging and QoS / QoE monitoring » CDR collection, session reconstruction and analysis at scale  4G Wireless Network Performance » Identify and prevent QoS exceptions » 4G cell performance data analysis in real-time
  • 28. | 28Copyright © 2013 | +1 877 571 5775 | inquiries@sqlstream.com DATA EXPLOSION COMPLEXITY BUSINESS AGILITY Streaming Operational Intelligence Eliminates the development risk •Simplifies development, rapid time to market Lowest Cost of Performance for Real-time Apps •Efficient scale-out for high velocity data Adding new applications on the fly •With dynamic sharing of data streams across Apps
  • 29. Damian Black Email | damian.black@sqlstream.com Website | www.sqlstream.com