• Share
  • Email
  • Embed
  • Like
  • Private Content
No Time Like the Present – The Case for Streaming Analytics
 

No Time Like the Present – The Case for Streaming Analytics

on

  • 743 views

The Briefing Room with John Myers and SQLstream ...

The Briefing Room with John Myers and SQLstream
Live Webcast Jan. 14, 2014
Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?AT=pb&SP=EC&rID=8028437&rKey=f7805e721c625b13

Many of today's opportunities and challenges come and go faster than a data warehouse can perform. That's one reason why streaming analytics are starting to take off. For all kinds of Big Data, streaming analytics can provide business value almost instantaneously. Whether for protecting critical infrastructure, knowing which offer to make to a high-value customer, or detecting fraud before serious damage is done, streaming analytics can provide tangible solutions in near-real-time.

Register for this episode of The Briefing Room to hear industry Analyst John Myers as he explains the changing dynamics of modern analytics. He'll be briefed by Damian Black of SQLstream who will discuss his company's streaming Big Data Platform which uses a standards-based approach to provide new levels of visibility into a wide range of log file, sensor, network and machine-generated data. He'll also detail their latest offerings – StreamApp templates, which are pre-configured streaming application components for specific industries.

Visit InsideAnalysis.com for more information

Statistics

Views

Total Views
743
Views on SlideShare
711
Embed Views
32

Actions

Likes
1
Downloads
12
Comments
0

1 Embed 32

http://insideanalysis.com 32

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    No Time Like the Present – The Case for Streaming Analytics No Time Like the Present – The Case for Streaming Analytics Presentation Transcript

    • Grab some coffee and enjoy the pre-show banter before the top of the hour!
    • No Time Like the Present – The Case for Streaming Analytics The Briefing Room
    • Welcome Host: Eric Kavanagh eric.kavanagh@bloorgroup.com Twitter Tag: #briefr The Briefing Room
    • Mission !   Reveal the essential characteristics of enterprise software, good and bad !   Provide a forum for detailed analysis of today s innovative technologies !   Give vendors a chance to explain their product to savvy analysts !   Allow audience members to pose serious questions... and get answers! Twitter Tag: #briefr The Briefing Room
    • Topics This Month: ANALYTICS February: BIG DATA March: CLOUD 2014 Editorial Calendar at www.insideanalysis.com/webcasts/the-briefing-room Twitter Tag: #briefr The Briefing Room
    • Analytics What do you MEAN you need your data NOW? Twitter Tag: #briefr The Briefing Room
    • Analyst: John Myers John Myers is Research Director of Business Intelligence at Enterprise Management Associates Twitter Tag: #briefr The Briefing Room
    • SQLstream ! SQLstream is an enterprise software company focused on making businesses responsive to real-time Big Data assets !   Its platform provides a relational stream for analyzing large volumes of service, sensor, and machine and log file data !   SQL queries in SQLstream generate results continuously as data becomes available Twitter Tag: #briefr The Briefing Room
    • Guests: Damian Black & Christian Lees Damian Black CEO, SQLstream •  Career in high tech, real-time software sector, with senior positions at HP, XACCT (now Amdocs) and Followap (now Neustar) •  Holds 11 US patents •  Finalist in the 1995 International Management Challenge Christian Lees CTO, InfoArmor •  •  Twitter Tag: #briefr Over 15 years of information security, network security and intrusion detection experience CTO of InfoArmor, with previous experience at Level 3 Communications, Trustwave and owner of Sage Technologies The Briefing Room
    • S Q L s t r e a m : Re a l - t i m e B i g D a t a P l a t fo r m Streaming Analytics from High-velocity Machine Data facts capabilities innovations o  Launched 2009 o  Unstructured and structured data o  Massively scalable streaming data platform o  Deployments across many industries o  Accelerates and extends Hadoop & RDBMS o  Only standard SQL streaming engine o  Real world benchmarks o  Not only SQL o  Five patents for stream processing Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 10
    • S e l e c t e d C u s t o m e r s & Pa r t n e r s Telecommunications Intelligent Transportation Security Intelligence IT Operations Internet of Things & Sensors Smarter Internet Selected Strategic Partners Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 11
    • Bridging The Chasm Operational Intelligence integrates Operations and BI “ Operations Business Intelligence Transaction Processing Post-hoc Analysis Machine Data Data Warehousing Everyday business Strategic insights As we move toward a real-time business environment, the capability to process data flows swiftly and flexibly will become increasingly important. SQLstream leads the industry in this kind of ” capability. Robin Bloor Chief Analyst for Bloor Group Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 12
    • Bridging The Chasm Operational Intelligence integrates Operations and BI “ Operational Intelligence Optimizes tactical decisions from real-time actionable insights Combines operations data with BI data continuously Provides Real-time integrated view of the business and operations Operations Transaction Processing Machine Data Everyday business Security Compliance Fraud Quality Promotion Advertising Cross-selling Business Intelligence Post-hoc Analysis Data Warehousing Strategic insights As we move toward a real-time business environment, the capability to process data flows swiftly and flexibly will become increasingly important. SQLstream leads the industry in this kind of ” capability. Robin Bloor Chief Analyst for Bloor Group Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 13
    • T h e I n f o r m a t i o n Va l u e C h a i n Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 14
    • T h e I n f o r m a t i o n Va l u e C h a i n Make it happen! What might happen? What is happening? What just happened? Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 15
    • S T R E A M I N G A N A LY T I C S
    • Analytics previously meant High-latency Current architectures o  Multi-stage processing o  Batch ETL o  Interim operational data stores IMPACT o  High Cost of Ownership o  Delays to internal customers and consumers o  Delays to external customers and partners WAREHOUSE ETL PLATFORMS Near-term data storage Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 17
    • Streaming Analytics Massively parallel with incremental evaluation ¤  Continuous queries on unstructured & structured streaming data ¤  Incremental query results ¤  Predictive analytics & automated actions Operational Intelligence M2M Radio Logs Wireless Networks Mobile Security gateways Sensors Internet Enhancing with historical information Storage of intermediate & final query results Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 18
    • SQL Where is the intelligence? Transaction Log Details TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342 Web Server Logs [Sun Feb 17 15:30:49 2013] [notice] srv-sfo-08 caught SIGTERM, shutting down [Sun Feb 17 15:30:49 2013] [notice] Apache/2.2.21 -- resuming normal operations CDRs TERMINATE,ctl09gsx,01299796304,GMT-08:00,02-17-13,15:21:00,9,387,64ms,02-17-13,15:30:55,0005, IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60, 234.234.60.75,65678,411,399,SIP,SANFRANCISCO,0x4B1698,0x0005E,0x49768,4157588342,0198873465 <id>1597831220</id><deviceid>0198873465</deviceid><lat>lat=47.643957</lat><lon>lon= -122.3269</lon><time>2013-02-17T15:37:26Z</time><bearing>223.4535</bearing> Device Locations <id>1597865781</id><deviceid>0198873465</deviceid><lat>lat=47.645982</ lat><lon>lon=-122.327500</lon><time>2013-02-17T15:37:26Z</time><bearing>200.6138</bearing> <id>1597940125</id><deviceid>0198873465</deviceid><lat>lat=47.647381</ lat><lon>lon=-122.326501</lon><time>2013-02-17T15:37:26Z</time><bearing>87.4357</bearing> Twitter {"created_at:Thu Feb 17 15:30:55 +0000 2013,id:304612775055998976,id_str: 304612775055998976,text:@MyServiceProvider today sucks, keeps dropped!,source:u006ca href=http:www.url.com rel=nofollow,followers_count:147,friends_count:10142, location: San Francisco, time_zone: Pacific, geo_enabled:true, location:u00dcT: -6.1987552,106.8661953, screen_name:APerson Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 19
    • SQL Where is the intelligence? Transaction Log Details Timestamp TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342 Timestamp Web Server Logs [Sun Feb 17 15:30:49 2013] [notice] srv-sfo-08 caught SIGTERM, shutting down [Sun Feb 17 15:30:49 2013] [notice] Apache/2.2.21 -- resuming normal operations Timestamp CDRs TERMINATE,ctl09gsx,01299796304,GMT-08:00,02-17-13,15:21:00,9,387,64ms,02-17-13,15:30:55,0005, IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60, 234.234.60.75,65678,411,399,SIP,SANFRANCISCO,0x4B1698,0x0005E,0x49768,4157588342,0198873465 <id>1597831220</id><deviceid>0198873465</deviceid><lat>lat=47.643957</lat><lon>lon= -122.3269</lon><time>2013-02-17T15:37:26Z</time><bearing>223.4535</bearing> Device Locations <id>1597865781</id><deviceid>0198873465</deviceid><lat>lat=47.645982</ lat><lon>lon=-122.327500</lon><time>2013-02-17T15:37:26Z</time><bearing>200.6138</bearing> Timestamp <id>1597940125</id><deviceid>0198873465</deviceid><lat>lat=47.647381</ lat><lon>lon=-122.326501</lon><time>2013-02-17T15:37:26Z</time><bearing>87.4357</bearing> Timestamp Twitter {"created_at:Thu Feb 17 15:30:55 +0000 2013,id:304612775055998976,id_str: 304612775055998976,text:@MyServiceProvider today sucks, keeps dropped!,source:u006ca href=http:www.url.com rel=nofollow,followers_count:147,friends_count:10142, location: San Francisco, time_zone: Pacific, geo_enabled:true, location:u00dcT: -6.1987552,106.8661953, screen_name:APerson Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 20
    • SQL Where is the intelligence? Transaction Log Details Customer Timestamp Server [Sun Feb 17 15:30:49 2013] [notice] srv-sfo-08 caught SIGTERM, shutting down [Sun Feb 17 15:30:49 2013] [notice] Apache/2.2.21 -- resuming normal operations Timestamp Mobile # CDRs Mobile # TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342 Timestamp Web Server Logs Fail Code TERMINATE,ctl09gsx,01299796304,GMT-08:00,02-17-13,15:21:00,9,387,64ms,02-17-13,15:30:55,0005, Device ID Term Reason IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60, 234.234.60.75,65678,411,399,SIP,SANFRANCISCO,0x4B1698,0x0005E,0x49768,4157588342,0198873465 Device ID Location <id>1597831220</id><deviceid>0198873465</deviceid><lat>lat=47.643957</lat><lon>lon= -122.3269</lon><time>2013-02-17T15:37:26Z</time><bearing>223.4535</bearing> Device Locations <id>1597865781</id><deviceid>0198873465</deviceid><lat>lat=47.645982</ lat><lon>lon=-122.327500</lon><time>2013-02-17T15:37:26Z</time><bearing>200.6138</bearing> Timestamp <id>1597940125</id><deviceid>0198873465</deviceid><lat>lat=47.647381</ lat><lon>lon=-122.326501</lon><time>2013-02-17T15:37:26Z</time><bearing>87.4357</bearing> Timestamp Twitter {"created_at:Thu Feb 17 15:30:55 +0000 2013,id:304612775055998976,id_str: 304612775055998976,text:@MyServiceProvider today sucks, keeps dropped!,source:u006ca href=http:www.url.com rel=nofollow,followers_count:147,friends_count:10142, location: San Francisco, Service Provider time_zone: Pacific, geo_enabled:true, location:u00dcT: -6.1987552,106.8661953, screen_name:APerson Location Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 21
    • Streaming Analytics Platfor m Billing Network Analysis Rating CLEANING & FILTERING STREAMING ANALYTICS Log M2M Mobile Fraud Monitoring STREAMING AGGREGATION Networks Radio towers QoE CONTINUOUS INTEGRATION Sensors Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 22
    • Re a l - t i m e A r c h i t e c t u r e Continuous Raw Data Ingestion, Integration, Analysis and Output of Derived Data in Real-time Real-time Dashboards & Visualization Streaming SQL Real-time Applications SQL Developer Tools Query Planner & Optimizer for MPP Execution Platform Administration Streaming Agent/Adapter Layer + JDBC API Impala SQL HBase Logs Networks M2M Servers Telematics Sensors GPS Social Media External Data Warehouses & Systems HDFS / MR Data Warehouse Hadoop for Stream Persistence, Enrichment & Replay (Optional) Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 23
    • Geo-Analytics for Location-based Applications s-Analyzer s-Visualizer Drag and Drop Application Builder for Streaming Analytics Applications Advanced Enterprise Visualization s-Server Dashboards Data Management Platform for Streaming Big Data s-Cloud Fast Start Streaming Apps s-Transport StreamApps Developer & Admin Console s-Studio S Q L s t r e a m s - S t r e a m i n g P r o d u c t Po r t f o l i o s-Server EC2 AMI Deployment Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 24
    • Case studies
    • CLOUD INFRASTRUCTURE MONITORING Cloud infrastr ucture monitoring with Bollinger bands SELECT STREAM ROWTIME, url, numErrorsLastMinute FROM ( SELECT STREAM ROWTIME, url, numErrorsLastMinute, AVG(numErrorsLastMinute) OVER lastMinute AS avgErrorsPerMinute, STDDEV(numErrorsLastMinute) OVER lastMinute AS stdDevErrorsPerMinute FROM ServiceRequestsPerMinute WINDOW lastMinute AS (PARTITION BY url RANGE INTERVAL ‘1’ MINUTE PRECEDING) ) AS S WHERE S.numErrorsLastMinute > S.avgErrorsPerMinute + 2 * S.stdDevErrorsPerMinute; BUSINESS NEED: Detect run-away applications before resource consumption becomes an issue. Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 26
    • C u s t o m e r B e n c h m a r ke d Pe r fo r m a n c e Large network & telecom equipment manufacturer SYSTEM CHARACTERISTICS PERFORMANCE STATISTICS Collection: Intelligent Remote Agents (Distributed) System Throughput: 1.35M events / sec Enrichment: Streaming data augmentation Server Configuration: 1 x 4-core CPU Analytics: Temporal & spatial pattern detection Event Size: ~1KB Output: Data warehouse + applications (JDBC) Data Sources: Many Network Data Remote Agent Network Data Remote Agent Network Data Remote Agent Network Data Remote Agent Network Data Remote Agent SQLstream ENRICH ANALYZE Data Warehouse SHARE External Systems External Data Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 27
    • C a s e s t u dy : C a l l Ra t i n g & Fra u d Veracity Networks “SQLstream allows Veracity to provide vital real-time reports to our customers that previously took hours to create. SQLstream also provides real-time monitoring and insight into network concerns allowing Veracity to proactively address any such issues.” Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 28
    • C a s e s t u dy : f ra u d p r e ve n t i o n ( c o n t . ) duration Customer call profile Mo Tue Wed Thu Fri Sat Destination ①  LA ②  SF ③  NY ④  …. ①  LA ②  Nairobi ③  NY ④  ….. Location ①  LA ②  LA1 IP spoofing alerts ①  LA ②  Detroit Sun S T R E A M I N G A N A L Y T I C S Alerts Triggers •  Call suspension •  Acct. suspension •  Emails Reports Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 29
    • I n f o A r m o r c a s e s t u dy
    • C a s e s t u dy : C y b e r s e c u r i t y InfoArmor ¤ Founded by Washington Mutual to protect 10M credit card holders ¤ Growing at triple digit rates ¤ Engaged, satisfied subscribers NEEDS ¤  Decision engine ¤ Consume agnostic data sources ¤ Scalable ¤ Real-time Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 31
    • C a s e s t u dy : C y b e r s e c u r i t y a g r ow i n g m a r k e t $207 Billion Entrepreneur.com ¤  No longer an unorganized hacker world ¤  Innovation and technology ¤  Global economy In 2012, U.S. Navy databases were hacked and 200,000 sailors’ information was put at risk. ¤  Political support Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 32
    • C y b e r A t t a c k s | D A M AG E S î  12.6 Million Americans were ID Theft victims last year î  608,271,950 and growing records have been compromised due to security breaches since 2005 î  94% of healthcare organizations surveyed had at least one data breach in the past 2 years î  1 in 4 data breach notification recipients became a victim of identity fraud î  5 times more likely to be a fraud victim if your Social Security Number has been compromised in a data breach Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 33
    • I N T E R N E T S U RV E I L L A N C E What is the Underground Economy? An ever-evolving complex of compromised machines, networks and web services identified by InfoArmor and leading cyber security firms. InfoArmor Internet Surveillance uses bots to continuously monitor the Underground Economy to uncover compromised, sensitive information. Whether it is personal identifying data or a medical insurance card, Internet Surveillance uncovers breached data and alerts in real time. What We Monitor: ¤  Malicious Command & Control Networks ¤  Phishing Networks ¤  Black Market Forums ¤  Exploited Websites ¤  Known Compromised Machines & Servers Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 34
    • I N T E R N E T S U RV E I L L A N C E X INFOARMOR BOTS monitor UNDERGROUND ECONOMY COMPROMISED DATA sent back to INFOARMOR SENSOR compares compromised to subscriber data in secure environment, creating ALERTS with 100% accuracy How We Monitor: ¤  Proprietary hardware and software solution ¤  Unparalleled alert accuracy (minimized false positives) ¤  Secure: separate reconnaissance and analysis efforts, plus no refined search queries What We Monitor: ¤  Credentials, SSNs, names, addresses, emails and DOBs ¤  Wallet items (i.e. credit cards, medical insurance card) Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 35
    • C a s e s t u dy : S t r e a m i n g a n a l y t i c s SQLstream BENEFITS ¤ Ability to adapt to many data sources ¤ Real Time analysis and alerting ¤ Offset database load ¤ Data Hygiene prior to data warehousing RESULTS ¤ Real-time actionable alerts ¤ Unity in Ingress Data points ¤ Dual Purpose solution •  Helps Compliance ¤ Plans to expand engagement offline online Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 36
    • Damian Black Email | damian.black@sqlstream.com Website | www.sqlstream.com DOWNLOADS | http://www.sqlstream.com/downloads/
    • Perceptions & Questions Analyst: John Myers Twitter Tag: #briefr The Briefing Room
    • Importance of Speed of Response in Big Data John L Myers Enterprise Management Associates Research Director JMyers@EnterpriseManagement.com © 2012 Enterprise Management Associates, Inc.
    • Speaker John L Myers Enterprise Management Associates Research Director John Myers joined Enterprise Management Associates in 2011 as senior analyst of the business intelligence (BI) practice area. John has 10+ years of experience working in areas related to business analytics in professional services consulting and product development roles, as well as helping organizations solve their business analytics problems, whether they relate to operational platforms, such as customer care or billing, or applied analytical applications, such as revenue assurance or fraud management. Slide 40 JohnLMyers44 © 2013 Enterprise Management Associates, Inc.
    • Disruptive Forces in Data Management: Changing the Speed of Business Slide 41 75 65 55 45 35 25 © 2013 Enterprise Management Associates, Inc.
    • Use Cases met with Big Data Implementations •  Speed of processing response •  Combining data by structure •  Pre-processing data •  Utilization of streaming data •  Staging structured data •  Online archiving Slide 42 Rogers, Myers and Devlin, "Big Data: Operationalizing the Buzz", Enterprise Management, http://research.enterprisemanagement.com/big-data-2013-webinar-nl.html © 2013 Enterprise Management Associates, Inc.
    • Big Data Platforms have Multiple Use Cases Slide 43 © 2013 Enterprise Management Associates, Inc.
    • Top 5 Business Challenges Met with Big Data Projects •  Risk management •  Fraud Analysis, Liquidity Risk Assessment •  Ad-hoc operational queries •  Customer Relations Management •  Asset optimization •  Staff Scheduling, Logistical Asset Planning •  Operational event and policy processing •  Billing, Rating •  Campaign Optimization •  Market Basket Analysis, Cross-sell/Up-sell Recommendation •  Clustering, social graph analysis Slide 44 •  Grouping and Relationship Analysis, Geographic Optimization Rogers, Myers and Devlin, "Big Data: Operationalizing the Buzz", Enterprise Management, http://research.enterprisemanagement.com/big-data-2013-webinar-nl.html © 2013 Enterprise Management Associates, Inc.
    • Building the Bridge between Operational Processes and Analytical Results Slide 45 © 2013 Enterprise Management Associates, Inc.
    • Hybrid Data Ecosystem 2013: From Requirements to Consumers Slide 46 © 2013 Enterprise Management Associates, Inc.
    • Questions Slide 47 •  This version of “streaming analytics” sounds a lot like “complex event processing.” How does SQLstream differentiate from those solutions? •  The open source community, such as Apache Hadoop, has been coming up with solutions to problems like streaming. What advantages does a proprietary solution like SQLstream have over these solutions? •  “Streaming analytics” appears to be well suited for the upcoming trends in the “location based services” in mobile telecom and “telematics” in automotive. Which use cases appear to have the best chances of success? Marketing activities such as “location coupons?” Operational optimization such as “managed highways?” © 2013 Enterprise Management Associates, Inc.
    • Questions Slide 48 •  What are the best types of datasets to be used in the world of “streaming analytics?” Structured big data or large volumes of single row event data (i.e., log information)? Formatted multi-row event data (i.e., JSON)? •  What types of datasets should be avoided? •  What types of analytical techniques are best used with “streaming analytics?” Advanced analytical models associated with predictive or clustering algorithms? Rules-based, policy techniques (i.e., decision trees)? Simple descriptive analytics? •  What types of analytics techniques should be avoided? © 2013 Enterprise Management Associates, Inc.
    • Twitter Tag: #briefr The Briefing Room
    • Upcoming Topics This Month: ANALYTICS February: BIG DATA March: CLOUD 2014 Editorial Calendar at www.insideanalysis.com/webcasts/the-briefing-room www.insideanalysis.com Twitter Tag: #briefr The Briefing Room
    • Thank You for Your Attention Twitter Tag: #briefr The Briefing Room