Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Windows of Opportunity: Big Data on Tap

820
views

Published on

The Briefing Room with Robin Bloor and SQLstream …

The Briefing Room with Robin Bloor and SQLstream
Live Webcast on Jan. 8, 2013

Most business opportunities are moving targets these days, rendering static analytical solutions rather ineffective. Instead, organizations need technologies that enable a much bigger picture, complete with multiple data streams that can be combined to show what's happening in real-time. And increasingly, companies need to analyze both traditional structured data as well as Big Data, including machine-generated data from all manner of enterprise systems.

Check out this episode of The Briefing Room to hear veteran Analyst Robin Bloor explain how a confluence of market forces has opened the door to a new analytical paradigm, one in which companies can leverage a vast array of data streams to pinpoint windows of opportunity as or even just before they appear. Bloor will be briefed by Damian Black of SQLstream, who will discuss his company's analytical platform, which enables the management of dynamic information assets in much the way that traditional databases do for stored assets.

Visit: http://www.insideanalysis.com

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
820
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. The Briefing Room
  • 2. Welcome Host: Eric Kavanagh eric.kavanagh@bloorgroup.comTwitter Tag: #briefr The Briefing Room
  • 3. Mission !   Reveal the essential characteristics of enterprise software, good and bad !   Provide a forum for detailed analysis of today s innovative technologies !   Give vendors a chance to explain their product to savvy analysts !   Allow audience members to pose serious questions... and get answers!Twitter Tag: #briefr The Briefing Room
  • 4. JANUARY: Big Data February: Analytics March: Open Source April: IntelligenceTwitter Tag: #briefr The Briefing Room
  • 5. Big DataTwitter Tag: #briefr THERE IS NO MORE SMALL DATAThe Briefing Room Copyrighted property. May not be copied or downloaded without permission from 123RF Limited.
  • 6. Analyst: Robin Bloor  Robin Bloor is Chief Analyst at The Bloor Group robin.bloor@bloorgroup.comTwitter Tag: #briefr The Briefing Room
  • 7. SQLstream !   SQL stream is an enterprise software company focused on making businesses responsive to real-time big data assets. !   Its core s-Server streaming data management platform collects, analyzes and shares high volume, high velocity structured and unstructured data from any source, in any format. ! SQLstream recently introduced s-Server 3.o, which includes distributed streaming data processing, machine data collection, and integration with Google Big Query and Hadoop Hbase.Twitter Tag: #briefr The Briefing Room
  • 8. Damian Black Damian Black is the founder and CEO of SQLstream, a pioneer in Streaming Big Data. Damian has worked for almost two decades in Silicon Valley, with senior roles in a variety of companies including Hewlett-Packard, Neustar, Xacct Technologies and Followap. He has always focused on real-time data platforms for the largest Internet scale applications. He has spoken at many conferences, and was on GigaOM’s first Big Data panel in 2008. Damian graduated from Manchester University and was one of the first research scientists to join HPLabs Europe. He was selected for the International Management Challenge in conjunction with the Financial Times and Ashridge business school while at Hewlett- Packard. Damian is the author of eleven granted patents with five more pending.Twitter Tag: #briefr The Briefing Room
  • 9. Windows of Oppor tunity: Big Data on Tap™ January 2013 Damian Black CEO, SQLstream Copyright © 2013 SQLstream Inc.
  • 10. S Q L s t r e a m V i s i o n IN 2013 STREAMING DATA MANAGEMENT WILL EMERGE AS THE COREINTEGRATION AND OPERATIONAL INTELLIGENCE PLATFORM FORREAL-TIME BIG DATA SOLUTIONS WITHIN THE ENTERPRISE. PROVEN OPEN INNOVATIVE ➔ Founded in 2003. ➔ 100% standard SQL. ➔ Leaders in Streaming Big Data Management. ➔150+ engineering years. ➔ Dynamically extendable using C++ Java and more. ➔ Best real-time technology➔ Over 25 customers and and most complete platform. focused on Fortune 1000 ➔ Comprehensive set of companies. adapters. ➔ Holds 5 key streaming patents (with 3 pending). Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 10
  • 11. W h a t i s S t r e a m i n g B i g D a t a M a n a g e m e n t ? DEFINITION Streaming Big Data = Big Data + Real-time Data Capture & Collection + Continuous Integration & ETL + Low Latency Transformation & Analysis EFFECT Businesses become “real-time responsive” to Big Data. Unlocks the power and value of real-time Big Data. WHERE ARE THE REAL-TIME DATA SOURCES? REAL- TIME DATA Log and Machine Data ✔ Cloud and Device health ✔ Sensor Networks ✔ Social Interaction & Feeds ✔ CDR and Service Data ✔ Automotive & Telematics ✔ Wireless Networks ✔ Streaming media QoS ✔ GPS and Location Data ✔ Application transactions ✔ Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 11
  • 12. S t r e a m i n g B i g D a t a i n A c t i o n Telematics Cloud Intelligent Transportation Real-time traffic flow analytics from vehicleDevice health monitoring Real-time prediction of GPS data feeds with intelligent integration resource over-utilization High volume, high velocity,Social Media structured and unstructured data Telecomm Real-time semantic from software platforms, Real-time QoS andstreaming for QoE capacity monitoring frommonitoring applications and systems. CDR data Log Files Sensors Services Markets Internet Location Networks Devices HPC Banking Big Data log monitoring on Real-time fraud and a massive scale security event prediction Internet Real-time content, activity and securityevent monitoring Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 12
  • 13. P l a t f o r m R e q u i r e m e n t s f o r R e a l - T i m e B i g D a t a Continuous data analysis and integration using distributed streaming platform Both On-Cloud Parallel Dataflow & 100% Standards- and On-Premise Distributed Stream compliant with Deployment Processing Architecture true SQL:2008 Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 13
  • 14. S t r e a m i n g B i g D a t a – P a i n Po i n t s DATA EXPLOSION Too costly to analyse voluminous real-time data BUSINESS AGILITY Too slow to respond to new requirements COMPLEXITY Too difficult to build & maintain real-time apps Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 14
  • 15. S t r e a m i n g B i g D a t a – P a i n Po i n t s DATA EXPLOSION Too costly to analyse voluminous real-time data SQLstream slashes TCO for real-time analysis. BUSINESS AGILITY Too slow to respond to new requirements SQLstream allows you to add new apps easily. COMPLEXITY Too difficult to build & maintain real-time apps SQLstream eliminates your development risk. Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 15
  • 16. T h e R e a l - t i m e D a t a M a n a g e m e n t H e a d a c h e … Finance Supply Chain CRM Operations Business Intelligence: & & & & Hadoop HBase & Accounting ERP Billing Management Data Warehouses TIME, MONEY, COMPLEXITY Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 16
  • 17. T h e R e a l - t i m e D a t a M a n a g e m e n t H e a d a c h e … Finance Supply Chain CRM Operations Business Intelligence: & & & & Hadoop HBase & Accounting ERP Billing Management Data Warehouses STREAMING STEAMING EVENT STREAMING CONTINUOUS ANALYTICS AND CORRELATION ALERTS & ALARMS ETL AGGREGATION Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 17
  • 18. M o v i n g f ro m H i g h L a t e n c y t o R e a l - t i m eR e s p o n s i v e n e s s ➔  Traditional ETL approach leads to high latency COLLECT CLEANSE ENRICH ANALYZE SHARE Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 18
  • 19. M o v i n g f ro m H i g h L a t e n c y t o R e a l - t i m eR e s p o n s i v e n e s s ➔  Traditional ETL approach leads to high latency COLLECT ➔  SQLstream Streaming Approach: »  Continuous Parallel Dataflow Execution CLEANSE »  Generate real-time answers immediately »  Deliver and share the results immediately ENRICH ANALYZE SHARE LOW LATENCY Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 19
  • 20. S Q L s t r e a m D a t a fl o w Te c h n o l o g y P i p e l i n i n g a n d S u p e r s c a l a r P a r a l l e l P r o c e s s i n g   = Query Processor   Fine-grained parallelism: simple, massively scalable, super fast. Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 20
  • 21. A S t r e a m i n g S Q L Q u e r y  C l o u d I n f r a s t r u c t u r e M o n i t o r i n g w i t h B o l l i n g e r B a n d s SELECT STREAM ROWTIME, url, numErrorsLastMinute FROM ( SELECT STREAM ROWTIME, url, numErrorsLastMinute, AVG(numErrorsLastMinute) OVER lastMinute AS avgErrorsPerMinute, STDDEV(numErrorsLastMinute) OVER lastMinute AS stdDevErrorsPerMinute FROM ServiceRequestsPerMinute WINDOW lastMinute AS (PARTITION BY url RANGE INTERVAL ‘1’ MINUTE PRECEDING) ) AS S WHERE S.numErrorsLastMinute > S.avgErrorsPerMinute + 2 * S.stdDevErrorsPerMinute; Business need: Predict run-away applications before resource consumption becomes an issue. Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 21
  • 22. C u s t o m e r B e n c h m a r ke d E x a m p l e A p p l i c a t i o n SYSTEM CHARACTERISTICS PERFORMANCE STATISTICS Collection: Intelligent Remote Agents (Distributed) System Throughput: 1.35M events / sec Enrichment: Streaming data augmentation Server Configuration: 1 x 4-core CPU Analytics: Temporal & spatial pattern detection Event Size: ~1KB Output: Data warehouse + applications (JDBC) Data Sources: Many Network Data Remote Agent Network Data Remote SQLstream Data Agent Warehouse Network Data Remote Agent ENRICH ANALYZE SHARE Network Data Remote Agent External Remote Systems Network Data Agent External Data Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 22
  • 23. S Q L s t r e a m P r o d u c t Po r t f o l i o ➔ s-Server Core Streaming Data Management and Integration Platform ➔ s-Analyzer Real-time data stream visualization and dashboards ➔ s-Studio Developer and administration console ➔ s-Cloud Cloud-based EC2 offering ➔ s-Transport GPS, location-based and geospatial analytics module Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 23
  • 24. R e a l - t i m e We b S e r v e r L o g M o n i t o r i n g  M o z i l l a ( G o o g l e : “ Yo u t u b e M o z i l l a G l o w ” ) Real-time monitoring across all download web Web Server Log Files (Remote) servers across the world simultaneously. Streaming  collecDon,  real-­‐Dme   analysis  and  conDnuous  ➔  Collect integraDon  by  locaDon   Remote agents transform log files into real-time streams ➔  Analyze Hadoop HBase Real-time analysis & aggregation by location ➔  Share Continuous ETL into Hadoop Hbase Internet ‘Glow’ app for real-time visualization Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 24
  • 25. R e a l - t i m e Tr a f fi c A n a l y t i c s  Tr a n s f o r m G P S d a t a i n t o r e a l - t i m e t r a f fi c fl o w i n f o r m a t i o n Real-time traffic flow and congestion prediction GPS Vehicle Data Feeds from vehicle GPS data. Streaming  transformaDon  of  GPS   data  into  Traffic  Flow  and  ➔  Collect CongesDon  PredicDon  Events   Collect, cleanse and filter vehicle GPS data feeds ➔  Analyze Geo-DB ‘Snap-to-map’ Transform GPS records into traffic flow information Prediction events for congestion alerts ➔  Share Real-time Google Maps and Google Earth Displays Web and Smartphone access Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 25
  • 26. R e a l - t i m e O p e r a t i o n a l I n t e l l i g e n c e M a r k e t C o m p a r i s o n TRADITIONAL BIG SQLSTREAM ENTERPRISE DATA OPERATIONAL STREAMING BIG CAPABILITY INTELLIGENCE TOOLS DATA PLATFORM SQLSTREAM BENEFITS True Real-time Moderate to high latency. Real-time low latency. Instant results Incomplete answers. Complete answers. Sophisticated Simple patterns. Full SQL power. Elegantly handle every No real power. Very high-level, concise. business need. Analytics Joins & Correlation Operates on a single feed Join & correlate across Compare and combine only. multiple different feeds info in real-time. Data Enrichment Weak, simplistic. Continuous, powerful. Create complete Incomplete. Comprehensive. answers continuously. & Integration Big Data Scalability Limited scalability. Massively scalable. Delivers low cost, high Cost prohibitive. Inexpensive. performance needed No parallel processing. Massively parallel. for real-time big data. Development Ease Proprietary and low level. Standard SQL. Instant productivity. Expensive. Optimized. No hidden obstacles. Time consuming. Parallel. Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 26
  • 27. A N e w D a t a M a n a g e m e n t Q u a d r a n t High Level Declarative Language & Operation (SQL) Stale snapshots Always current Not real-time Streaming integration Costly recalculations Rapid development DATA STREAMING WAREHOUSES BIG DATA Historical Analysis Continuous Analysis Periodic Batches Real-time Processing BATCHED MESSAGING BIG DATA MIDDLEWARE Batch processing Low-level software Low-level but scalable Scattered business logic Extensive coding Brittle with high TCO Low Level Procedural Language & Operation (C++, C#, Java, Pig, JCL, etc.) Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 27
  • 28. B I G DATA O N TA P ™ – D e l i v e r e d . Slashes the TCO for real-time analysis. DATA EXPLOSION •  Scales easily and continuously processes data in real-time. Makes adding new apps easy. BUSINESS AGILITY •  Create powerful real-time apps, and share results easily. Eliminates the development risk and pain. COMPLEXITY •  Real-time parallel processing made simple, scalable and fast. Copyright  ©  2013  1 Big Data on Tap™ |    Damian  Black  |    +1  877  571  5775  |    enquiries@sqlstream.com | 28
  • 29. Windows of Oppor tunity: Big Data on Tap™ Thanks! Damian Black CEO, SQLstream Copyright © 2013 SQLstream Inc.
  • 30. Perceptions & Questions Analyst: Robin BloorTwitter Tag: #briefr The Briefing Room
  • 31. Harnessing Data Flow The Bloor Group
  • 32. Hadoop Is The Reservoir•  Because of its flexibility and scalability as a data store, Hadoop has become the natural reservoir for data, but… –  Hadoop is a multi-purpose engine, but not a performance engine – do not be fooled by its parallelism –  Sometimes you don’t have time to drop the data into Hadoop first; it is not necessarily the first port of call for data –  Sometimes it may be better to leave data where it is, and just replicate The Bloor Group
  • 33. Event Processing The Bloor Group
  • 34. Event Stream Processing The Bloor Group
  • 35. Operational Intelligence•  Real-time BI could also be called operational Intelligence (OI)•  It poses three problems: –  How to establish the stream data flow (at an acceptable speed) –  How to process the data –  How to manage the data The Bloor Group
  • 36. A Side Comment…•  We are familiar with the issue of “Data Life Cycle”•  This issue didn’t just evaporate with the advent of Big Data and Streams Processing – it became more important The Bloor Group
  • 37. !  Does the platform include its own database?!  You use an enhanced SQL for streams processing. Can it handle unstructured data (such as a tweet stream)?!  You characterize your analytics as being “advanced.” Can you expand on what you mean by that? What analytic capabilities does it include?!  It wasn’t entirely clear to me as to how you integrate with legacy data warehouse data flows. Consider data cleansing for example. How does the SQLstream Platform accommodate that? The Bloor Group
  • 38. !  Which sectors/businesses do you expect to be able to make best use of this technology?!  Which companies/products do you regard as competitors (either directly or close competitors)?!  Which companies/products do you partner with?!  How is the product/platform priced? How is the cloud version priced? The Bloor Group
  • 39. Twitter Tag: #briefr The Briefing Room
  • 40. Upcoming Topics This month: Big Data February: Analytics March: Open Source April: Intelligence www.insideanalysis.comTwitter Tag: #briefr The Briefing Room
  • 41. Thank You for Your AttentionTwitter Tag: #briefr The Briefing Room