SAP Sybase Event Streaming Processing


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SAP Sybase Event Streaming Processing

  1. 1. High Performance CEP:The SAP Sybase Event Stream ProcessorTechnical Product OverviewSpeaker’s Name/Department (delete if not needed)Month 00, 2011
  2. 2. Insight from fast-moving data: Continuous IntelligenceAnalyze events as they occur• Continuous insight• Immediate responseRapid application development• Reduce/eliminate dependence on specialist programming skills• Cut implementation/deployment time• Improve business agilityNon-intrusive deployment• Event-driven architecture• Adapt to existing data models© 2012 SAP AG. All rights reserved. 2
  3. 3. The SAP Sybase Event Stream Processor INPUT Studio STREAMS (Authoring) Reporting Sensor Data Tools SAP® Dashboard ? Transactions Sybase® Applications Message Events ESP Bus SAP HANA, Sybase IQ Reference Data • Unlimited number of input • Send output to apps, dashboards streams • Capture data in SAP HANA, Sybase IQ, • Incoming data is processed as it Sybase RAP for historical analysis, arrives, according to the compliance business logic defined using high level authoring tools© 2012 SAP AG. All rights reserved. 3
  4. 4. Sample Architecture of a CEP-based Application:Real-time P&L Market Prices  Reuters Live Dashboard Market Data Middleware  Bloomberg  Nasdaq  … Order Management Systems Alerts Trading Systems Sybase Historical Analytics, Ad-hoc Analysis, IQ Compliance© 2012 SAP AG. All rights reserved. 4
  5. 5. Two approaches to CEP Continuous Rules Queries© 2012 SAP AG. All rights reserved. 5
  6. 6. Key ConceptsInput Streams Derived Streams,Events arrive on input streams Windows Apply continuous query operators to one or more input streams to produce a new stream Database Analogy Windows can Have State • Streams are like tables • Retention rules define how many or how • Events are rows in the table long events are kept • Fields in an event message are • Opcodes in events can indicate columns in the table insert/update/delete and can be automatically applied to the window© 2012 SAP AG. All rights reserved. 6
  7. 7. ESP Studio: Both visual and textual authoring Visual Dataflow Authoring Language-based Authoring • ANALYST-LEVEL SKILLS (EXCEL, VBA) • RAPID PROGRAMMING • EASY TO UNDERSTAND COMPLEX MODELS • EASY TO USE LANGUAGE (CCL) • NO NEED TO LEARN LANGUAGE SYNTAX • MODULAR, PROJECT BASED APPROACH ESP Studio supports both Visual and Textual authoring • Visual diagram produces CCL (code-behind approach) • User can switch back and forth between diagram and language • Eclipse-based© 2012 SAP AG. All rights reserved. 7
  8. 8. CCL: Continuous Computation LanguageSQL-based Event Processing Language Leverage familiarity and simplicity of SQL Instead of “snapshot” queries, continuous queries Filter (WHERE), Aggregate (GROUP BY), Join, Compute Extensions for event streams Windows, Patterns Example CCL Query Calculation of 10 CREATE LOCAL WINDOW VWAP10 minute VWAP PRIMARY KEY DEDUCED AS SELECT TICKER.Symbol AS Symbol , Sum ( TICKER.Price * TICKER.Size) / Sum ( TICKER.Size) AS VWAP FROM TICKER KEEP 10 MINUTES GROUP BY TICKER.Symbol ;© 2012 SAP AG. All rights reserved. 8
  9. 9. Key CCL Language Features• Stateless and stateful elements. • Full support for JOINS– STREAMS (stateless without keys) – ANSI JOIN Syntax– WINDOWS (stateful with keys) – Stream-Window Joins and Window-Window joins– Windows push updates and automatically process opcodes. • Filter• Retention policies on Windows • Aggregate– Named windows (windows created with “CREATE – Group By, Group Filter, Group Order WINDOW..”)– Unnamed windows (FROM A KEEP 5 ROWS) • Pattern Matching– Time and Count based retention policies – Watch for specific sequence of events within time interval– Custom windows with EventCache – Within or across streams• Parameters, Variables – Pattern can include missing events– Variables can maintain state • Transform/Project – Change event schema, compute new fields© 2012 SAP AG. All rights reserved. 9
  10. 10. SPLASH scripting adds extensibility to CCLSPLASH (Stream Processing LAnguage SHell) was introduced in Aleri to provideextensibility and overcome the limitations of SQLIt complements CCL, providing the ability to write custom functions and stream/windowoperators using a simple, procedural language that is optimized for stream processingSyntax is similar to C and Java, it’s spirit is closer to little languages like AWK or PerlESP 5.0 has full support for SPLASH Control Structures: Data Structures: Flex Streams: if, while, switch if (temperature < 0) { - Variables - Custom event “handlers” display := below zero; - Vectors written in SPLASH } else if (temperature = 0) { - Dictionaries display := zero; } else { - Event Cache display := above zero; } For more info see the SPLASH Tutorial on (in the Aleri Product Documentation/Manuals)© 2012 SAP AG. All rights reserved. 10
  11. 11. ModularityBenefits: CREATE MODULEFacilitates re-use • Defines a module consisting of windows and streamsSimplifies team development • Define inputs, outputs, parametersImproves manageability oflarge projects IMPORT • Equiv. to #include – include external .ccl file(s) • Beyond modules, re-use declarations, etc LOAD MODULE • At load point, define bindings and parameters • Same module can be loaded multiple times© 2012 SAP AG. All rights reserved. 11
  12. 12. Support for External function libraries (UDF)• Call out to external function libraries• Support for C and Java• Can be used to supplement built-in functions• Can also be used to “trigger” external actions© 2012 SAP AG. All rights reserved. 12
  13. 13. ESP Studio: RUN/TEST ToolsSingle click to Run a projectRich Suite of Testing, Tuning and Debugging toolsStream viewerRecord/playbackManual event inputPerformance MonitorEvent TracerDebugger with BreakpointsMulti-server controlView/monitor projects running on multiple servers and clusters© 2012 SAP AG. All rights reserved. 13
  14. 14. ESP Advanced Cluster Architecture Manager Private Cloud Manager Manager Manager Controller Controller Controller Project Project Project Project Project Project Project Project Project Project Project Project Project Project Project© 2012 SAP AG. All rights reserved. 14
  15. 15. ESP Cluster ArchitectureKey FeaturesManager • Equi-peer management of cluster. • Program deployment and lifecycle control. • Program load balancing at deployment. • Launcher failure detection. • Container failure detection. • Program failover. • Publishes lifecycle updates for Managers, Controllers and Programs. • Name resolution.Controller • Acts as a daemon for Managers to launch Container processes. • Allows for launching of any configured Program type.Container • Process that hosts a Program (i.e. – esp_server).© 2012 SAP AG. All rights reserved. 15
  16. 16. Choice of high availability configurations FT Cluster ManagerConfigurable to your level ofavailability and flexibility needsInstant failover to "hot" spare Synchronized• highest degree of availability Hot Spare• recovery in seconds; no data lossShared resources "warm" spares• flexible configurations• recovery in minutes• parallel & pipeline configurations Parallel & Pipelined Shared State Shared Warm SparesAll managed by fault tolerantcluster manager © 2012 SAP AG. All rights reserved. 16
  17. 17. ADAPTERS Standard Optional Custom (Built-in) (Add-on) (SDK) • JMS, MQ, • Market Data • C/C++ • Reuters, NYSE Tech TIBCO • FIX • Java • Sockets • Rep Server • .NET • Databases • Files • MS Excel • Email (out)© 2012 SAP AG. All rights reserved. 17
  18. 18. Integration/Connectivity Application with In-Process External Adapters Native ESP Adapters Support • Entirely under the • Independent of • Embed ESP control of the ESP ESP Server pub/sub API server. • Uses ESP directly in • Performance pub/sub API producer or advantages • TCP socket consumer • No hot-standby at connectivity to applications the adapter level ESP Server • C/C++ only • Auto-failover in hot-standby configurations • C/C++, Java, .NET© 2012 SAP AG. All rights reserved. 18
  19. 19. USING Sybase Replication Server for Change CaptureReal-time change capture: turn database transactions into event streams Transactions Database (insert, update, Replication Replication Agent Server Real-time Event delete) Stream SAP Sybase ESP© 2012 SAP AG. All rights reserved. 19
  20. 20. High Level ESP Architecture Cluster Manager Security Load Balancing Failover Lifecycle/Status Distributed Cache SDK Interface URI Resolution Client Launcher SDK - .NET JAVAC++ ESP Project Security (Kerberos, PAM, SSL) Adapter Query Engine Command & Control SQL Query Gateway SDK Studio Window Adapter External Stores Framework Adapter Tools s Internal Adapter Compiler s Log File Data Source/Destination© 2012 SAP AG. All rights reserved. 20
  21. 21. What sets SAP Sybase ESP apart• Performance • State Management − Scalable for extreme throughput − Unique ability to automatically apply − Consistent low-latency incoming events as − Only CEP vendor to submit to STAC inserts/updates/deletes to a table benchmarking − Simplifies modeling• SPLASH scripting − Extremely efficient − Overcome limitations of SQL • Dynamic − Flexibility and productivity − Add continuous queries to live system• Authoring flexibility • High Availability − Language or UI/IDE − Hot-Hot with Auto-Failover• “Private Cloud” Architecture • Advanced Subscriptions − Scalable, Dynamic − Initial state followed by updates − Subscriptions with predicates © 2012 SAP AG. All rights reserved. 21
  22. 22. High Performance STAC Certified measurement of throughput and latency The only CEP vendor to publish STAC benchmarks • Linear scaling with addition of more cores: 180,000 messages/sec on 16 cores 300,000 messages/sec on 24 cores • Consistent/predicable low latency. Only minor increase in latency from 20,000 to 300,000 messages/sec: Mean: 1.296 - 1.568 ms Medium: 1.177 - 1.502 ms Std Dev: 0.567 - 1.104 ms Min: 0.091 – 0.115 ms 99th percentile: 2.479 – 3.165 ms Source: STAC Research Sep 08 – conducted against Aleri 3.1; 5.0 performance is similar© 2012 SAP AG. All rights reserved. 22
  24. 24. Visually Rich Real-time Dashboards withHistorical Drill-down and Analysis© 2012 SAP AG. All rights reserved. 24
  25. 25. Panopticon EX FeaturesVisual Data AnalysisReal-time streaming from Sybase ESPHistoric Time Series data from Sybase RAP or IQDrag-and-Drop DesignerRich palette of charting objectsSimple configurationNo coding requiredMulti-dimensional Analysis (in-memory OLAP)Delivered via Desktop (thick client) or Web (browser)SDK provides APIs for ability to customize, embed© 2012 SAP AG. All rights reserved. 25
  26. 26. Panopticon EX for ESPNative integration with Sybase ESP, RAP, and IQ Panopticon EX Reduces InsightTrue real-time dashboards, Latency subscribing to ESP streaming output  It’s Not Just the Time it Takes to Get the DataSeamless drill-down into  It’s also the time it historical time series data in takes to understand the Sybase RAP data Rich Palette of Visualization Tools Designed to Make it Easier to Understand the Data© 2012 SAP AG. All rights reserved. 26
  27. 27. Thank youContact information:F name MI. L nameTitleAddressPhone number