MICROSOFT CEP Rahadian Dustrial Dewandono TECHNOLOGY Progres Tesis 28-10-2012
UNDERSTANDING STREAMING DATA (1) Question: “how many red cars are in the parking lot”. Answering with a relational database: Walk out to the parking lot. Count vehicles that are Red CarsSELECT COUNT(*) FROM ParkingLotWHERE type = ‘AUTO’AND color = ‘RED’
UNDERSTANDING STREAMING DATA (2)What about: “How many red cars have passed the 40th street exit on the 520 in the lasthour”?Answering with a relational database:Pull over and park all vehicles in a lot, keeping them there for an hour.Count vehicles that are in the lot. Doesn’t seem like a great solution…
UNDERSTANDING STREAMING DATA (3)Different kinds of questions require different ways of answeringthem.Answering the question with a streaming data processing engine:Stand by the freeway, count red cars as they pass by.Write down the answer, deliver the answer. This is the streaming data paradigm in a nutshell – ask questions about data in flight.
WHAT IS STREAMINSIGHTMicrosoft StreamInsight is a powerful platform that you can use to develop anddeploy complex event processing (CEP) applications.Its high-throughput stream processing architecture and the Microsoft .NETFramework-based development platform enable you to quickly implementrobust and highly efficient event processing applications.
HIGHLY OPTIMIZED PERFORMANCEStreamInsight implements a lightweight streamingarchitecture that supports highly parallel execution ofcontinuous queries over high-speed data. The use of in-memory caches and incremental result computation provide High Event- throughput, drivenexcellent performance with high data throughout and low low latency computationlatency. Low latency is achieved because the events areprocessed without costly data load or storage operations inthe critical processing path. With StreamInsight, allprocessing is automatically triggered by incoming events. Inparticular, applications do not have to incur any overhead forevent polling. The platform provides the functionality forhandling out-of-order events. In addition, static reference orhistorical data can be accessed and included in the low-latency analysis.
FLEXIBLE DEPLOYMENTCAPABILITYStreamInsight supports two deployment scenarios. It can be fully integratedinto the application as a hosted (embedded) DLL or deployed as a stand-alone server with multiple applications and users sharing the server. In itsstand-alone configuration, the CEP server runs in a wrapper such as anexecutable, or the server could be packaged as a Windows Service.
DECLARATIVE QUERYLANGUAGE (LINQ)Microsoft StreamInsight use LINQ as a Event Processing Language.
MANAGEABILITYThe monitoring and manageability features built into the CEP server providefor low total cost of ownership (TCO) of CEP applications. The managementinterface and diagnostic views that are provided in the CEP server allow theadministrator to monitor and manage the CEP application. Themanageability framework also allows for ISVs and system integrators toremotely monitor and support CEP-deployed systems at manufacturing andother scale-out installations.
DIAGNOSTIC INTERFACEThe StreamInsight Event FlowDebugger enables you, as adeveloper or administrator of acomplex event processing (CEP)application, to inspect, debug, andreason about the flow of eventsthrough a StreamInsight query.
MANUFACTURING PROCESSMONITORING AND CONTROLTo ensure that products and processes are running optimally andwith the least amount of downtime, manufacturing companiesrequire low-latency data collection and analysis of plant-floordevices and sensors. The typical manufacturing scenarioincludes the following requirements:• Asset-based monitoring and aggregation of machine-born data.• Sensor-based observation of plant floor activities and output.• Observation and reaction through device controllers.• Ability to handle up to 10,000 data events per second.• Event and alert generation the moment something goes wrong.• Proactive, condition-based maintenance on key equipment.• Low-latency analysis of aggregated data (windowed and log-scales).
CLICKSTREAM ANALYSISAn optimal customer experience from a commercial Web site requireslow-latency processing of user behavior and interactions at the site.The typical click stream analysis application includes the followingrequirements:• Ability to drive page layout, navigation, and presentation based on low-latency click stream analysis.• Ability to handle up to 100,000 data events per second during peak traffic times.• Immediate click-stream pattern detection and response with targeted advertising.
ALGORITHMIC TRADING IN A FINANCIALSERVICES ENVIRONMENTAlgorithmic trading, with its high volume data processing needs,typically has the following requirements:Ability to handle up to 100,000 data events per second.Time-critical query processing.Monitoring and capitalizing on current market conditions with very short windows of opportunity.Smart filtering of input data.Ability to define patterns over multiple data sources and over time to automatically trigger buy/sell/hold decisions for assets in a portfolio.
POWER UTILITIESThe utility sector requires an efficient infrastructure formanaging electric grids and other utilities. These systemstypically have the following requirements.• Immediate response to variations in energy or water consumption, to minimize or avoid outages or other disruptions of service.• Gaining operational and environmental efficiencies by moving to smart grids.• Multiple levels of aggregation along the grid.• Ability to handle up to 100,000 events per second from millions of data sources.
STREAMINSIGHT ARCHITECTURE OVERVIEWThe StreamInsight runtime is the CEP server. It consists of thecore engine and the adapter framework. The adapterframework allows developers to create interfaces to eventsources such as Web servers, devices or sensors, and stocktickers or news feeds and event sinks such as pagers,monitoring devices, KPI dashboards, trading stations, ordatabases. Incoming events are continuously streamed intostanding queries in the CEP server, which processes andtransforms the data according to the logic defined in eachquery. The query result at the output can then be used totrigger specific actions.
EVENT SOURCES AND EVENTTARGETS Event Header Event Model• The header of an event • The event model defines defines the event kind the event shape based and event model. on its temporal characteristics. StreamInsight supports three event models: interval, point, and edge.
EVENT KINDThe event kind indicates whether the event is a new eventin the stream or the event is declaring the completeness ofthe existing events in the stream.The INSERT event kind adds an event with its payload into the event stream. In addition, the header of the INSERT event identifies the start and end time for the event.The CTI event kind is a special punctuation event that indicates the completeness of the existing events in the stream. The CTI event structure consists of a single field that provides a current timestamp. It is used to manage out-of-order events or latency in the event stream.
INTERVAL MODELThe interval event model represents an event whose payload is valid for agiven period of time. Event Kind Start Time End Time Payload (Power Consumption) INSERT 2009-07-15 2009-07-15 100 09:13:33.317 09:14:09.270 INSERT 2009-07-15 2009-07-15 200 09:14:09.270 09:14:22.253 INSERT 2009-07-15 2009-07-15 100 09:14:22.255 09:15:04.987
POINT MODELA point event model represents an event occurrence as of a single point intime. Event Kind Start Time End Time Payload (Consumption) INSERT 2009-07-15 2009-07-15 100 09:13:33.317 09:13:33.317 + t INSERT 2009-07-15 2009-07-15 200 09:14:09.270 09:14:09.270 + t INSERT 2009-07-15 2009-07-15 100 09:14:22.255 09:14:22.255 + t
EDGE MODELAn edge event model represents an event occurrence whose payload isvalid for a given interval of time, however, only the start time is known uponarrival to the CEP server. Event Kind Edge Type Start Time End Time Payload INSERT Start t0 ∞ a INSERT End t0 t1 a INSERT Start t1 ∞ b INSERT End t1 t3 b INSERT Start t3 ∞ c … and so on
STREAMINSIGHT ADAPTEROVERVIEWA StreamInsight query processes events pushed through it from its input sources,and pushes results to its output target. In order to communicate with event sourcesand targets, StreamInsight uses input and output adapters to perform the translation.An adapter can be developed using the StreamInsight adapter SDK. Alternatively,one can rely on the built-in adapters for the IEnumerable and IObservable interfacesin .NET 4. Receive/get events from the data source Enqueue events for processing in the engine Insertions of new events Changes to event durations
INPUT ADAPTERAn input adapter instance accepts incoming event streams from externalsources such as databases, files, ticker feeds, network ports, and so on. Theinput adapter reads the incoming events in the format in which they aresupplied and translates this data into the event format that is consumable bythe CEP server.
OUTPUT ADAPTERSYou create an output adapter to receive the events processed by the CEPserver, translate the events into a format expected by the event target, andemit the data to that device. Designing and creating an output adapter issimilar to designing and creating an input adapter.
QUERY TEMPLATEOVERVIEWA query template is the fundamental unit of query composition. It is thestructure the defines the business logic required to continuously analyze andprocess events submitted to the CEP server from the input adapter andgenerate an event stream that is consumed by the output adapter.Query templates are written in LINQ combined with a .NET language.A query is created by binding a query template to specific input and outputadapters. Queries can be managed and monitored.
USE OF QUERY TEMPLATES, QUERIES, AND ADAPTERSThe following illustration shows the CEP query and adapterecosystem. The CEP server consumes and processes the eventwhen the instance of the input adapter is bound to an instanceof a query. The processed data is then pushed to the instanceof the output adapter that is bound to the same queryinstance.
MS EVENT FLOW DEBUGGERStreamInsight ships with a stand-alone debugging tool, the Event FlowDebugger. This tool enables you, as a developer or administrator of acomplex event processing (CEP) application, to inspect, debug, and reasonabout the flow of events through a CEP query.
FEATURES OF MS EVENTFLOW DEBUGGERStep Through Time - Using this feature, you can step through the eventstream one event at a time and watch its progress from one operator to thenext.Root Cause Analysis - Using this feature, you can "look back" at the "rootcause" - or the sequence of operations or changes that caused the event toreach its present condition.Event Propagation Analysis – Using this feature, you can analyze theeffects of this event down the stream either in terms of the changes theparticular event itself goes through, or in terms of how it impacts otherevents, or causes the generation of new events. This feature is the reverseof Root Cause Analysis.
MS EVENT FLOW DEBUGGER PREVIEWThe monitoring dashboard allows you toselect various query execution statistics toanalyze the latency and throughoutcharacteristics of the events that areconsumed and produced and to understandtheir memory requirements.
ORCHESTRATION OVERVIEWOrchestration describes the automated arrangement, coordination, andmanagement of complex computer systems, middleware, and services.
CHOREOGRAPHY OVERVIEWChoreography is a form of service composition in which the interactionprotocol between several partner services is defined from a globalperspective
WEB SERVICESCHOREOGRAPHY INTERFACEWeb Service Choreography Interface (WSCI) is a programming languagethat provides a global, message-oriented view of processes involvingmultiple Web services.
ORCHESTRATION ANDCHOREOGRAPHY WITH .NET TECHNOLOGY
MICROSOFT BIZTALKSERVERMicrosoft BizTalk Server is an Enterprise Service Bus. Through the use of"adapters" which are tailored to communicate with different softwaresystems used in an enterprise, it enables companies to automate businessprocesses
BIZTALK SERVER CAPABILITIESBizTalk Server provides a sophisticated, scalable model for enablingmessage exchange between different systems.
ADAPTATIONSome systems support File Transport Protocol (FTP) or message queueprotocols. Others implement web service interfaces using RepresentationalState Transfer (REST) or Simple Object Access Protocol (SOAP).BizTalk Server separates these concerns from its hub-bus model through theuse of an adaptation layer. It provides a host environment for managingrunning instances of adapters together with a programmatic framework forcreating adapters or extending existing adaptation functionality so that it caninteract with BizTalk Server
MEDIATIONOne of the advantages of the BizTalk Server model is that it providesa natural location for message mediation. Mediation involvesprocessing to decode and encode, validate, and enrich messages.
EXCEPTION HANDLINGIf all messages followed “happy” paths between different systems,integration would be simple and cheap. A core part of the valueproposition of BizTalk Server is its capability to handle the “unhappy”paths where messages cannot flow as intended or desired. This mighthappen because external systems are unavailable, contracts areinadvertently broken, messages are invalid or malformed, they aredelivered in an incorrect order, or for many other reasons.
ORCHESTRATION ANDCHOREOGRAPHYInterchange between different systems and services must often beorchestrated to manage data flow, decision points, parallelism,exception handling, and other requirements. Orchestration servicessupport these requirements. A BizTalk orchestration is a customservice that sits directly on the BizTalk message bus and uses thesame subscription mechanisms used for message routing.Orchestrations can consume and publish messages freely and extendthe built-in message-correlation features of the pub-sub mechanismwith additional capabilities. Orchestrations support long-lived,recoverable business activities through asynchronous serviceinterchange and can nest atomic transactions within these long-livedactivities.
RECEIVE PORTSThe Receive Ports are how messages come into BizTalk. Messagescan come into BizTalk by any means of transport like HTTP, SOAP,SQL, file, EDI; just like the way packages arrive via planes, ships,buses, or trucks. How they are transported does not change how theyare processed later. This is a very powerful feature because it givesyou the ability to change the transport type and the source of the dataeven after your application is deployed, without changing yourapplication implementation.
ADAPTORSAdaptors are the BizTalk way of handling additional transport. Just likeyou can have a private plane fly your package in, you can write yourown adaptor for a custom transport, like say POP3. So there is alwaysscope for buying or writing your own custom adaptor to integrate withall the other available transports. For instance, you can have a DB2adaptor or a SAP adaptor.
RECEIVE PIPELINEThe Receive Pipeline contains different mechanisms like "Decoding" (ifencoded), "Disassembling" (if there are multiple parts), and "PartyResolution". This is similar to the unpacking and identifying the sender, inour example. Just like we have an express lane for certain packages, thereis a "Pass Through" pipeline facility in BizTalk which is essential a smoothtransfer to the Message Box. Just like you can have special handlinginstructions for certain packages, you can write custom pipeline componentsusing the framework.
MESSAGE BOX ANDSUBSCRIPTIONSThe BizTalk Message Box lies in the heart of the BizTalk messagingsubsystem; this is like our central store. This message box is, in fact,realized as a SQL Store; so this way, BizTalk ensures reliability foryour messages in terms of guaranteed delivery. Even if a node inBizTalk fails, the message will be picked up by another node in thefarm. BizTalk also ensures that it does not delete the file from a pickupfolder location while using file/any transport, unless it is persistedcompletely in SQL.
ORCHESTRATIONSOrchestrations are processes that are defined in the BusinessProcess Execution Language (BPEL). This could be something like,say: if quantity is greater than threshold, apply discount, else standarddiscount. At this stage, you can also change the format of themessage using Transformation Maps, or you can call some other webservice to do an operation like credit card verification. This is like ourexample in which we have the customs inspection process andchecking with the bank if the excise payment is cleared.
BUSINESS RULES ENGINEBusiness Rules Engine is used by BizTalk to load the current policieswhich are a collection of Rules from the Orchestration. These rulesare kept separately as these could change from time to time. In ourexample above, this would be something like, during Olympics therecould be certain relaxations in guidelines or certain promotions ineffect. These change from time to time, and they are kept separate.This is the idea behind the Rule Store in BizTalk.
SEND PIPELINEThe "Send Pipelines" are just the opposite of the "Receive Pipelines"and they do operations like "Encoding" the message, or "Assembling"or "Signing" the message. This is like packing and sealing a packagebefore we send it. Just like we have different types of packaging likebubble wrapping and cushion mailers, there could be different pipelinecomponents at this stage.
SEND PORTSSend Ports also handle transport like receive ports, but their directionis the reverse. Here, the message direction is from the Message Boxto the destination. Send Ports can subscribe to a message directlyfrom the Message Box, without passing through the orchestration.This is like having a package coming in from a certain shipper whichhas a prior clearance attached with it. This is called Content BasedRouting (CBR).
BIZTALK SERVER FORORCHESTRATION AND CHOREOGRAPHY
ORCHESTRATED TASK SERVICES APPEARIDENTICAL TO TASK SERVICES WHENVIEWED FROM A LOGICAL SERVICE LAYERSPERSPECTIVE
WF VS BIZTALK SERVERSupport for patterns mapped to different parts of the Microsoft technology platform. (1 = Nativeand configuration-only support, 2 = Can be created, 3 = Not applicable or not supported.) Notethat with both WF and BizTalk Server, State Repository is applied with the use of SQL Server. Orchestration Patterns WF BizTalk Server Process Abstraction 2 1 Process Centralization (tools) 2 1 State Repository 2 1 Compensating Service Transaction 2 1 Rules Centralization 2 1 Atomic Service Transaction 2 1 Data Model Transformation 2 1
WF VS BIZTALK SERVER A comparison of orchestration-related characteristics provided by WF and BizTalk Server platformsCharacteristic WF BizTalk ServerAPIs .NET, WF object model XML, .NET orchestration library, BizTalk object modelarchitecture foundation .NET XML and .NETautonomy orchestrations are autonomous or embedded in other applications orchestrations are fully autonomouscomposability composition of Web services and custom WF activities, and orchestrations are composition of Web services and legacy applications, and compostable as services or directly orchestrations are compostable as services or directlycost free of charge as part of the .NET framework licensed by processordiscovery services are discoverable via published WSDL definitions or inside the developer services are discoverable via published WSDL definitions toolextensibility high extensibility for customized execution, state persistence and support for extensibility to support additional transports, protocols, and other protocols and transports via custom activities message processingformal contracts light support for service contracts in the form of .NET interfaces strong support for XML Schema and WSDL-based contractsindustry standards Support for SOAP, WSDL. WS-* strong support for XML, SOAP, WSDL, and WS-* with WCF integration and limited support for WS-BPELmessaging general support for asynchronous messaging, resource management, and XML-based message exchangesreusability reusable via Web services and directly inside the orchestration engine (other reusable via Web services, queue endpoints, file drop, and forms of reuse possible with custom development) directly inside the orchestration enginerules engine rules based on facts in .NET objects complex rules based on facts in XML documents, .NET objects, and database recordsscalability medium scalability through custom extensions for persistence and activity high scalability through clustering and distributable scheduling functionalitystate management process state is stored in memory, SQL Server, or custom data store process state is stored in SQL Servertools re-host-able orchestration design, Web service publishing orchestration design, data mapping, Web service publishing,
INTEGRATING MSSTREAMINSIGHT WITH BIZTALK SERVER
INTEGRATING CONCEPT (1)We can extend the use of the native WCF adaptersto interact with a StreamInsight server byincluding a series of pipeline components thatpromote StreamInsight specific information intothe BizTalk runtime. For instance, these pipelinecomponents could promote specific properties suchas the name of the StreamInsight application,query and event streams associated with a certainevent message. BizTalk applications could use thisinformation to dynamically adapt the interactionswith the StreamInsight server.
INTEGRATING CONCEPT (2)The entire StreamInsight infrastructure ishosted within BizTalk Server. Forinstance, we can use StreamInsight to BizTalkfilter the events produced by a specific Server StreamInsight Adapterline of business system. We can StreamInsig ht StreamInsig StreamInsigaccomplish this by implementing a WCF- Pipeline Component ht BizTalk Query ht Adapterbased adapter that hosts the AdapterStreamInsight runtime and executesfiltering queries against the dataproduced by the LOB system. The finalresult of those queries can be processedby BizTalk Server using a StreamInsight
CONSUMING STREAMINSIGHTEVENTS FROM BIZTALKSERVERIn this scenario, BizTalk Server will use the WCF adapters as thefundamental mechanism to interface with the WCF service. This is mostlybecuase the StreamInsight application is exposed as a server-centric modelfront ended by the StreamInsight management WCF service. BizTalk Start Query StreamInsight Server WCF Server Send Adapter Events WCF StreamInsight Receive Query WCF Adapter Adapter
BIZTALK SERVER AS ASTREAMINSIGHT SUBSCRIBERFollowing the model illustrated above, BizTalk Server could receivemessages using any of the available receive adapters such as FTP,HTTP, WCF or MSMQ. The use of WCF is particularly attractive in thisscenario because the channel/encoder model abstracts theinteractions with different transports and encoding mechanisms usingthe same programming model. What this really means is that we canpotentially use a single StreamInsight WCF output adapter to interactwith BizTalk Server using various transports such as HTTP or TCP orencoding mechanisms such as SOAP or Binary
MONITORING STREAMINSIGHTEVENTS USING BIZTALKSERVER BAMIn its current release, StreamInsight does not provide a default long-termmechanism to track events processed by the StreamInsight adapters.However StreamInsight complements that with a very flexible adapter modelthat allows developers to build specific adapters that can track the eventsinto a specific data store. StreamInsight Server BAM Streamin Begin/Update/End g API StreamInsigh Quer y t BAM activity Adapter
TRACKING STREAMINSIGHTEVENTS USING BAMDepending on the application scenario, we can use different BAM activitymodels to track the events processed by StreamInsight. Some of thesemodels can provide information about the StreamInsight application purelyfrom an operational standpoint while others can surface the informationmore from a business intelligence/analytics standpoint. Following our trafficmonitoring sample, a BAM model could capture BI-centric information suchas the average speed of vehicles in area of a speedway while another BAMmodel can provide more operation-centric information such as the number ofevents captured during certain period of time.
REPLAYING STREAMINSIGHTEVENTS USING BIZTALKSERVER BAMBizTalk Server BAM provides a great infrastructure to track and monitorevents processed by a StreamInsight application. Given that tracking is afundamental component of any event replaying strategy, we can conceive ofan event replaying mechanism that uses BAM as the fundamental eventsource. StreamInsight Server BizTalk Server StreamInsight Events BAM Quer y BAM Input Primary Adapter Import DB
INTEGRATINGSTREAMINSIGHT WITH LOBSYSTEMS USING THEBIZTALK LOB systems share a lot ofPACK in aspectsThe interactions with ADAPTER commonalitiessuch as interaction patterns, metadata management or connection pooling.These commonalities are abstracted by a Microsoft programming modelknown as the WCF LOB Adapter SDK which provides the fundamentalbuilding blocks for implementing WCF-based adapters that can interact withLOB systems. StreamInsight Application SAP StreamInsight SQL Query WCF LOB Server Adapter Oracle EBS
EXTENDING STREAMINSIGHTQUERIES WITH BIZTALKSERVER BUSINESS RULESWe need to extend the StreamInsight BizTalk Server BREqueries with a declarative condition-actionpredicate that can be extended andmodified without changing the syntax of Policy Policy Policythe query itself. This form of declarativecondition-action modeling is the essenceof business rules technologies such asthe Business Rules Engine (BRE)included in BizTalk Server. User User User Defined Defined Defined Function Aggregate Operator StreamInsight Query
MICROSOFT CEP Rahadian Dustrial Dewandono TECHNOLOGY Progres Tesis 28-10-2012