Drive Smarter Decisions with Big Data Using Complex Event Processing


Published on

This webinar described what CEP is and how it has been deployed in several client organizations to provide more agile, cost-effective and real-time integration across multiple data stores including:
Analysis of large amounts of complex, unstructured and semi-structured data
Harnessing the power big data, social/mobile data stores and BI projects for real-time decision making
Predicting events before they happen based on patterns and rules

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Drive Smarter Decisions with Big Data Using Complex Event Processing

  1. 1. 1 Drive Smarter Decisions with Big Data Using Complex Event Processing Eric Roch ▪ July 16, 2013
  2. 2. Our Speaker Eric Roch • Principal for Perficient’s Connected Solutions Practice – SOA – Cloud - Mobile • 25+ years of experience in Information Technology • Previous roles include: executive level management, technical architect, and software development in top tier technology organizations including TIBCO Software and Deloitte Consulting • Strategic planning and commercialization of methodologies and software • Technical architecture for multi-platform application and systems integration at organizations • Guest speaker and author 2
  3. 3. 3 Perficient is a leading information technology consulting firm serving clients throughout North America. We help clients implement business-driven technology solutions that integrate business processes, improve worker productivity, increase customer loyalty and create a more agile enterprise to better respond to new business opportunities. About Perficient
  4. 4. 4 • Founded in 1997 • Public, NASDAQ: PRFT • 2012 revenue $327 million • Major market locations throughout North America • Atlanta, Boston, Charlotte, Chicago, Cincinnati, Cleveland, Columbus, Dallas, Denver, Detroit, Fairfax, Houston, Indianapolis, Los Angeles, Minneapolis, New Orleans, New York City, Northern California, Philadelphia, Southern California, St. Louis, Toronto and Washington, D.C. • Global delivery centers in China, Europe and India • ~2,000 colleagues • Dedicated solution practices • ~85% repeat business rate • Alliance partnerships with major technology vendors • Multiple vendor/industry technology and growth awards Perficient Profile
  5. 5. 5 Business Solutions • Business Intelligence • Business Process Management • Customer Experience and CRM • Enterprise Performance Management • Enterprise Resource Planning • Experience Design (XD) • Management Consulting Technology Solutions • Business Integration/SOA • Cloud Services • Commerce • Content Management • Custom Application Development • Education • Information Management • Mobile Platforms • Platform Integration • Portal & Social Our Solutions Expertise
  6. 6. Agenda • Big Data Trends and Categories • Analysis of large amounts of complex, unstructured and semi- structured data • Harnessing the power big data, social/mobile data stores and BI projects for real-time decision-making • Predictive Analytics and Event Processing for decision management 6
  7. 7. Evolution of Big-Data • Mainframe • Client-Server • Web • Mobile • Cloud • Social • Internet of Things 7 Source:
  8. 8. State of Technology Adoption 8
  9. 9. Categories of Big-Data 9 Source: splunk
  10. 10. 10 Characteristics of Big Data • Data in motion analyzes data before storage • Data at rest analytics are based on a historic snapshot Source: IBM
  11. 11. Big Data Technologies • MapReduce frameworks implements pattern recognition though classification algorithms – what happened? • Data Visualization presents information views graphically and/or statistically – what happened and what might happen? • Predictive Analytics uses mathematical pattern recognition in historical data – what’s going to happen? • Complex Event Processing uses pattern recognition on event streams and can apply rules to predict logical events – what is going to happen and what do we do about it? 11 Source: TIBCO Spotfire
  12. 12. Log Analysis vs. Business Analytics • Ingest – Versus ETL • Big Data – Bidirectional integration with Hadoop • Query language – MapReduce function on unstructured data • Drill anywhere – Investigate on all the data versus a predefined schema or cube • Information discovery – Discover relationships based on patterns in the data • Ad-hoc versus dimensional – Log analysis is not based a predefined structure based a point-in-time set of requirements 12 Source: splunk Implementation
  13. 13. Predictive Analysis • Predict the future state of variables associated with business goals • Describe human detectable patterns • Data mining techniques • Rule Discovery – describe • Pattern Discovery – describe • Clustering – describe • Classification – predict • Regression – predict • Deviation – predict 13 Source: InformationBuilders
  14. 14. Event-driven Architecture • Event-driven architecture (EDA) is a software architecture pattern promoting the production, detection, consumption of, and reaction to events • Complex event processing (CEP) consists in processing many events happening across all the layers of an organization, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time. 14
  15. 15. A Holist View of Decision Optimization 15 Source: James Taylor
  16. 16. Barriers to Big Data Analytics • Information throughout the enterprise • Silos of data • Decentralized control • No one single solution • No cohesive strategy • Legacy systems difficult to make part of the strategy 16
  17. 17. SOA and Integration 17 HTTP HTTP/S SOAP/HTTP SOAP/JMS FTP SMTP EMS/JMS EDI Enterprise Service Bus (ESB) Credit Check Place Order Check Quantity Issue Invoice Alert Large Order Notify Customer Process Order Check Customer Account • Connect • Transport • Route Services Backbone Enterprise Service Bus (ESB) • Mediate • Event notification • Exception Handling Abstract the data format and the behavior of legacy systems to publish events
  18. 18. The SOA Information Gap ″SOA by itself does nothing to address the question of how data should be managed within this architecture. ... data remains fragmented despite the best efforts to rationalize it. This issue is motivating the creation of a new class of middleware that Forrester calls the information fabric.” The Forrester Report Information Fabric: Enterprise Data Virtualization 18 ″ You will waste your investment in SOA unless you have enterprise information that SOA can exploit.” Gartner
  19. 19. Data Virtualization Layer 19 Data Warehouse Packaged Application Legacy Application • Master Data Management and Data Virtualization • Data federation for consistent packaging of data • Leverages understanding of metadata relationships • Applies consistent rules to data • Centralized control and maintenance • Flexibility to change information sources and formatsar Create Quote Process Flow Trigger Create Estimate Process Flow Trigger Information as a Service (Shared Metadata)
  20. 20. Business Process Management and Workflow • The term Business Process Management refers to activities performed by businesses to optimize and adapt their processes. • Although it can be said that organizations have always been using BPM, a new impetus based on the advent of software tools which allow for • Direct execution of the business processes without a costly and time intensive development of the required software. • In addition, these tools can also monitor the execution of the business processes, providing managers of an organization with the means to analyze their performance and make changes to the original processes in real-time • BPM has a tight link to componentized and service oriented IT architecture 20
  21. 21. BPM and Services 21 Service X Service U Service Y Service Z Human Task A Human Task D Human Task F Human Task B Human Task C Workflow Invoke Invoke Invoke Invoke • Workflows implement business processes • Workflow engine navigates the network of activities • Typically invoking automatic (service choreography) or manual activities • Mostly visual programming/modeling
  22. 22. Process Orchestration Layer - BPMS • Designer and repository • Execution engine • Database – case state • Database – case history • Case history reporting – KPIs, task timings, timings by role • Starting a new case is resource intensive 22 State Management Design Repository Process History Execution Engine
  23. 23. BRMS Architecture • Manages the lifecycle of the rules • Author rules • Execute stateless rules • Statistical reports about rule execution • Rule execution is embedded in business applications – e.g. a decision service 23 Source: IBM
  24. 24. Using BRMS in BPMS • Lifecycle of rules are external to the BPMS • Business processes “call” rules e.g. via services • Rules make a stateless decision • Rules have to have a driving workflow or application 24 Rule Repository Rule Engine Rule Authoring BPMS
  25. 25. CEP Architecture 25 • Consistent operational rules applied to business events • Declarative rules and implicit state management • Event driven, non-linear, closed-loop, agile business processes • Component failure (fine grain) – outage (logical /predictive) Concept State Rule Bases BPMS CEP Engine Logical Events – Notifications, Consequences Actions SOA Business Applications Fine-grain Business Events System(s) of Record Integration and Business Components Flexible Workflows ESB Event Channel(s)
  26. 26. CEP High-level Architecture Patterns • Situation awareness is about "knowing" the state of the product, person, document, or entity of interest at any point in time. • Sense and respond is about detecting some significant fact about the product, person, document or entity of interest, and responding accordingly • Track and trace is about tracking the product, person, document or entity of interest over time and tracing pertinent facts Source: TIBCO Software 26
  27. 27. CEP Benefits • Manage events, state transitions, and event correlation reducing code in the application layer • Control logic • Persistence logic • Business Rules • Drive business processes with correlated events • Create operational efficiencies with the same events and drive longer-term strategic decision support • Less complex rules with the event driven concepts • Persistent business objects • Known context of the event 27
  28. 28. CEP Roadmap and Methodology • Target critical business events for process automation and decision optimization • Inventory relevant events, rules and concepts • Identify candidate business (sub)process to automate • Project LoE(s) and Roadmap • Integrate systems used in key business processes • Instrument applications to emit events • Define process activities • Mine candidate rules – code and predictive analytics • Model events, rules and concepts • Iterate through business processes 28
  29. 29. Telco CEP Case Study • Provisioning • Track missing provisioning notifications and sends complex events to Billing Ops on missing notifications • Open Orders • Used to track Orders that have not been closed due to a missing event. CEP detects the missing event and auto closes the Order in the Payment Processing system. • Pending Payments • Used to process payments that are pended by the payment processing system. CEP stores the payment data within the cache and closes the payment at a later via SOA. • Customer Coupon Offers • CEP is used to monitor, alert and prevent Stores from going over a threshold of the discount funds that they are allocated. • Logistics Alerts • CEP is used to track location and Product updates from logistics and to invoke GEH to republish failed messages • CEP Framework • Created CEP developer guide and logging framework to log and search events in Splunk 29
  30. 30. TIBCO BusinessEvents is a CEP Platform • Platform Features • Stateful Rule Engine • State for Temporal Rules • Persistence Object Manager • High Performance Rules Engine • TIBCO integration platform • 150+ Adapters • Channels approach • Continuous queries and Event Stream Patterns • Decision Manager for Business User Rules Authoring (can write can upload rules from Excel!) • Distributed Agents Architecture for dynamic scalability • Data Grid • BE Views (Dashboard) 30 Source: TIBCO Software
  31. 31. Event Enabled Enterprise 31 Transformation Projects 2009-2011 Business Solutions 2011-2012 • Last minute addition • Concept to launch in 6 weeks • Decoupled architecture – no risk Implementation • Customers: Ensure timely activations • Operations: Immediate visibility into order provisioning times Customer Service • Stores: Reduce inventory issues • Operations: Automate fall out of shipping notices Supply Chain • Customers: Added security to account access • Operations: Report/alert on suspicious access attempts Security/CPNI • Customers: More access to discounts • Revenue: Manage discount limits by individual location Retail Sales • Customers: Use IVR to set up payment agreements • Customer Service: Reduced call center volumes Self-Service Event Enabled 2013 • Proven success in real-time, value- based activities – ready for prime- time! • Sense. Model. Respond. The Tipping Point • Adapt and respond to real-time customer behaviors/experiences • Example: Proactive retention offers Fast Response • Abandon one-size-fits-all customer limitations • Enable event-driven decisions for best customer experience Customer Flexibility
  32. 32. CEP Solution Architecture 32
  33. 33. CEP References • event-processing-reference-materials-48348 • event-processing-patterns-message-routing- 48987 • content/uploads/2008/02/1-a-short-history-of-cep- part-1.pdf • content/uploads/2008/07/2-final-a-short-history-of- cep-part-2.pdf 33
  34. 34. Questions 34
  35. 35. Daily unique content about content management, user experience, portals and other enterprise information technology solutions across a variety of industries. 35
  36. 36. Thank you for your time and attention today. Please visit us at 36