Advanced Logging and Analysis for SOA, Social, Cloud and Big Data


Published on

In this presentation, Perficient experts examined techniques, frameworks and products to log and analyze complex systems and events. Discussions included:

• Logging and exception management as service within your SOA
• Frameworks for event and transaction monitoring
• Event sense and response techniques
• Highly scalable logging product
• Logging in the context of business transaction and events

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Monitoring systems should catch “should not occur” conditions not just machine and platform performance or outage issues
  • In many cases humans do not recognize the glitch. Or much damage is done before the glitch is noticed,
  • We need layers of monitoring – a transaction monitor to monitor applications with escalations to humans based on rules
  • Web Service Distributed Management Common Base Event Simple Network Management Protocol (SNMP)
  • What’s missing in this picture is business events. I will come back to that.
  • Splunk – Google for IT data but what about the use case Google for Business Events
  • Events and big data.A single complex event can be an abstraction of millions of raw events A tweet is an event – but we don’t likely care about individual tweets we can about consumer sentiment – a complex event in time – event processing can drastically reduce the amount of big data Example the real time trending of voter sentiment during the presidential debate – a line graph in time
  • 10s of millions of events logged as 3M complex events per day
  • Advanced Logging and Analysis for SOA, Social, Cloud and Big Data

    1. 1. Advanced Logging and Analysis forSOA, Social, Cloud and Big Data
    2. 2. About PerficientPerficient is a leading information technology consulting firm serving clientsthroughout North America.We help clients implement business-driven technology solutions that integratebusiness processes, improve worker productivity, increase customer loyalty and createa more agile enterprise to better respond to new business opportunities.
    3. 3. Perficient Profile Founded in 1997 Public, NASDAQ: PRFT 2011 Revenue of $260 million Major market locations throughout North America — Atlanta, Austin, Charlotte, Chicago, Cincinnati, Cleveland, Columbus, Dallas, Denver, Detroit, Fairfax, Houston, Indianapolis, Los Angeles, Minneapolis, New Orleans, Philadelphia, San Francisco, San Jose, Southern California, St. Louis and Toronto Global delivery centers in China, Europe and India 2,000+ colleagues Dedicated solution practices 87% repeat business rate Alliance partnerships with major technology vendors Multiple vendor/industry technology and growth awards
    4. 4. Our Solutions Expertise & ServicesBusiness-Driven Solutions Perficient Services• Enterprise Portals  End-to-End Solution Delivery• SOA and Business Process Mgmt  IT Strategic Consulting• Business Intelligence  IT Architecture Planning• User-Centered Custom Applications  Business Process & Workflow• Interactive Design Consulting• CRM Solutions  Usability and UI Consulting• Enterprise Performance Management  Custom Application Development• Customer Self-Service  Offshore Development• eCommerce & Product Information  Package Selection, Implementation Management and Integration• Enterprise Content Management  Architecture & Application Migrations• Enterprise Resource Planning  Education• Management Consulting  Interactive Design• Industry-Specific Solutions• Mobile Technology Perficient brings deep solutions expertise and offers a complete set of flexible services to help clients• Security Assessments implement business-driven IT solutions
    5. 5. Our SpeakersEric Roch • Principal SOA | Mobile | Cloud • 30+ years of experience in various aspects of Information Technology including: • IT executive level management within industry and consulting • technical architecture • application and systems development. • He has also been an IT industry speaker and author for many years. Ben Hahn • Sr. Technical Architect • 20+ years of experience in various aspects of Information Technology. • Software Solutions Architect • Enterprise Infrastructure Architect • Product Management • Software community contributor
    6. 6. Agenda• Glitches and outages drive the need for more comprehensive logging and monitoring• Big Data and Logging• Events and Logging• Logging and exception management as a service• Explicit Transaction Monitoring• Instrumentation for Logging• Logging and Exception Management Framework and Demo 6
    7. 7. The Need for Application Monitoring• Systems are distributed, increasingly more complex and we are progressively more dependent upon them• Application glitches are costly in dollar terms, customer loyalty and brand image: – Department of Commerce study found software errors cost U.S. economy $59.5 billion annually – Information Week says IT downtime costs $26.5 billion in lost revenue – The 2003 North America blackout was triggered when a local outage was not detected by monitoring software – Glitches in the reporting of prices on the NYSE may have caused the “flash crash” – Knight Capital Group Inc. lost $440 million from software trading error – about $10M a minute for 45 minutes – Programming error at Visa Debit Processing Services caused a billing error of $23,148,855,308,184,500 7
    8. 8. Humans Monitoring for GlitchesTourist follows GPS into Bay UK woman follows GPS into river 8
    9. 9. Application Monitoring“Oh yeah, well I’m watching you watch him” Source: E*TRADE Baby 9
    10. 10. Machine Monitoring• Log analysis is well established for IT operations, security and compliance• Enterprise management software standards to detect platform and network problems• LogLogic appliance can ingest up to 250,000 events per second with high speed filtering and routing capabilities• Splunk provides general-purpose search language for analysis and reporting for time-series data using the MapReduce framework 10
    11. 11. Finding Glitches in the DataSource: splunk 11
    12. 12. Log Analysis vs. Business Analytics• Ingest – Versus ETL• Big Data – Bidirectional integration with Hadoop• Query language – MapReduce function on unstructured data• Drill anywhere – Investigate on all the data versus a predefined schema or cube• Information discovery – Discover relationships based on patterns in the data• Ad-hoc versus dimensional – Log analysis is not based a predefined structure based a point-in-time set of requirements Source: splunk Implementation 12
    13. 13. Business Events and Event Processing• Event-driven architecture (EDA) is a software architecture pattern promoting the production, detection, consumption of, and reaction to events.• Complex event processing (CEP) consists in processing many events happening across all the layers of an organization, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time. 13
    14. 14. CEP High-Level Use Cases• Situation awareness is about "knowing" the state of the product, person, document, or entity of interest at any point in time.• Sense and respond is about detecting some significant fact about the product, person, document or entity of interest, and responding accordingly• Track and trace is about tracking the product, person, document or entity of interest over time and tracing pertinent facts Source: TIBCO Software 14
    15. 15. CEP Architecture Fine-grain• Millions of raw events can CEP Business Events Engine be represented in one complex event Concept State• Component status (fine grain) – outage (logical Rule /predictive) Bases• Events are process with Logical Events – Notifications, Consequences Actions BPMS ESB declarative rules and Event Channel(s) implicit state management• Events drive agile business processes Flexible Workflows Integration and SOA Business Components System(s) of Record Business Applications 15
    16. 16. Business Event Logging with CEP 16
    17. 17. Application and Business Event Logging• Monitoring logical transaction state vs. monitoring transaction through-put• Exception management vs. error logging• Explicit application event logging versus machine logging• Application instrumentation to a framework• Exception management and log correlation – Transaction and conversation IDs – User defined fields – XML processing - XSD, XSLT and XPath – Agents and appenders – Time and dependencies 17 17
    18. 18. Logging and Exception Management Design Goals• Highly configurable and user friendly GUI• High-speed• Non-blocking asynchronous calls• Open architecture / standards support• Technology agnostic• Service-oriented 18
    19. 19. What is GAL?Generic Audit Logger• A configurable message logger to record, view or reply message• Fully configurable via a GUI• Map logs to custom fields• Filter and query logs• Supports message query from GEH• Works standalone and supports our exception handler 19
    20. 20. GAL DEMO
    21. 21. What is GEH?• Generic Exception Handling• Provides – Centralized Exception Handling – Process/Workflow Exception Resolution – Issue Resolution Knowledge Base – Centralized Audit Trail – Centralized Transaction Monitor• Benefits – Standardized Issue Resolution – Minimal Project Based Exception Handling – Dynamic Views / Dynamic Workflows – Faster Learning Curve – External to Internal Exception Mappings 21
    22. 22. GEH DEMO
    23. 23. Transaction Monitoring via Logging• Leveraging GAL and GEH to define a transactionUsing recognition rules, each log entry can define a• Transaction demarcation – begin or end• Transaction context – using a conversation identifier• Transactional checkpoints – define completed stages in a transaction.• Transactional errorsOnce a transaction is defined we get• Monitoring• Instrumentation• Error tracking via an error handler like GEH• State and through-put 23
    24. 24. Transaction Capturing via Logs 24
    25. 25. Events via TransactionsNow that we have transactions we can also define events• Each log entry can now also define an event• Events can be aggregated to make more logical events• Events can also be generated for “non-events” e.g. if number of transactions do not reach a certain threshold (sales quotas) 25
    26. 26. Transaction Event GenerationAnd eventually published out to a CEP 26
    28. 28. Q&A