Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Gain insights into your business operations with BPM Analytics

1,027 views

Published on

We discussed the ability for IBM BPM to send business events into analytics + visualization framework such as Elasticsearch + Kibana.

Published in: Software
  • Be the first to comment

Gain insights into your business operations with BPM Analytics

  1. 1. Gain insights into your business operations with BPM Analytics Allen Chan Chief Architect, Business Process Management
  2. 2. IBM Cloud University 2017 | October Please Note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
  3. 3. Agenda • Digital Process Automation • Business Monitoring • Elasticsearch + Kibana with IBM BPM 3 9/29/17
  4. 4. … to deliver the end-to-end customer journey … built on an extensible framework for innovation and insight … enabled by a platform for rapid delivery and iteration, led by the business Digital Process Automation is… IBM’s platform for rapidly creating business–led process apps that deliver new levels of customer experience and digital self service Key attributes of the Digital Process Automation platform: Customer focus Speed Scale
  5. 5. Digital Process Automation capabilities Define customer journeys and business operations to discover new opportunities for innovation Design, automate and govern business operations to deliver a scalable, consistent customer experience Analyze data and events to augment knowledge work and take the right action at the right time Actionable Insights Collaborative Discovery Business Automation Digital Process Automation
  6. 6. To get Actionable Insights – we need business monitoring • Increase profitability – Reduce costs by ensuring timely completion of work – Real-time insight into operational processes for • Improved operational efficiency • Dynamic business response • Increase customer satisfaction – Proactive alerts to business situations – Access to relevant information to resolve situations • Increase team productivity – Monitor overall team productivity – Monitor individual worker productivity – Identify high performing workers to mentor lower productivity workers ! CICS ! IMS ! WAS Executive! Management! Customer! Service! Risk Management
 Teams! Finance! and Ops! Account 
 Administration! 6
  7. 7. Drill down to specific situations Track your KPIs and get alerted as situations occur Trend analysis and prediction Analyze a range of business data with BAM Take action faster with real-time information on personalized dashboards § Drill down to investigate why John Smith’s new bank account request has not been processed within the specified service level agreement § See if the average amount of time to open a new bank account is within an acceptable or desired range – from your desktop or mobile § Compare new account opening KPIs across time periods to manage resources more effectively and predict future problems 7
  8. 8. 1. Track 2. Store 3. Report 4. Analyze 5. Respond 8 9/29/17 5 Steps to manage your Operational Intelligence
  9. 9. • Built-in data store for business events. • BPM will send out business events with: • Auto-Tracking • Tracking Points • Timing Intervals • Built-in Performance Dashboard • Team Performance Dashboard • Process Performance Dashboard 9 9/29/17 Performance Data Warehouse
  10. 10. • Once turned on, will track all process level events, e.g pre- and post- task execution. • Individual variable values can also be tracked. 10 9/29/17 Auto Tracking A database table by the “Autotracking name” will be created in the PDW database.
  11. 11. 1. Add custom Tracking Groups with a specific set of variables. 2. Specify in the BPD where to write Tracking Groups data before or after a certain activity. 11 9/29/17 Tracking Points & Tracking Groups In this case, we will write an entry in the Tracking Group table TG_TRAVELREQUEST
  12. 12. Using Intermediate Event, one can create timing interval with tracking groups to measure elapsed time between any two events. 12 9/29/17 Tracking Points
  13. 13. BPM has a built-in Performance Dashboard that will allow you to visualize the information about process performance. 13 9/29/17 Performance Dashboard
  14. 14. 1. Customize the information we want to show in the Dashboard? 2. Federate information across multiple BPM systems? 3. Perform data mining and analytics with business data? 4. Roll Your Own analytics solutions by using 3 rd part analytic software? 14 9/29/17 But, what if we want to …
  15. 15. BPM Analytics Tech Preview • Enable the BPM users to gain insight of the processes with big data and analytics • Performance overhead ~ 10% or less • Expose the rich information of the processes – system view and business view • Near real time • Archive and restore historical data to form new monitor model and new analysis • Out of the box sample dashboards • 3rd Party Analytics • Integrate the data which do not come to the BPM system • Publish the process data to external data and analytics solutions provided by IBM, customer and partners • For evaluation only, not for production use
  16. 16. Process Server JMS Business Events BPM Cell defconfig.x ml BPMEvent Emitter Kibana Elasticsearch Topology reference Process Server JMS Business Events BPM Cell defconfig.x ml BPMEvent Emitter Not packaged by BPM Event Summary Agent Raw Event Combined Event BPM Pre- defined Dashboard BPM Analytics components Object Storage Event Summary Agent Archive Archive Restore Restore Optional • Two level storages, Elasticsearch for the online storage, the Object Storage is the archive storage • Support multiple object storage interfaces, like: OpenStack Swift and Amazon S3(to be ready) • Raw events will be stored at the archive object storage in the bulks • Archive storage offline by accident will not impact the analytics function • Archive operation will avoid peak time automatically • Archived events can be selected by the time range to re-build the indices at the ES
  17. 17. Topology – option 1 • Kafka can be configured as the event cache in front of all event consumers Process Server JMS Business Events BPM Cell defconfig. xml BPMEvent Emitter Kafka Not the OOTB capability Kibana Elasticsearch Event Summary Agent Raw Event Combined Event Object Storage Event Summary Agent Archive Archive Restore Restore
  18. 18. Topology – option 2 • Kafka can be the message broker for any analytics solutions Process Server JMS Business Events BPM Cell defconfig. xml BPMEvent Emitter Kafka Not the OOTB capability Any Analytics Solution Not the OOTB capability
  19. 19. BPMEventEmitter Design • If the BPM enable emit the JSON format DEF event, the “XML to JSON” will be skipped automatically • ES and Kafka are event consumers, they can be enabled together or separately • Events emit to the different consumers are identical, it’s easy to build up event tunnel between: • Kafka to Analytics – customized analytics solution • Kafka to ES – event cache in front of BPM Analytics • Kafka to Object Storage – customized archive solution MDB XML to JSON Def event (xml or json) To Elasticsearch To Kafka BPMEventEmitter config Activity transformer Process transformer
  20. 20. Activity Event • Contains all the information from activity level events. • One activity level event generates one Activity Event document • One task has several Activity Event documents Activity Summary • Contains information that represents one task, created by combining information from several Activity Event documents. • Simple flat structure – simplify search query and visualzation by Kibana 20 9/29/17 Elasticsearch Indexes
  21. 21. Raw Event Type (immutable) Elasticsearch Data model at Elasticsearch Combined Type (mutable) Kibana Elasticsearch Index Activity Event Process Event Activity Summary Process Summary MDB Transform (aggregate) Aggregatipn Handler • Aggregation is the batch processing with low latency • Not stream processing because the aggregated event’s time range can be weeks, or even months • Data as Combined Types is mutable: • Update the process status for example from the running to finished • Data as Raw Event Types is immutable • Support re-derive Combined Types over again based on the Raw Event Types Elasticsearch Index Progress Record
  22. 22. Process Event • Contains all the information from a Process instance as it happens. • One process event generates one Process Event document • One process instance can have several Process Event documents Process Summary • A single document containing the most useful information of a process instance by combining data from several Process Event documents • One Process Summary document represents one process instance • Simple flat structure – simplify search query and visualization by Kibana 22 9/29/17 Elasticsearch Indexes { "cellName":"BPM1", "eventVersion": "2010-11-11", "eventKind": "bpmnx:PROCESS_TERMINATED", "occurrenceTime": "2017-02-06T13:24:54.283+08:00", "processApplicationId": "532cc340-7d17-4ce7-a1c1-6e8380fec134", "processApplicationVersion": "2064.2142be53-2334-4338-a194-4e3861992ae2", "processApplicationName": "Hiring Sample", "processApplicationSnapshotName": "v5", "startingProcessInstanceFullId":"cd974bd9-0ba2-499d-abdc-319c3be66993.2064.ca663330- 14b8-40c2-8a57-2c63a173c677.44000000435", "processFullId": "cd974bd9-0ba2-499d-abdc-319c3be66993.2064.ca663330-14b8-40c2-8a57- 2c63a173c677.44000000435", "processName":"Standard HR Open New Position", "processInstanceId":44000000435, "processState":"Terminated", "processVersion":"2064.ca663330-14b8-40c2-8a57-2c63a173c677", "processPOId":"cd974bd9-0ba2-499d-abdc-319c3be66993", "processSnapshotName": "v5" } { "cellName":"BPM1", "processStartTime":"2017-02-06T13:00:00.384+08:00", "processTerminatedTime": "2017-02-06T13:24:54.283+08:00", "processApplicationId": "532cc340-7d17-4ce7-a1c1-6e8380fec134", "processApplicationVersion": "2064.2142be53-2334-4338-a194-4e3861992ae2", "processApplicationName": "Hiring Sample", "processApplicationSnapshotName": "v5", "startingProcessInstanceFullId":"cd974bd9-0ba2-499d-abdc-319c3be66993.2064.ca663330- 14b8-40c2-8a57-2c63a173c677.44000000435", "processFullId": "cd974bd9-0ba2-499d-abdc-319c3be66993.2064.ca663330-14b8-40c2-8a57- 2c63a173c677.44000000435", "processName":"Standard HR Open New Position", "processInstanceId":44000000435, "processState":"Terminated", "processVersion":"2064.ca663330-14b8-40c2-8a57-2c63a173c677", "processPOId":"cd974bd9-0ba2-499d-abdc-319c3be66993", "processSnapshotName": "v5" } Info calculated from several Process Event document Process Event Process Summary
  23. 23. 23 9/29/17 { "cellName":"BPM1", "eventVersion": "2010-11-11", "eventKind": "bpmnx:ACTIVITY_ACTIVE", "occurrenceTime":2017-01-25T17:32:32.084+08:00, "performerId":"deadmin", "performerName":"deadmin", "activityType":"userTask", "activityName":"HT1", "activityMonId":"bpdid:54fb3da14f1aa466:3550961c:159b077f5d7:-7f6b", "activityMonVersion":"2064.402aa322-1a07-4886-a848-c4af1f2a8c3e", "processApplicationId": "532cc340-7d17-4ce7-a1c1-6e8380fec134", "processApplicationVersion": "2064.2142be53-2334-4338-a194-4e3861992ae2", "processApplicationName": "PA01", "processApplicationSnapshotName": "v5", "activityFullId": "a0884aa1-5112-4142-a79a-e87657b142fb.2064.402aa322-1a07-4886-a848- c4af1f2a8c3e.234-3-6.16", "startingProcessInstanceFullId":"cf3e2fcc-5aba-43fb-abc5-76347b9fd7b0.2064.c7e467ab-e7b3-4307- 88e6-44fbd1b74fbf.234", } { "cellName":"BPM1", "eventVersion": "2010-11-11", "eventKind": "bpmnx:ACTIVITY_RESOURCE_ASSIGNED", "occurrenceTime": 2017-01-25T14:54:20.490+08:00 "taskInstanceId":"280", "potentialPerformerId":"All Users_T_da7e4d23-78cb-4483-98ed-b9c238308a03.5e6546c5- 5ae4-429c-bc57-cdbc3bb015f5", "potentialPerformerName":"All Users", "potentialPerformerDocumentation":"All Users", "activityType":"userTask", "activityName":"HT1", "activityMonId":"bpdid:54fb3da14f1aa466:3550961c:159b077f5d7:-7f6b", "activityMonVersion":"2064.402aa322-1a07-4886-a848-c4af1f2a8c3e", "processApplicationId": "532cc340-7d17-4ce7-a1c1-6e8380fec134", "processApplicationVersion": "2064.2142be53-2334-4338-a194-4e3861992ae2", "processApplicationName": "PA01", "processApplicationSnapshotName": "v5", "activityFullId": "a0884aa1-5112-4142-a79a-e87657b142fb.2064.402aa322-1a07-4886-a848- c4af1f2a8c3e.234-3-6.16", "startingProcessInstanceFullId":"cf3e2fcc-5aba-43fb-abc5-76347b9fd7b0.2064.c7e467ab- e7b3-4307-88e6-44fbd1b74fbf.234", } { "cellName":"BPM1", "taskInstanceId":"280", "potentialPerformerId":"All Users_T_da7e4d23-78cb-4483-98ed-b9c238308a03.5e6546c5-5ae4-429c-bc57-cdbc3bb015f5", "potentialPerformerName":"All Users", "potentialPerformerDocumentation":"All Users", "performerId":"deadmin", "performerName":"deadmin", "taskStartTime":2017-01-25T14:54:20.490+08:00, "taskClaimedTime":2017-01-25T17:32:32.084+08:00, "taskCompletedTime": 2017-01-25T17:34:32.084+08:00, "taskState":"Completed", "activityType":"userTask", "activityName":"HT1", "activityMonId":"bpdid:54fb3da14f1aa466:3550961c:159b077f5d7:-7f6b", "activityMonVersion":"2064.402aa322-1a07-4886-a848-c4af1f2a8c3e", "processApplicationId": "532cc340-7d17-4ce7-a1c1-6e8380fec134", "processApplicationVersion": "2064.2142be53-2334-4338-a194-4e3861992ae2", "processApplicationName": "PA01", "processApplicationSnapshotName": "v5", "activityFullId": "a0884aa1-5112-4142-a79a-e87657b142fb.2064.402aa322-1a07-4886-a848-c4af1f2a8c3e.234-3-6.16", "startingProcessInstanceFullId":"cf3e2fcc-5aba-43fb-abc5-76347b9fd7b0.2064.c7e467ab-e7b3-4307-88e6-44fbd1b74fbf.234", } Activity Event Activity Summary Info calculated from several Activity Event documents Elasticsearch Indexes
  24. 24. BusinessData • Contains business data and KPI 24 9/29/17 Elasticsearch Indexes { "cellName":"BPM1", "activityFullId": "a0884aa1-5112-4142-a79a-e87657b142fb.2064.402aa322-1a07-4886-a848-c4af1f2a8c3e.234.16", "startingProcessInstanceFullId":"cf3e2fcc-5aba-43fb-abc5-76347b9fd7b0.2064.c7e467ab-e7b3-4307-88e6-44fbd1b74fbf.234", "processApplicationName": "Hiring Sample", "processApplicationSnapshotName": "v5", "processName":"Standard HR Open New Position", "processInstanceId":234, "taskId":280 "activityName":"Submit position request ", "wleGroupId": "19b73424-1bdc-4ee4-aa5a-f4ed8784d4a8", "wleName": "Submit position request (PRE)", "wleGroupVersion": "2064.be7c5d58-e3b6-4550-a253-09be00e5434e", "wleGroupName": "aEmpRequisition121381434563922", "wleTime": 2017-01-17T15:31:06.852+08:00, "wleVersion": "2064.be7c5d58-e3b6-4550-a253-09be00e5434e", "wleId": "-f4ed8784d4a8bpdid431b0753c33842e23d5457c0141a2fd3448-75fb (PRE)", "kpiData": { "Rework.integer":0 } "trackedField": { "GMApproval.boolean":false, "NumberRequested.integer": "2", "Location.keyword": "Chicago", "EmploymentStatus.keyword": "Full-time", "Department.keyword": "Finance", "HiringManager.keyword": "Jack Smith" } } Business Data sample
  25. 25. Out of box Dashboard – Process Performance 2 Identify total process count Process statistics based on BPM system and state Average process running duration Process instance started per 3 hours – help to find busy time Process instance completed per 3 hours – help to find busy time Process count for each process
  26. 26. Out of box Dashboard – Task Performance 2 Drill down for specific task: review new position request Identify total task count Task count by BPM system and state Task count for different tasks Task started and completed per 3 hours – help to find busy time 26,124 out of 128,926 tasks are “review new position request” Average task running duration increases from 35.07m to 63.86m Average Task Running Duration
  27. 27. Out of box Dashboard – Team Performance 2 Task assigned to each team and team members Average task duration for each team member Task completed per 3 hours by each team
  28. 28. Sample Custom Dashboard – KPI 2 Average Task Time, Average Execute Time and Average Wait Time for each task Average task time change trend per 3 hours Average task time has a sudden increase here. It may indicate abnormal.
  29. 29. Sample Dashboard – Business at a Glance 2 • Business Data dashboard is different for each process and can only be customized by yourself. • Here we show a sample business dashboard for Hiring Sample How many people are hired at each location and what type How many people are hired in each department and their job type Tile map shows hiring people at each location. Deeper color indicates more people are hired there Approval rate for each manager’s requests
  30. 30. Create “Business at a Glance” Dashboard for Hiring Sample • On discover tab, search ”Location:*” • Save the save search as “HiringBusinessData” • On visualization tab, select “Pie Chart” 30 9/29/17 Create Your Dashboard
  31. 31. Create “Business at a Glance” Dashboard for Hiring Sample • Select “HiringBusinessData” save search • Choose “Split Slices” • Select “Terms” as Aggregation • Select “Location” as Field • Save pie as “Hiring Request by Location” 31 9/29/17 Create Your Dashboard
  32. 32. Create “Business at a Glance” Dashboard for Hiring Sample • On dashboard tab, add a new dashboard • Select “Hiring Requests by Location” • Save the dashboard “Business at a Glance” • Create other visualization and add to the dashboard 32 9/29/17 Create Your Dashboard
  33. 33. Install BPM Analytics Technology Preview • Install BPM 8.6 • Download add-ons from: https://github.com/ibmbpm/bpm_analytics • Contents: • EventSummaryAgent.tar • Monitors the new events added in the configured elasticsearch index and then generate summary events • BPMDashboardKibana.zip • Contains five Kibana dashboards • IBM Business Process Manager Analytics.pdf • Install instructions CF 2017.09 (8.6)
  34. 34. IBM Cloud University 2017 | October Please Note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
  35. 35. IBM Cloud University 2017 | October Notices and disclaimers Copyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.” Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
  36. 36. IBM Cloud University 2017 | October Notices and disclaimers continued Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services®, Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X- Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

×