Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BenchFlow, a Framework for Benchmarking BPMN 2.0 Workflow Management Systems

695 views

Published on

The talk I gave at the BPM 2015 Conference about BenchFlow, a Framework for Benchmarking BPMN 2.0 Workflow Management Systems

Published in: Technology
  • Be the first to comment

BenchFlow, a Framework for Benchmarking BPMN 2.0 Workflow Management Systems

  1. 1. Architecture, Design and Web Information Systems Engineering Group Vincenzo  Ferme,    Ana  Ivanchikj,  Cesare  Pautasso   Faculty  of  Informatics   University  of  Lugano  (USI)   Switzerland A FRAMEWORK FOR BENCHMARKING BPMN 2.0 WORKFLOW MANAGEMENT SYSTEMS BENCHFLOW
  2. 2. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 2 BPMN 2.0: A Widely Adopted Standard https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines 2013 8 19 2014 2 21 2015 0 21 Grand Total 21 NumberofBPMN2.0WfMSs 0 5 10 15 20 25 Year of the First Version Supporting BPMN 2.0 2009 2010 2011 2012 2013 2014 2015 Jan 2011 BPMN 2.0 Jan 2014 BPMN 2.0.2 ISO/IEC 19510 Aug 2009 BETA BPMN 2.0 Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  3. 3. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme Application Server 3 WES Workflow Management System’s Main Components Job Executor Core Engine … Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  4. 4. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme Application Server 3 WES Workflow Management System’s Main Components Job Executor Core Engine Process Navigator D A B C … Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  5. 5. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme Application Server 3 WES Workflow Management System’s Main Components Job Executor Core Engine Process Navigator D A B C … Task Dispatcher Users Application Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  6. 6. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme Application Server 3 WES Workflow Management System’s Main Components Job Executor Core Engine Process Navigator D A B C … Task Dispatcher Users Application Event Handler Service Invoker Web Service Application Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  7. 7. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme Application Server 3 WES Workflow Management System’s Main Components Job Executor Core Engine Process Navigator D A B C … Instance Database DBMS Persistent Manager Transaction Manager Task Dispatcher Users Application Event Handler Service Invoker Web Service Application Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  8. 8. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 4 Workflow Management System’s Diversification Supported Languages • BPMN, BPEL, Petri-Nets, YAML System’s Architecture • Distributed workflow support • Migrating workflow objects support • Transactional workflow support Functionalities • Dynamic workflow changes • Integration capabilities Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » … Deployment Infrastructure • Standalone • Cluster Deployment • Cloud Deployment • Mobile Deployment
  9. 9. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 5 Workflow Management System’s Diversification Supported Languages • BPMN, BPEL, Petri-Nets, YAML System’s Architecture • Distributed workflow support • Migrating workflow objects support • Transactional workflow support Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » … Functionalities • Dynamic workflow changes • Integration capabilities Deployment Infrastructure • Standalone • Cluster Deployment • Cloud Deployment • Mobile Deployment
  10. 10. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 6 Workflow Management System’s Diversification Supported Languages • BPMN, BPEL, Petri-Nets, YAML • Distributed workflow support • Migrating workflow objects support • Transactional workflow support System’s Architecture Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » … Functionalities • Dynamic workflow changes • Integration capabilities Deployment Infrastructure • Standalone • Cluster Deployment • Cloud Deployment • Mobile Deployment
  11. 11. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 7 Workflow Management System’s Diversification Deployment Infrastructure • Standalone • Cluster Deployment • Cloud Deployment • Mobile Deployment Supported Languages • BPMN, BPEL, Petri-Nets, YAML System’s Architecture • Distributed workflow support • Migrating workflow objects support • Transactional workflow support Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » … Functionalities • Dynamic workflow changes • Integration capabilities
  12. 12. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 8 The BenchFlow Project Design the first benchmark to assess and compare the performance of WfMSs that are compliant with Business Process Model and Notation 2.0 standard. ” “ Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  13. 13. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 9 BenchFlow Framework: Requirements & Functionalities • Automate the SUT deployment • Simplify the SUT’s deployment configuration • Adapt to different API provided by different WfMSs • Deal with the asynchronous execution of business processes System Under Test (SUT) Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  14. 14. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 10 BenchFlow Framework: Requirements & Functionalities • Simulate all the entities interacting with the WfMS • Accomodate and automate different kinds of performance test: • Ensure reliable execution • Ensure repeatability • Automate the performance data collection and analyses Performance Benchmark SOABench, SOArMetrics, Betsy, LoadUI + SoapUI Similar Tools: Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  15. 15. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 11 BenchFlow Framework: Requirements & Functionalities SOABench, SOArMetrics, Betsy, LoadUI + SoapUI Similar Tools: • Simulate all the entities interacting with the WfMS • Accomodate and automate different kinds of performance test: • Ensure reliable execution • Ensure repeatability • Automate the performance data collection and analyses Performance Benchmark Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  16. 16. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme Instance Database 12 BenchFlow Framework DBMSFaban Drivers ContainersServers DATA
 CLEANERS ANALYSERS Performance Metrics Performance KPIs harness WES TestExecutionAnalyses Faban + Web Service Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  17. 17. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 13 BenchFlow Framework Faban Drivers ContainersServers TestExecution Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  18. 18. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 13 BenchFlow Framework Faban Drivers ContainersServers TestExecution Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  19. 19. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 14 BenchFlow Framework DBMSFaban Drivers ContainersServers harness WES TestExecution Web Service Adapters Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  20. 20. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 15 BenchFlow Framework DBMSFaban Drivers ContainersServers harness WES TestExecution Web Service MONITOR Adapters Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  21. 21. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 15 BenchFlow Framework DBMSFaban Drivers ContainersServers harness WES TestExecution Instance Database Analyses Web Service MONITOR COLLECTORS Adapters Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  22. 22. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 16 BenchFlow Framework DBMSFaban Drivers Instance Database ContainersServers DATA
 CLEANERS ANALYSERS Performance Metrics Performance KPIs harness WES TestExecutionAnalyses Web Service Data Mappers MONITOR COLLECTORS Adapters Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  23. 23. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 17 Performance Metrics and KPIs Empty Script Task Wait 2 seconds TEST PROCESS Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  24. 24. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 17 Performance Metrics and KPIs LOAD FUNCTION Users 0 6 12 18 24 30 0 20 60 120 180 240 300 Empty Script Task Wait 2 seconds TEST PROCESS Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  25. 25. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 17 Performance Metrics and KPIs LOAD FUNCTION Users 0 6 12 18 24 30 0 20 60 120 180 240 300 Empty Script Task Wait 2 seconds TEST PROCESS TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  26. 26. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 17 Performance Metrics and KPIs LOAD FUNCTION Users 0 6 12 18 24 30 0 20 60 120 180 240 300 Empty Script Task Wait 2 seconds TEST PROCESS TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WES Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  27. 27. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 17 Performance Metrics and KPIs LOAD FUNCTION Users 0 6 12 18 24 30 0 20 60 120 180 240 300 Empty Script Task Wait 2 seconds TEST PROCESS TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WES CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  28. 28. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 17 Performance Metrics and KPIs LOAD FUNCTION Users 0 6 12 18 24 30 0 20 60 120 180 240 300 Empty Script Task Wait 2 seconds TEST PROCESS TEST ENVIRONMENT CPU 64 Cores 
 @ 1400 MHz RAM 128 GB Load Drivers CPU 12 Cores 
 @ 800 MHz RAM 64 GB WES CPU 64 Cores 
 @ 2300 MHz RAM 128 GB DBMS 10 Gbit/s Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  29. 29. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 18 Throughput MHz, 128GB of RAM) can issue su cient load and that the res at 2300MHz, 128GB of RAM) can handle the requests r each test we verify the absence of measurement noise by ment metrics (CPU, RAM and network usage) and the WfE ll the BP instances are completed. riment on two open-source WfMSs supporting native exe- We test them on top of Apache Tomcat 7.0.59 using Oracle Community Server 5.5.42. We use the default configuration ors’ websites. analyze is the Throughput = #BP Instances(bp) T ime(s) [9, ch. 11] e B does not scale well after 25 and the throughput starts ers. Engine A can handle a load up to 125, with the through- 0 25 50 75 100 125 150 0 50 100 Concurrent Users Throughput(bp/s) Engine A Engine B Fig. 3: Throughput Duration(s) 5 25 50 75 100 110 125 2 4 6 8 (a)EngineA 135 150 0 2,000 4,000 6,000 8,000 Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  30. 30. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 19 Instance Duration Time Empty Script Task Wait 2 seconds Instance Duration Time Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  31. 31. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 20 0 25 50 75 100 125 150 0 50 Concurrent UsersThroug Fig. 3: Throughput InstanceDuration(s) 5 25 50 75 100 110 125 2 4 6 8 (a)EngineA 135 150 0 2,000 4,000 6,000 8,000 5 25 50 0 10 20 30 40 (b)EngineB 75 100 125 150 0 200 400 600 Concurrent Users Fig. 4: Aggregated Process Instance Duration Comparison The BP instance duration is the time di↵erence between the start and the completion of a BP instance. It is presented in the box and whisker plot in Fig. 4(a) for Engine A and Fig. 4(b) for Engine B. This type of plot displaysContext » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » … Instance Duration Time
  32. 32. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 21 Instance Duration Time and CPU Utilisation Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » … Fig. 3: Throughput InstanceDuration(s) 5 25 50 75 100 110 125 2 4 6 8 (a)EngineA 135 150 0 2,000 4,000 6,000 8,000 600Fig. 4: Aggregated Process Instance Duration Comparison 5 25 50 75 100 110 125 135 150 0 10 20 Concurrent Users CPU(%) Fig. 5: Aggregated CPU Usage (Engine A) (a) Engine A 1 (s) (b) Engine B 1 (s)
  33. 33. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 22 5 25 50 75 100 110 125 135 150 0 10 20 Concurrent Users CPU(%) Fig. 5: Aggregated CPU Usage (Engine A) (a) Engine A 1 (s) (b) Engine B 1 (s) Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » … Fig. 3: Throughput InstanceDuration(s) 5 25 50 75 100 110 125 2 4 6 8 (a)EngineA 135 150 0 2,000 4,000 6,000 8,000 600Fig. 4: Aggregated Process Instance Duration Comparison Instance Duration Time and CPU Utilisation
  34. 34. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 23 Future Work • Perform the first real-world experiments • Increase the number of supported WfMSs • Simplify and automate the execution of common performance tests: Load Test, Spike Test, Scalability Test, … Experiments BenchFlow Framework • Release a development version on GitHub benchflow Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
  35. 35. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 24 Highlights … » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights
  36. 36. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 24 Highlights … » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights Workflow Management System
  37. 37. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 24 Highlights … » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights Workflow Management System BenchFlow Project
  38. 38. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 24 Highlights … » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights Workflow Management System BenchFlow Project BenchFlow Framework
  39. 39. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 24 Highlights … » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights Workflow Management System BenchFlow Project BenchFlow Framework Proof of Concept
  40. 40. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme 25 Call for Action •We want to characterise the Workload using Real-World process models • Send us your executable BPMN process models, even anonymised! Process Models vincenzo.ferme@usi.ch •We want to characterise the Workload using Real-World behaviours • Send us your execution logs, even anonymised! Execution Logs
  41. 41. Architecture, Design and Web Information Systems Engineering Group benchflow benchflow Vincenzo  Ferme  (@VincenzoFerme),    Ana  Ivanchikj,  Cesare  Pautasso   Faculty  of  Informatics   University  of  Lugano  (USI)   Switzerland A FRAMEWORK FOR BENCHMARKING BPMN 2.0 WORKFLOW MANAGEMENT SYSTEMS BENCHFLOW vincenzo.ferme@usi.ch http://benchflow.inf.usi.ch
  42. 42. Architecture, Design and Web Information Systems Engineering Group Vincenzo Ferme Join Us @ ICWE 2016 in Lugano! http://icwe2016.inf.usi.ch

×