The talk I gave at the BPM 2015 Conference about BenchFlow, a Framework for Benchmarking BPMN 2.0 Workflow Management Systems

1. Architecture, Design and Web Information Systems Engineering Group
Vincenzo
Ferme,
Ana
Ivanchikj,
Cesare
Pautasso
Faculty
of
Informatics
University
of
Lugano
(USI)
Switzerland
A FRAMEWORK FOR BENCHMARKING
BPMN 2.0
WORKFLOW MANAGEMENT SYSTEMS
BENCHFLOW

2. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
2
BPMN 2.0: A Widely Adopted Standard
https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines
2013 8 19
2014 2 21
2015 0 21
Grand
Total 21
NumberofBPMN2.0WfMSs
0
5
10
15
20
25
Year of the First Version Supporting BPMN 2.0
2009 2010 2011 2012 2013 2014 2015
Jan 2011
BPMN 2.0
Jan 2014
BPMN 2.0.2
ISO/IEC 19510
Aug 2009
BETA
BPMN 2.0
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

3. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
Application Server 3
WES
Workflow Management System’s Main Components
Job
Executor
Core
Engine
…
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

4. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
Application Server 3
WES
Workflow Management System’s Main Components
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
…
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

5. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
Application Server 3
WES
Workflow Management System’s Main Components
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
…
Task Dispatcher Users
Application
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

6. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
Application Server 3
WES
Workflow Management System’s Main Components
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
…
Task Dispatcher Users
Application
Event Handler
Service Invoker
Web
Service
Application
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

7. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
Application Server 3
WES
Workflow Management System’s Main Components
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
…
Instance
Database
DBMS
Persistent Manager
Transaction Manager
Task Dispatcher Users
Application
Event Handler
Service Invoker
Web
Service
Application
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

8. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
4
Workflow Management System’s Diversification
Supported Languages
• BPMN, BPEL, Petri-Nets, YAML
System’s Architecture
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
Functionalities
• Dynamic workflow changes
• Integration capabilities
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
Deployment Infrastructure
• Standalone
• Cluster Deployment
• Cloud Deployment
• Mobile Deployment

9. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
5
Workflow Management System’s Diversification
Supported Languages
• BPMN, BPEL, Petri-Nets, YAML
System’s Architecture
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
Functionalities
• Dynamic workflow changes
• Integration capabilities
Deployment Infrastructure
• Standalone
• Cluster Deployment
• Cloud Deployment
• Mobile Deployment

10. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
6
Workflow Management System’s Diversification
Supported Languages
• BPMN, BPEL, Petri-Nets, YAML
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
System’s Architecture
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
Functionalities
• Dynamic workflow changes
• Integration capabilities
Deployment Infrastructure
• Standalone
• Cluster Deployment
• Cloud Deployment
• Mobile Deployment

11. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
7
Workflow Management System’s Diversification
Deployment Infrastructure
• Standalone
• Cluster Deployment
• Cloud Deployment
• Mobile Deployment
Supported Languages
• BPMN, BPEL, Petri-Nets, YAML
System’s Architecture
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
Functionalities
• Dynamic workflow changes
• Integration capabilities

12. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
8
The BenchFlow Project
Design the first benchmark to assess and
compare the performance of WfMSs that are
compliant with Business Process Model and
Notation 2.0 standard.
”
“
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

13. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
9
BenchFlow Framework: Requirements & Functionalities
• Automate the SUT deployment
• Simplify the SUT’s deployment configuration
• Adapt to different API provided by different WfMSs
• Deal with the asynchronous execution of business processes
System Under Test (SUT)
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

14. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
10
BenchFlow Framework: Requirements & Functionalities
• Simulate all the entities interacting with the WfMS
• Accomodate and automate different kinds of performance test:
• Ensure reliable execution
• Ensure repeatability
• Automate the performance data collection and analyses
Performance Benchmark
SOABench, SOArMetrics, Betsy, LoadUI + SoapUI
Similar Tools:
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

15. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
11
BenchFlow Framework: Requirements & Functionalities
SOABench, SOArMetrics, Betsy, LoadUI + SoapUI
Similar Tools:
• Simulate all the entities interacting with the WfMS
• Accomodate and automate different kinds of performance test:
• Ensure reliable execution
• Ensure repeatability
• Automate the performance data collection and analyses
Performance Benchmark
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

16. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
Instance
Database
12
BenchFlow Framework
DBMSFaban Drivers
ContainersServers
DATA
CLEANERS
ANALYSERS
Performance
Metrics
Performance
KPIs
harness
WES
TestExecutionAnalyses
Faban
+
Web
Service
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

17. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
13
BenchFlow Framework
Faban Drivers
ContainersServers
TestExecution
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

18. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
13
BenchFlow Framework
Faban Drivers
ContainersServers
TestExecution
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

19. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
14
BenchFlow Framework
DBMSFaban Drivers
ContainersServers
harness
WES
TestExecution
Web
Service
Adapters
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

20. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
15
BenchFlow Framework
DBMSFaban Drivers
ContainersServers
harness
WES
TestExecution
Web
Service
MONITOR
Adapters
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

21. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
15
BenchFlow Framework
DBMSFaban Drivers
ContainersServers
harness
WES
TestExecution
Instance
Database
Analyses
Web
Service
MONITOR
COLLECTORS
Adapters
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

22. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
16
BenchFlow Framework
DBMSFaban Drivers
Instance
Database
ContainersServers
DATA
CLEANERS
ANALYSERS
Performance
Metrics
Performance
KPIs
harness
WES
TestExecutionAnalyses
Web
Service
Data Mappers
MONITOR
COLLECTORS
Adapters
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

23. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
17
Performance Metrics and KPIs
Empty Script
Task
Wait 2 seconds
TEST PROCESS
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

24. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
17
Performance Metrics and KPIs
LOAD FUNCTION
Users
0
6
12
18
24
30
0
20
60
120
180
240
300
Empty Script
Task
Wait 2 seconds
TEST PROCESS
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

25. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
17
Performance Metrics and KPIs
LOAD FUNCTION
Users
0
6
12
18
24
30
0
20
60
120
180
240
300
Empty Script
Task
Wait 2 seconds
TEST PROCESS
TEST ENVIRONMENT
CPU
64 Cores
@ 1400 MHz
RAM 128 GB
Load Drivers
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

26. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
17
Performance Metrics and KPIs
LOAD FUNCTION
Users
0
6
12
18
24
30
0
20
60
120
180
240
300
Empty Script
Task
Wait 2 seconds
TEST PROCESS
TEST ENVIRONMENT
CPU
64 Cores
@ 1400 MHz
RAM 128 GB
Load Drivers
CPU
12 Cores
@ 800 MHz
RAM 64 GB
WES
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

27. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
17
Performance Metrics and KPIs
LOAD FUNCTION
Users
0
6
12
18
24
30
0
20
60
120
180
240
300
Empty Script
Task
Wait 2 seconds
TEST PROCESS
TEST ENVIRONMENT
CPU
64 Cores
@ 1400 MHz
RAM 128 GB
Load Drivers
CPU
12 Cores
@ 800 MHz
RAM 64 GB
WES
CPU
64 Cores
@ 2300 MHz
RAM 128 GB
DBMS
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

28. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
17
Performance Metrics and KPIs
LOAD FUNCTION
Users
0
6
12
18
24
30
0
20
60
120
180
240
300
Empty Script
Task
Wait 2 seconds
TEST PROCESS
TEST ENVIRONMENT
CPU
64 Cores
@ 1400 MHz
RAM 128 GB
Load Drivers
CPU
12 Cores
@ 800 MHz
RAM 64 GB
WES
CPU
64 Cores
@ 2300 MHz
RAM 128 GB
DBMS
10 Gbit/s
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

29. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
18
Throughput
MHz, 128GB of RAM) can issue su cient load and that the
res at 2300MHz, 128GB of RAM) can handle the requests
r each test we verify the absence of measurement noise by
ment metrics (CPU, RAM and network usage) and the WfE
ll the BP instances are completed.
riment on two open-source WfMSs supporting native exe-
We test them on top of Apache Tomcat 7.0.59 using Oracle
Community Server 5.5.42. We use the default configuration
ors’ websites.
analyze is the Throughput = #BP Instances(bp)
T ime(s) [9, ch. 11]
e B does not scale well after 25 and the throughput starts
ers. Engine A can handle a load up to 125, with the through-
0 25 50 75 100 125 150
0
50
100
Concurrent Users
Throughput(bp/s)
Engine A
Engine B
Fig. 3: Throughput
Duration(s)
5 25 50 75 100 110 125
2
4
6
8
(a)EngineA
135 150
0
2,000
4,000
6,000
8,000
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

30. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
19
Instance Duration Time
Empty Script
Task
Wait 2 seconds
Instance Duration Time
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

31. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
20
0 25 50 75 100 125 150
0
50
Concurrent UsersThroug
Fig. 3: Throughput
InstanceDuration(s)
5 25 50 75 100 110 125
2
4
6
8
(a)EngineA
135 150
0
2,000
4,000
6,000
8,000
5 25 50
0
10
20
30
40
(b)EngineB
75 100 125 150
0
200
400
600
Concurrent Users
Fig. 4: Aggregated Process Instance Duration Comparison
The BP instance duration is the time di↵erence between the start and the
completion of a BP instance. It is presented in the box and whisker plot in
Fig. 4(a) for Engine A and Fig. 4(b) for Engine B. This type of plot displaysContext » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
Instance Duration Time

32. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
21
Instance Duration Time and CPU Utilisation
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
Fig. 3: Throughput
InstanceDuration(s)
5 25 50 75 100 110 125
2
4
6
8
(a)EngineA
135 150
0
2,000
4,000
6,000
8,000
600Fig. 4: Aggregated Process Instance Duration Comparison
5 25 50 75 100 110 125 135 150
0
10
20
Concurrent Users
CPU(%)
Fig. 5: Aggregated CPU Usage (Engine A)
(a) Engine A
1
(s)
(b) Engine B
1
(s)

33. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
22
5 25 50 75 100 110 125 135 150
0
10
20
Concurrent Users
CPU(%)
Fig. 5: Aggregated CPU Usage (Engine A)
(a) Engine A
1
(s)
(b) Engine B
1
(s)
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …
Fig. 3: Throughput
InstanceDuration(s)
5 25 50 75 100 110 125
2
4
6
8
(a)EngineA
135 150
0
2,000
4,000
6,000
8,000
600Fig. 4: Aggregated Process Instance Duration Comparison
Instance Duration Time and CPU Utilisation

34. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
23
Future Work
• Perform the first real-world experiments
• Increase the number of supported WfMSs
• Simplify and automate the execution of common performance tests:
Load Test, Spike Test, Scalability Test, …
Experiments
BenchFlow Framework
• Release a development version on GitHub
benchflow
Context » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » …

35. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
24
Highlights
… » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights

36. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
24
Highlights
… » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights
Workflow Management System

37. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
24
Highlights
… » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights
Workflow Management System BenchFlow Project

38. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
24
Highlights
… » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights
Workflow Management System BenchFlow Project
BenchFlow Framework

39. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
24
Highlights
… » WfMS Components » WfMSs Diversification » BenchFlow » Requirements » BenchFlow Framework » Experiments » Future Work » Highlights
Workflow Management System BenchFlow Project
BenchFlow Framework Proof of Concept

40. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
25
Call for Action
•We want to characterise the Workload using Real-World process models
• Send us your executable BPMN process models, even anonymised!
Process Models
vincenzo.ferme@usi.ch
•We want to characterise the Workload using Real-World behaviours
• Send us your execution logs, even anonymised!
Execution Logs

41. Architecture, Design and Web Information Systems Engineering Group
benchflow
benchflow
Vincenzo
Ferme
(@VincenzoFerme),
Ana
Ivanchikj,
Cesare
Pautasso
Faculty
of
Informatics
University
of
Lugano
(USI)
Switzerland
A FRAMEWORK FOR BENCHMARKING
BPMN 2.0
WORKFLOW MANAGEMENT SYSTEMS
BENCHFLOW
vincenzo.ferme@usi.ch
http://benchflow.inf.usi.ch

42. Architecture, Design and Web Information Systems Engineering Group
Vincenzo Ferme
Join Us @ ICWE 2016 in Lugano!
http://icwe2016.inf.usi.ch