Trusted benchmarks should provide reproducible results obtained following a transparent and well-defined process. In this paper, we show how Containers, originally developed to ease the automated deployment of Cloud application components, can be used in the context of a benchmarking methodology. The proposed methodology focuses on Workflow Management Systems (WfMSs), a critical service orchestration middleware, which can be characterized by its architectural complexity, for which Docker Containers offer a highly suitable approach. The contributions of our work are: 1) a new benchmarking approach taking full advantage of containerization technologies; and 2) the formalization of the interaction process with the WfMS vendors described clearly in a written agreement. Thus, we take advantage of emerging Cloud technologies to address technical challenges, ensuring the performance measurements can be trusted. We also make the benchmarking process transparent, automated, and repeatable so that WfMS vendors can join the benchmarking effort.

1. Vincenzo Ferme, Ana Ivanchikj,
Prof. Cesare Pautasso
Faculty of Informatics
University of Lugano (USI), Switzerland
A CONTAINER-CENTRIC
METHODOLOGY FOR
BENCHMARKING WORKFLOW
MANAGEMENT SYSTEMS
Marigianna Skouradaki,
Prof. Frank Leymann
Institute of Architecture of Application Systems
University of Stuttgart, Germany
Cloud Computing Patt
Fundamentals to Design, Build, and Man
Christoph Fehling
Institute of Architecture of Applicatio
University of Stuttgart
Universitätsstr. 38
70569 Stuttgart

2. Vincenzo Ferme
2
The BenchFlow Project
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Design and implement the first benchmark to
assess and compare the performance of WfMSs
that are compliant with Business Process Model
and Notation 2.0 standard.
”
“

3. Vincenzo Ferme
3
What is a Workflow Management System?
WfMS
Application Server
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

4. Vincenzo Ferme
3
What is a Workflow Management System?
WfMS
Application Server
D
A
B
C
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

5. Vincenzo Ferme
3
What is a Workflow Management System?
WfMS Users
Applications
Application Server
Web
Service
D
A
B
C
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

6. Vincenzo Ferme
3
What is a Workflow Management System?
WfMS Users
Applications
Application Server
Instance
Database
DBMS
Web
Service
D
A
B
C
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

7. Vincenzo Ferme
4
Many Vendors of BPMN 2.0 WfMSs
https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines
Jan 2011
BPMN 2.0
Jan 2014
BPMN 2.0.2
ISO/IEC 19510
Aug 2009
BETA
BPMN 2.0
2016 0 23
Grand
Total 23
NumberofBPMN2.0WfMSs
0
5
10
15
20
25
Year of the First Version Supporting BPMN 2.0
2009 2010 2011 2012 2013 2014 2015 2016
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

8. Vincenzo Ferme
5
Benchmarking Requirements
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
• K. Huppler, The art of building a good benchmark, 2009
• J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993
• S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A
challenge to software engineering, 2003
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

9. Vincenzo Ferme
6
Why a new Methodology?
No available methodologies involving vendors
for both defining a standard benchmark and
benchmarking production systems
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

10. Vincenzo Ferme
6
Why a new Methodology?
Already standard benchmarks
No available methodologies involving vendors
for both defining a standard benchmark and
benchmarking production systems
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

11. Vincenzo Ferme
6
Why a new Methodology?
Already standard benchmarks
Research
No interaction with Vendors
No available methodologies involving vendors
for both defining a standard benchmark and
benchmarking production systems
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

12. Vincenzo Ferme
7
Why a Container-Centric Methodology?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Emerging Technology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

13. Vincenzo Ferme
7
Why a Container-Centric Methodology?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Emerging Technology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Lightweight

14. Vincenzo Ferme
7
Why a Container-Centric Methodology?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Emerging Technology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Lightweight
Negligible Performance Impact

15. Vincenzo Ferme
8
Why a Container-Centric Methodology?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Emerging Technology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Lightweight
Negligible Performance Impact

16. Vincenzo Ferme
9
Why a Container-Centric Methodology?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Emerging Technology
Lightweight
Negligible Performance Impact

17. Vincenzo Ferme
10
Why a Container-Centric Methodology?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Emerging Technology
Lightweight
Negligible Performance Impact

18. Vincenzo Ferme
11
Why a Container-Centric Methodology?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Emerging Technology
Lightweight
Negligible Performance Impact

19. Vincenzo Ferme
12
Why a Container-Centric Methodology?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Emerging Technology
Lightweight
Negligible Performance Impact

20. Vincenzo Ferme
13
What about the other Requirements?
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

21. Vincenzo Ferme
14
Benchmarking Choreography
Vendor
BenchFlow
Provide
Benchmarking
Methodology
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
Benchmarking Methodology Agreement Proposal
Signed Agreement
Vendor
BenchFlow
Provide Containerized
Distribution of WfMS
Containerized WfMS Request
Containerized WfMS
Vendor
BenchFlow
Verify the Benchmark
Results
Results Verification Outcome
Results Verification Request
Vendor
BenchFlow
Provide Draft
Benchmark Results
Draft Benchmark ResultsVerified Benchmark Results
Not Valid
Are the
Results
Valid?
Community
BenchFlow
Publish Benchmark
Results
Valid
Figure 4: Benchmarking Methodology ChoreographyContext » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

22. Vincenzo Ferme
Vendor
BenchFlow
Provide
Benchmarking
Methodology
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
Benchmarking Methodology Agreement Proposal
Signed Agreement
Pro
Di
Contain
Co
Results Verification RequestVerified Benchmark Results
Are the
15
Benchmarking Methodology
provide the methodology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

23. Vincenzo Ferme
16
The Benchmarking Process
Workload
Model
WfMS
Configurations
Test
Types
Workload
Metrics
KPIs
Derive
Input/Process/Output Model
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

24. Vincenzo Ferme
16
The Benchmarking Process
Workload
Model
WfMS
Configurations
Test
Types
Workload
Metrics
KPIs
Derive
Input/Process/Output Model
Workload
Mix
80%
C
A
B
20%
D
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

25. Vincenzo Ferme
16
The Benchmarking Process
Workload
Model
WfMS
Configurations
Test
Types
Workload
Metrics
KPIs
Derive
Input/Process/Output Model
Workload
Mix
80%
C
A
B
20%
D
Load Functions
Test Data
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

26. Vincenzo Ferme
16
The Benchmarking Process
Workload
Model
WfMS
Configurations
Test
Types
Workload
Metrics
KPIs
Derive
Input/Process/Output Model
Workload
Mix
80%
C
A
B
20%
D
Load Functions
Test Data
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Containers

27. Vincenzo Ferme
16
The Benchmarking Process
Workload
Model
WfMS
Configurations
Test
Types
Workload
Metrics
KPIs
Derive
Input/Process/Output Model
Workload
Mix
80%
C
A
B
20%
D
Load Functions
Test Data
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Performance
Data
Workflow
Instance
Duration
Throughput
Containers

28. Vincenzo Ferme
17
Benchmarking Methodology
agreement with vendors
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Vendor
BenchFlow
Provide
Benchmarking
Methodology
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
Benchmarking Methodology Agreement Proposal
Signed Agreement
Pro
Di
Contain
Co
Results Verification RequestVerified Benchmark Results
Are the
• Production Stable Release
• Provide defined APIs
• Share Containerised
WfMS
• Authorise Publishing of
Results
Main Agreement Points:

29. Vincenzo Ferme
18
Benchmarking Methodology
containerised WfMSs
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
ogy Agreement Proposal
Signed Agreement
Vendor
BenchFlow
Provide Containerized
Distribution of WfMS
Containerized WfMS Request
Containerized WfMS
Results Verification Request Draft Benchmark Resultsults
Are the

30. Vincenzo Ferme
19
Benchmarking MethodologyWfMS
Configurations
requirements from the WfMS
NON-CORECORE
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

31. Vincenzo Ferme
19
Benchmarking MethodologyWfMS
Configurations
requirements from the WfMS
NON-CORECORE
WfMS
Initialisation APIs
Deploy Process
Start Process
Instance
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

32. Vincenzo Ferme
19
Benchmarking MethodologyWfMS
Configurations
requirements from the WfMS
NON-CORECORE
WfMS
Initialisation APIs
Deploy Process
Start Process
Instance
+ WfMS
Claim Task Complete Task
User APIs
Create User Create Group
Pending User Tasks
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

33. Vincenzo Ferme
19
Benchmarking MethodologyWfMS
Configurations
requirements from the WfMS
NON-CORECORE
WfMS
Initialisation APIs
Deploy Process
Start Process
Instance
+ WfMS
Claim Task Complete Task
User APIs
Create User Create Group
Pending User Tasks
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Invoke WS
Web Service APIs

34. Vincenzo Ferme
19
Benchmarking MethodologyWfMS
Configurations
requirements from the WfMS
NON-CORECORE
WfMS
Initialisation APIs
Deploy Process
Start Process
Instance
Event APIs
Issue Event
WfMS
Pending Event Tasks
+ WfMS
Claim Task Complete Task
User APIs
Create User Create Group
Pending User Tasks
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Invoke WS
Web Service APIs

35. Vincenzo Ferme
20
Benchmarking MethodologyWfMS
Configurations
requirements from the WfMS
Table 1: Summary of Core and Non-core APIs to be imple-
mented by the WfMS
Functionality Min Response Data
CoreAPIs
Initialisation
APIs
Deploy a process Deployed process ID
Start a process instance Process instance ID
Non-coreAPIs
User APIs Create a user User ID
Create a group of users User group ID
Access pending tasks Pending tasks IDs
Claim a task*
Complete a task
Event APIs Access pending events Pending events IDs
Issue events
Web service
APIs
Map tasks to Web
service endpoints
*Optional depending on the WfMS implementation
ing
the
a u
(U
ate
by
acc
a g
and
mo
ing
(Pi
has
col
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

36. Vincenzo Ferme
21
Benchmarking Methodology
DBMS
WfMS
• At least two containers
• DBMS can refer to existing
one publicly available
• Provide a ready to use default
configuration (at least)
• Configurability of: DBMS,
WfMS, Logging Level (at least)
containerised WfMSs
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

37. Vincenzo Ferme
22
Benchmarking Methodology
Vendor
enchFlow
Provide
nchmarking
ethodology
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
king Methodology Agreement Proposal
Signed Agreement
Vendor
BenchFlow
Provide Containerized
Distribution of WfMS
Containerized WfMS Request
Containerized WfMS
Vendor
BenchFlow
Verify the Benchmark
Results
Results Verification Outcome
Results Verification Request
Vendor
BenchFlow
Provide Draft
Benchmark Results
Draft Benchmark Resultsenchmark Results
Not Valid
Are the
Results
Valid?
ommunity
enchFlow
sh Benchmark
Results
Valid
executing the benchmark and providing results
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

38. Vincenzo Ferme
23
BenchFlow Framework
architecture
Instance
Database
DBMSFaban Drivers
ContainersServers
DATA
TRANSFORMERS
ANALYSERS
Performance
Metrics
Performance
KPIs
WfMS
TestExecutionAnalyses
Faban
+
Web
Service
Minio
harness
benchflow
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
[BPM ’15] [ICPE ’16]

39. Vincenzo Ferme
24
Performance Metrics and KPIs
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

40. Vincenzo Ferme
24
Performance Metrics and KPIs
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

41. Vincenzo Ferme
24
Performance Metrics and KPIs
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

42. Vincenzo Ferme
24
Performance Metrics and KPIs
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
• Feature Level
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

43. Vincenzo Ferme
24
Performance Metrics and KPIs
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
• Feature Level
• Interactions
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

44. Vincenzo Ferme
24
Performance Metrics and KPIs
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
• Feature Level
• Interactions
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Environments

45. Vincenzo Ferme
25
Executing the Benchmark
minimal data requirements
Accessibility of the Data
DBMS
WfMS
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

46. Vincenzo Ferme
25
Executing the Benchmark
minimal data requirements
• Workflow & Construct:
• Start Time
• End Time
• [Duration]
Availability of Timing DataAccessibility of the Data
DBMS
WfMS
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

47. Vincenzo Ferme
25
Executing the Benchmark
minimal data requirements
• Workflow & Construct:
• Start Time
• End Time
• [Duration]
Availability of Timing DataAccessibility of the Data
DBMS
WfMS
Availability of Execution State
State of the workflow execution. E.g., running, completed, error
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

48. Vincenzo Ferme
26
Benchmarking Methodology
mock example of benchmark results
Workload
Model
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

49. Vincenzo Ferme
26
Benchmarking Methodology
mock example of benchmark results
Workload
Model
Ap.Tomcat 7.0.62
WfMS A
MySQL
MySQL: Community Server 5.6.26
O.S.: Ubuntu 14.04.01
J.V.M.: Oracle Serv. 7u79
App. Server:
WfMS A: v7.0.1
+
Hardware Configuration
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

50. Vincenzo Ferme
26
Benchmarking Methodology
mock example of benchmark results
Workload
Model
Ap.Tomcat 7.0.62
WfMS A
MySQL
MySQL: Community Server 5.6.26
O.S.: Ubuntu 14.04.01
J.V.M.: Oracle Serv. 7u79
App. Server:
WfMS A: v7.0.1
+
Hardware Configuration
+
Metrics and KPIs
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work

51. Vincenzo Ferme
27
Benchmarking Methodology
Vendor
BenchFlow
Provide
enchmarking
Methodology
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
arking Methodology Agreement Proposal
Signed Agreement
Vendor
BenchFlow
Provide Containerized
Distribution of WfMS
Containerized WfMS Request
Containerized WfMS
Vendor
BenchFlow
Verify the Benchmark
Results
Results Verification Outcome
Results Verification Request
Vendor
BenchFlow
Provide Draft
Benchmark Results
Draft Benchmark ResultsBenchmark Results
Not Valid
Are the
Results
Valid?
Community
BenchFlow
ish Benchmark
Results
Valid
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
executing the benchmark and providing results

52. Vincenzo Ferme
28
Benchmarking Methodology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Signed Agreement
Ven
Bench
Verify the B
Res
Results Verific
Results VerificVerified Benchmark Results
Are the
Results
Valid?
Community
BenchFlow
Publish Benchmark
Results
Valid
publish benchmark results

53. Vincenzo Ferme
29
Advantages of using Containers
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Accomplish some Benchmarking Requirement:
Portability, Repeatability,Accessibility, Efficiency

54. Vincenzo Ferme
29
Advantages of using Containers
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Accomplish some Benchmarking Requirement:
Portability, Repeatability,Accessibility, Efficiency
• Common way to deploy systems provided by
different vendors
Docker Compose Docker Swarm

55. Vincenzo Ferme
29
Advantages of using Containers
• Standard APIs to access Environment Metrics
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Accomplish some Benchmarking Requirement:
Portability, Repeatability,Accessibility, Efficiency
• Common way to deploy systems provided by
different vendors
Docker Compose Docker Swarm

56. Vincenzo Ferme
30
First Application of the Methodology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
[CAiSE ’16]
M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-
Benchmarking BPMN 2.0 Workflow Management Systems with
Workflow Patterns . In Proc. of CAiSE ’16, June, 2016.

57. Vincenzo Ferme
30
First Application of the Methodology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
[CAiSE ’16]
M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-
Benchmarking BPMN 2.0 Workflow Management Systems with
Workflow Patterns . In Proc. of CAiSE ’16, June, 2016.
Workload
Empty
Script 1
Empty
Script 2
…
0
750
1500
0:00 4:00 10:00

58. Vincenzo Ferme
30
First Application of the Methodology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
[CAiSE ’16]
M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-
Benchmarking BPMN 2.0 Workflow Management Systems with
Workflow Patterns . In Proc. of CAiSE ’16, June, 2016.
WfMS
MySQL
3 WfMSsWorkload
Empty
Script 1
Empty
Script 2
…
0
750
1500
0:00 4:00 10:00

59. Vincenzo Ferme
30
First Application of the Methodology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
[CAiSE ’16]
M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-
Benchmarking BPMN 2.0 Workflow Management Systems with
Workflow Patterns . In Proc. of CAiSE ’16, June, 2016.
WfMS
MySQL
3 WfMSsWorkload
Empty
Script 1
Empty
Script 2
…
0
750
1500
0:00 4:00 10:00
Metrics
• Engine Level
• Process Level
• Environment

60. Vincenzo Ferme
30
First Application of the Methodology
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
[CAiSE ’16]
M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-
Benchmarking BPMN 2.0 Workflow Management Systems with
Workflow Patterns . In Proc. of CAiSE ’16, June, 2016.
Results: relevant differences among WfMSs
WfMS
MySQL
3 WfMSsWorkload
Empty
Script 1
Empty
Script 2
…
0
750
1500
0:00 4:00 10:00
Metrics
• Engine Level
• Process Level
• Environment

61. Vincenzo Ferme
31
Future Work
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Continue to Apply and Improve the Methodology

62. Vincenzo Ferme
31
Future Work
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Continue to Apply and Improve the Methodology
• Involve moreVendors and Researchers as part of
the Benchmarking Effort

63. Vincenzo Ferme
32
Future Work
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Continue to Apply and Improve the Methodology
• Involve more Vendors and Researchers as part of
the Benchmarking Effort

64. Vincenzo Ferme
32
Future Work
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Continue to Apply and Improve the Methodology
• Involve more Vendors and Researchers as part of
the Benchmarking Effort

65. Vincenzo Ferme
33
Future Work
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Continue to Apply and Improve the Methodology
• Involve more Vendors and Researchers as part of
the Benchmarking Effort

66. Vincenzo Ferme
33
Future Work
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Continue to Apply and Improve the Methodology
• Involve more Vendors and Researchers as part of
the Benchmarking Effort
Workshop
September 5th, 2016
http://uniba-dsg.github.io/peace-ws/
1st International Workshop on
Performance and Conformance of Workflow Engines

67. Vincenzo Ferme
34
Highlights

68. Vincenzo Ferme
34
Highlights
Vincenzo Ferme
5
Benchmarking Requirements
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
• K. Huppler, The art of building a good benchmark, 2009
• J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993
• S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A
challenge to software engineering, 2003
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Benchmarking Requirements

69. Vincenzo Ferme
34
Highlights
Vincenzo Ferme
5
Benchmarking Requirements
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
• K. Huppler, The art of building a good benchmark, 2009
• J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993
• S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A
challenge to software engineering, 2003
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Benchmarking Requirements
Vincenzo Ferme
14
Benchmarking Choreography
Vendor
BenchFlow
Provide
Benchmarking
Methodology
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
Benchmarking Methodology Agreement Proposal
Signed Agreement
Vendor
BenchFlow
Provide Containerized
Distribution of WfMS
Containerized WfMS Request
Containerized WfMS
Vendor
BenchFlow
Verify the Benchmark
Results
Results Verification Outcome
Results Verification Request
Vendor
BenchFlow
Provide Draft
Benchmark Results
Draft Benchmark ResultsVerified Benchmark Results
Not Valid
Are the
Results
Valid?
Community
BenchFlow
Publish Benchmark
Results
Valid
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Benchmarking Methodology

70. Vincenzo Ferme
34
Highlights
Vincenzo Ferme
5
Benchmarking Requirements
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
• K. Huppler, The art of building a good benchmark, 2009
• J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993
• S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A
challenge to software engineering, 2003
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Benchmarking Requirements
Vincenzo Ferme
14
Benchmarking Choreography
Vendor
BenchFlow
Provide
Benchmarking
Methodology
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
Benchmarking Methodology Agreement Proposal
Signed Agreement
Vendor
BenchFlow
Provide Containerized
Distribution of WfMS
Containerized WfMS Request
Containerized WfMS
Vendor
BenchFlow
Verify the Benchmark
Results
Results Verification Outcome
Results Verification Request
Vendor
BenchFlow
Provide Draft
Benchmark Results
Draft Benchmark ResultsVerified Benchmark Results
Not Valid
Are the
Results
Valid?
Community
BenchFlow
Publish Benchmark
Results
Valid
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Benchmarking Methodology
Vincenzo Ferme
29
Advantages of using Containers
• Standard APIs to access Environment Metrics
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Accomplish some Benchmarking Requirement:
Portability, Repeatability,Accessibility, Efficiency
• Common way to deploy systems provided by
different vendors
Docker Compose Docker Swarm
Advantages of Containers

71. Vincenzo Ferme
34
Highlights
Vincenzo Ferme
5
Benchmarking Requirements
• Relevant
• Representative
• Portable
• Scalable
• Simple
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
• K. Huppler, The art of building a good benchmark, 2009
• J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993
• S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A
challenge to software engineering, 2003
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Benchmarking Requirements
Vincenzo Ferme
14
Benchmarking Choreography
Vendor
BenchFlow
Provide
Benchmarking
Methodology
Vendor
BenchFlow
Agree on adding
Vendor's WfMS to the
Benchmark
Benchmarking Methodology Agreement Proposal
Signed Agreement
Vendor
BenchFlow
Provide Containerized
Distribution of WfMS
Containerized WfMS Request
Containerized WfMS
Vendor
BenchFlow
Verify the Benchmark
Results
Results Verification Outcome
Results Verification Request
Vendor
BenchFlow
Provide Draft
Benchmark Results
Draft Benchmark ResultsVerified Benchmark Results
Not Valid
Are the
Results
Valid?
Community
BenchFlow
Publish Benchmark
Results
Valid
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
Benchmarking Methodology
Vincenzo Ferme
29
Advantages of using Containers
• Standard APIs to access Environment Metrics
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Accomplish some Benchmarking Requirement:
Portability, Repeatability,Accessibility, Efficiency
• Common way to deploy systems provided by
different vendors
Docker Compose Docker Swarm
Advantages of Containers
Vincenzo Ferme
31
Future Work
Context » Benchmarking Requirements » Methodology Overview » Methodology Details » Advantage of Containers » 1st Application » Future Work
• Continue to Apply and Improve the Methodology
• Involve moreVendors and Researchers as part of
the Benchmarking Effort
Future Work

72. benchflow
benchflow
vincenzo.ferme@usi.ch
http://benchflow.inf.usi.ch
A CONTAINER-CENTRIC
METHODOLOGY FOR
BENCHMARKING WORKFLOW
MANAGEMENT SYSTEMS
Vincenzo Ferme (@VincenzoFerme),
Ana Ivanchikj, Prof. Cesare Pautasso
Faculty of Informatics
University of Lugano (USI), Switzerland
Marigianna Skouradaki,
Prof. Frank Leymann
Institute of Architecture of Application Systems
University of Stuttgart, Germany
Cloud Computing Patt
Fundamentals to Design, Build, and Man
Christoph Fehling
Institute of Architecture of Applicatio
University of Stuttgart
Universitätsstr. 38
70569 Stuttgart

73. BACKUP SLIDES
Vincenzo Ferme, Ana Ivanchikj,
Prof. Cesare Pautasso
Faculty of Informatics
University of Lugano (USI), Switzerland
Marigianna Skouradaki,
Prof. Frank Leymann
Institute of Architecture of Application Systems
University of Stuttgart, Germany
Cloud Computing Patt
Fundamentals to Design, Build, and Man
Christoph Fehling
Institute of Architecture of Applicatio
University of Stuttgart
Universitätsstr. 38
70569 Stuttgart

74. Vincenzo Ferme
37
Published Work
[BTW ’15]
C. Pautasso, V. Ferme, D. Roller, F. Leymann, and M. Skouradaki. Towards workflow
benchmarking: Open research challenges. In Proc. of the 16th conference on
Database Systems for Business,Technology, and Web, BTW 2015, pages 331–350, 2015.
[SSP ’14]
M. Skouradaki, D. H. Roller, F. Leymann, V. Ferme, and C. Pautasso. Technical open
challenges on benchmarking workflow management systems. In Proc. of the 2014
Symposium on Software Performance, SSP 2014, pages 105–112, 2014.
[ICPE ’15]
M. Skouradaki, D. H. Roller, L. Frank, V. Ferme, and C. Pautasso. On the Road to
Benchmarking BPMN 2.0 Workflow Engines. In Proc. of the 6th ACM/SPEC
International Conference on Performance Engineering, ICPE ’15, pages 301–304, 2015.

75. Vincenzo Ferme
38
Published Work
[CLOSER ’15]
M. Skouradaki,V. Ferme, F. Leymann, C. Pautasso, and D. H. Roller. “BPELanon”: Protect
business processes on the cloud. In Proc. of the 5th International Conference on
Cloud Computing and Service Science, CLOSER 2015. SciTePress, 2015.
[SOSE ’15]
M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph
Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models.
In Proc. of the 9th International IEEE Symposium on Service-Oriented System
Engineering, SOSE 2015, 2015.
[BPM ’15]
V. Ferme, A. Ivanchikj, C. Pautasso. A Framework for Benchmarking BPMN 2.0
Workflow Management Systems. In Proc. of the 13th International Conference on
Business Process Management, BPM ’15, pages 251-259, 2015.

76. Vincenzo Ferme
39
Published Work
[BPMD ’15]
A. Ivanchikj,V. Ferme, C. Pautasso. BPMeter: Web Service and Application for Static
Analysis of BPMN 2.0 Collections. In Proc. of the 13th International Conference on
Business Process Management [Demo], BPM ’15, pages 30-34, 2015.
[ICPE ’16]
V. Ferme, and C. Pautasso. Integrating Faban with Docker for Performance
Benchmarking. In Proc. of the 7th ACM/SPEC International Conference on Performance
Engineering, ICPE ’16, 2016.
[CAiSE ’16]
M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-Benchmarking
BPMN 2.0 Workflow Management Systems with Workflow Patterns . In Proc. of
the 28th International Conference on Advanced Information Systems Engineering, CAiSE
’16, 2016.

77. Vincenzo Ferme
40
Docker Performance
[IBM ’14]
W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An updated performance comparison
of virtual machines and Linux containers. IBM Research Report, 2014.
Although containers themselves have almost no overhead,
Docker is not without performance gotchas. Docker
volumes have noticeably better performance than files stored in
AUFS. Docker’s NAT also introduces overhead for workloads
with high packet rates. These features represent a tradeoff
between ease of management and performance and should
be considered on a case-by-case basis.
”
“Our results show that containers result in equal or better
performance than VMs in almost all cases.
“
”
BenchFlow Configures Docker for Performance by Default