Managing and running business processes in the Cloud changes how Workflow Management Systems (WfMSs) are deployed. Consequently, when designing such WfMSs, there is a need of determining the sweet spot in the performance vs. resource consumption trade-off. While all Cloud providers agree on the pay-as-you-go resource consumption model, every provider uses a different cost model to gain a competitive edge. In this paper, we present a novel method for estimating the infrastructure costs of running business processes in the Cloud. The method is based on the precise measurement of the resources required to run a mix of business process in the Cloud, while accomplishing expected performance requirements. To showcase the method we use the BenchFlow framework to run experiments on a widely used open-source WfMS executing custom workload with a varying number of simulated users. The experiments are necessary to reliably measure WfMS’s performance and resource consumption, which is then used to estimate the infrastructure costs of executing such workload on four different Cloud providers. My talk from The International Conference on Business Process Management 2016 (BPM2016). Cite us: http://link.springer.com/chapter/10.1007/978-3-319-45468-9_5

1. Vincenzo Ferme, Ana Ivanchikj, Cesare Pautasso
Faculty of Informatics
USI Lugano, Switzerland
ESTIMATING THE COST FOR
EXECUTING BUSINESS PROCESSES
IN THE CLOUD

2. Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
sweet spot in the performance vs. resource consumption trade-off

3. Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
D
A
B
C
sweet spot in the performance vs. resource consumption trade-off

4. Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
D
A
B
C
IT Resources
sweet spot in the performance vs. resource consumption trade-off

5. Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
D
A
B
C
$
IT Resources
sweet spot in the performance vs. resource consumption trade-off

6. Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
IaaS
D
A
B
C
$
IT Resources
sweet spot in the performance vs. resource consumption trade-off

7. Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
IaaS
D
A
B
C
P
a
a
S
Workload
Users
$
IT Resources
sweet spot in the performance vs. resource consumption trade-off
B
P
a
a
S

8. Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
IaaS
D
A
B
C
P
a
a
S
Workload
Users
$
$
IT Resources
sweet spot in the performance vs. resource consumption trade-off
B
P
a
a
S

9. Vincenzo Ferme
3
Challenges in Deploying WfMSs in the Cloud
evaluate the Cloud cost models, determine the best suitable provider
Evaluate the Cloud Offers

10. Vincenzo Ferme
3
Challenges in Deploying WfMSs in the Cloud
evaluate the Cloud cost models, determine the best suitable provider
Evaluate the Cloud Offers

11. Vincenzo Ferme
3
Challenges in Deploying WfMSs in the Cloud
evaluate the Cloud cost models, determine the best suitable provider
Evaluate the Cloud Offers Best Suitable Provider?

12. Vincenzo Ferme
3
Challenges in Deploying WfMSs in the Cloud
evaluate the Cloud cost models, determine the best suitable provider
Evaluate the Cloud Offers Best Suitable Provider?
$
Performance
Resources

13. Vincenzo Ferme
4
Goal
select candidate VMs according to workload and wanted performance
WfMS

14. Vincenzo Ferme
4
Goal
select candidate VMs according to workload and wanted performance
WfMS
Throughput
Resp. Time

15. Vincenzo Ferme
4
Goal
select candidate VMs according to workload and wanted performance
WfMS
RAM USAGE
CPU USAGE
Throughput
Resp. Time

16. Vincenzo Ferme
4
Goal
select candidate VMs according to workload and wanted performance
WfMS
RAM USAGE
CPU USAGE
CPU + RAM Bound
Throughput
Resp. Time

17. Vincenzo Ferme
4
Goal
select candidate VMs according to workload and wanted performance
WfMS
RAM USAGE
CPU USAGE
CPU + RAM Bound
Throughput
Resp. Time
Matching Clouds VMs
$$
$
$$
$

18. Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D

19. Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function

20. Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.

21. Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.
4. Run Experiments

22. Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.
4. Run Experiments5. Analyse Results
Resources
Perform
ance

23. Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.
4. Run Experiments5. Analyse Results
Resources
Perform
ance
6. Cloud Offers
$

24. Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.
4. Run Experiments5. Analyse Results
Resources
Perform
ance
6. Cloud Offers
$
7. VMs Selection
Res. Efficiency
Cost
$

25. Vincenzo Ferme
6
Assumptions
RAM and CPU, only WfMS no DBMS, workload dependent
CPU
RAM
Observed Resources

26. Vincenzo Ferme
6
Assumptions
RAM and CPU, only WfMS no DBMS, workload dependent
CPU
RAM
No Network, No I/O
Observed Resources

27. Vincenzo Ferme
6
Assumptions
RAM and CPU, only WfMS no DBMS, workload dependent
CPU
RAM
No Network, No I/O
Observed Resources
WfMS
DBMS
Only WfMS

28. Vincenzo Ferme
6
Assumptions
RAM and CPU, only WfMS no DBMS, workload dependent
CPU
RAM
No Network, No I/O
Observed Resources
WfMS
DBMS
Only WfMS
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
Workload
Dependent

29. Vincenzo Ferme
7
An Example of Application
applying the method for the Camunda WfMS
1.
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2.
Metrics
KPIs
3. 4.
5.
Resources
Perform
ance
6.
$
7.
Resources Efficiency
Cost
$

30. Vincenzo Ferme
8
1. The Workload Mix
realistic, mix of 5 models with different complexity
camunda_additional_approval
Change necessary?
Amend
something
(Empty
Script)
Do the next
task
(Empty
Script)
Do some
approval
(Decision
Script)
Approved?
Do the
next task
(Empty
Script)
Counter
instantiation
no
yes
yes
no
21 Elements (Nodes + Edges)

31. Vincenzo Ferme
9
1. The Workload Mix
realistic, mix of 5 models with different complexity
camunda_oopSignavio
Order
Received
Determine
Location
(Decision Script)
Exception
detected?
Escalation
(Empty
Script)
Order
amount?
Order Canceled
Approved?
Send Rejection
Email
(Empty Script)
Order
Rejected
Send Order to WH
(Empty Script)
Receive
Dispatch
Confirmation
(Empty
Script)
Control
Dispatch
(Empty
Script)
Send
Confirmation
Email
(Empty
Script) Order
Confirmed
Order Approval
(Decision Script)
>200
<=200no
yes
no
yes
32 Elements (Nodes + Edges)

32. Vincenzo Ferme
10
1. The Workload Mix
realistic, mix of 5 models with different complexity
camunda_invoice_collaboration
Scan Invoice
(Empty
Script)
Archive
Original
(Empty
Script)
Ask Approver
Assignment
(Empty Script)
Approve
Invoice
(Decision
Script)
Assign
Approver
(Empty
Script)
Invoice
Approved?
Prepare Bank
Transfer
(Empty Script)
Ask Invoice
Review
(Empty
Script)
Review
successful?
Review
Invoice
(Decision
Script)
Archive
Invoice
(Empty
Script)
noyes
no
yes
39 Elements (Nodes + Edges)

33. Vincenzo Ferme
11
1. The Workload Mix
realistic, mix of 5 models with different complexityactiviti_modelSignavio
In-depth Analysis
Medical
Analysis
(Empty
Script)
Risk
Assessment
(Empty
Script)
Send Meeting
Request
(Empty
Script)
F2F Meeting
(Empty
Script)
Calculate
Score
(Empty
Script)
Preliminary
Judgement
(Empty Script)
Send Rejection
Email
(Empty Script)
Approve Loan
Application
(Decision Script)
Approved?
Update
Application With
Calculation
(Empty Script)
Final Evaluation
(Decision Script)
yes
not ok
ok
no
41 Elements (Nodes + Edges)

34. Vincenzo Ferme
12
1. The Workload Mix
realistic, mix of 5 models with different complexity
COUNTERPARTY
Counterparty
Requested
Review
Onboarding
Request
(Decision
Script)
Override
requested?
Review Override
Request
(Decision Script)
Tax
(Empty Script)
Missing Tax
Documents
(Decision Script) Goto tax?
Check Tax Rules
(Decision Script)
Override
requested?
Tax
approved?
Amend Tax Data
(Decision Script)
Override
requested?
High Risk
Country
(Empty Script)
Missing
Compliance
Documents
(Empty Script)
World Check
(Decision
Script) Goto
compliance?
Check Compliance
Rules
(Decision Script)
Override
requested?
Compliance
approved?
Amend Compliance
Data
(Decision Script)
Override
requested?
Start SSI
Review
(Empty Script)
Override
approved?
override rejected
Ready to
publish?
Publish
Counterparty
(Empty Script)
Cancel SSI Review
(Empty Script)
rejected
published
yes
no
yes
no
no
maybe
no
yes
no
maybe
noyes
yes
yes
no
yes
no
yes
no
no
yes
no
yes
yes
84 Elements (Nodes + Edges)

35. Vincenzo Ferme
13
2. The Load Functions
realistic for different sized companies
• 50
• 500
• 1000
users (usr)

36. Vincenzo Ferme
13
2. The Load Functions
realistic for different sized companies
Users
0
200
400
600
800
1000
Time (min:sec)0:000:301:002:003:004:005:006:007:008:0010:00
Example for usr=1000
• 50
• 500
• 1000
users (usr)

37. Vincenzo Ferme
13
2. The Load Functions
realistic for different sized companies
Users
0
200
400
600
800
1000
Time (min:sec)0:000:301:002:003:004:005:006:007:008:0010:00
Example for usr=1000
• 50
• 500
• 1000
users (usr)
1 second think time

38. Vincenzo Ferme
14
3. Performance Requirements
determined with “unbounded” experiments
“Unbounded” (U) Configuration:
CPURAM
128 GB 64 Cores

39. Vincenzo Ferme
14
3. Performance Requirements
determined with “unbounded” experiments
“Unbounded” (U) Configuration:
CPURAM
128 GB 64 Cores
Target Performance

40. Vincenzo Ferme
14
3. Performance Requirements
determined with “unbounded” experiments
“Unbounded” (U) Configuration:
CPURAM
128 GB 64 Cores
Target Performance
• Number of Client Requests
per second (REQ/s)
• Response Time (RT)
• …
Client-side Metrics:

41. Vincenzo Ferme
14
3. Performance Requirements
determined with “unbounded” experiments
“Unbounded” (U) Configuration:
CPURAM
128 GB 64 Cores
Target Performance
• Number of Client Requests
per second (REQ/s)
• Response Time (RT)
• …
• Number of executed BP
Instances (N)
• BP Instance Duration (D)
• Throughput (T)
• …
Client-side Metrics: Server-side Metrics:

42. Vincenzo Ferme
15
4. Run Experiments
the BenchFlow framework
Instance
Database
DBMSFaban Drivers
ContainersServers
DATA
TRANSFORMERS
ANALYSERS
Performance
Metrics
Performance
KPIs
WfMS
TestExecutionAnalyses
Faban
+
Web
Service
Minio
harness
benchflow [BPM ’15] [ICPE ’16]
MONITOR
Adapters
COLLECTORS

43. Vincenzo Ferme
16
5. Analyse Results
define (determine) the wanted performance
WfMS
Configurations
Containers

44. Vincenzo Ferme
16
5. Analyse Results
define (determine) the wanted performance
WfMS
Configurations
Containers
3 Servers
10 Gbit/s 10 Gbit/s
WfMS, DBMS, Load Generator

45. Vincenzo Ferme
16
5. Analyse Results
define (determine) the wanted performance
Three Runs to Improve Results Reliability

46. Vincenzo Ferme
17
5. Analyse Results
determine the right amount of needed resources
Normalised performance over number of CPU cores with 500 Users

47. Vincenzo Ferme
17
5. Analyse Results
determine the right amount of needed resources
Normalised performance over number of CPU cores with 500 Users

48. Vincenzo Ferme
17
5. Analyse Results
determine the right amount of needed resources
Normalised performance over number of CPU cores with 500 Users

49. Vincenzo Ferme
18
5. Analyse Results
validate the performance results
Unbound vs Bound Performance Results

50. Vincenzo Ferme
18
5. Analyse Results
validate the performance results
Unbound vs Bound Performance Results

51. Vincenzo Ferme
19
6. Cloud Offers
match resource requirements with Cloud offers
$$
$
CPU
RAM
CPU + RAM Bound

52. Vincenzo Ferme
20
6. Cloud Offers
match resource requirements with Cloud offers
0 0.2 0.4 0.6 0.8 1 1.2
50 users 500 users 1000 users
Am Az p
Am
Gp
0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
Price ($/hr) 500 users 1000 users
Am Az Gc
Am
Gp
Gc
0.6 0.8 1 1.2 1.4 1.6 1.8 2
Price ($/hr) 1000 users

53. Vincenzo Ferme
21
6. Cloud Offers
match resource requirements with Cloud offers
0 0.2 0.4 0.6 0.8 1 1.2
50 users 500 users 1000 users
Am Az p
Am
Gp
0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
Price ($/hr) 500 users 1000 users
Am Az Gc
Am
Gp
Gc
0.6 0.8 1 1.2 1.4 1.6 1.8 2
Price ($/hr) 1000 users

54. Vincenzo Ferme
22
6. Cloud Offers
match resource requirements with Cloud offers
0 0.2 0.4 0.6 0.8 1 1.2
50 users 500 users 1000 users
Am Az p
Am
Gp
0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
Price ($/hr) 500 users 1000 users
Am Az Gc
Am
Gp
Gc
0.6 0.8 1 1.2 1.4 1.6 1.8 2
Price ($/hr) 1000 users

55. Vincenzo Ferme
23
7. VMs Selection
select the set of suitable VMs, and identify the cheapest ones
Normalised Cores Utilisation vs Price of VMs

56. Vincenzo Ferme
23
7. VMs Selection
select the set of suitable VMs, and identify the cheapest ones
Normalised Cores Utilisation vs Price of VMs

57. Vincenzo Ferme
23
7. VMs Selection
select the set of suitable VMs, and identify the cheapest ones
Normalised Cores Utilisation vs Price of VMs

58. Vincenzo Ferme
24
Limitations and Future Work
approximation of the actual performance on the Cloud
Performance on the Cloud has more Variance

59. Vincenzo Ferme
25
Limitations and Future Work
measure on the Cloud, validate the approach
1. Workload Mix
80%
C
A
B
20%
D

60. Vincenzo Ferme
25
Limitations and Future Work
measure on the Cloud, validate the approach
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.
4. Run Experiments6. Cloud Offers
$
7. VMs Selection
Res. Efficiency
Cost
$
5. Analyse Results
Resources
Perform
ance

61. Vincenzo Ferme
25
Limitations and Future Work
measure on the Cloud, validate the approach
Metrics
KPIs
3. Performance Req.
4. Run Experiments6. Cloud Offers
$
7. VMs Selection
Res. Efficiency
Cost
$
8. Cloud Exp.
5. Analyse Results
Resources
Perform
ance

62. Vincenzo Ferme
25
Limitations and Future Work
measure on the Cloud, validate the approach
Metrics
KPIs
3. Performance Req.
4. Run Experiments6. Cloud Offers
$
7. VMs Selection
Res. Efficiency
Cost
$
8. Cloud Exp.
Add Network, I/O
5. Analyse Results
Resources
Perform
ance

63. Vincenzo Ferme
25
Limitations and Future Work
measure on the Cloud, validate the approach
Metrics
KPIs
3. Performance Req.
4. Run Experiments6. Cloud Offers
$
7. VMs Selection
Res. Efficiency
Cost
$
8. Cloud Exp.
Add Network, I/O
5. Analyse Results
Resources
Perform
ance
Consider DBMS

64. Highlights
Deploying WfMSs to the Cloud
Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
IaaS
D
A
B
C
P
a
a
S
Workload
Users
$
$
IT Resources
sweet spot in the performance vs. resource consumption trade-off
B
P
a
a
S

65. Highlights
Deploying WfMSs to the Cloud
Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
IaaS
D
A
B
C
P
a
a
S
Workload
Users
$
$
IT Resources
sweet spot in the performance vs. resource consumption trade-off
B
P
a
a
S
Proposed Method
Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.
4. Run Experiments5. Analyse Results
Resources
Perform
ance
6. Cloud Offers
$

66. Highlights
Deploying WfMSs to the Cloud
Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
IaaS
D
A
B
C
P
a
a
S
Workload
Users
$
$
IT Resources
sweet spot in the performance vs. resource consumption trade-off
B
P
a
a
S
Proposed Method
Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.
4. Run Experiments5. Analyse Results
Resources
Perform
ance
6. Cloud Offers
$
Application to Camunda
Vincenzo Ferme
23
7. VMs Selection
select the set of suitable VMs, and identify the cheapest ones
Normalised Cores Utilisation vs Price of VMs

67. Highlights
Deploying WfMSs to the Cloud
Vincenzo Ferme
2
Deploying WfMSs to the Cloud
WfMS
IaaS
D
A
B
C
P
a
a
S
Workload
Users
$
$
IT Resources
sweet spot in the performance vs. resource consumption trade-off
B
P
a
a
S
Proposed Method
Vincenzo Ferme
5
The Steps of the Method
from performance requirements to cheapest VMs
1. Workload Mix
80%
C
A
B
20%
D
0
600
1200
0
1
2
3
4
5
6
7
8
9
10
2. Load Function
Metrics
KPIs
3. Performance Req.
4. Run Experiments5. Analyse Results
Resources
Perform
ance
6. Cloud Offers
$
Application to Camunda
Vincenzo Ferme
23
7. VMs Selection
select the set of suitable VMs, and identify the cheapest ones
Normalised Cores Utilisation vs Price of VMs
Limitations and Future Work
Vincenzo Ferme
24
Limitations and Future Work
approximation of the actual performance on the Cloud
Performance on the Cloud has more Variance

68. benchflow
benchflow
vincenzo.ferme@usi.ch
http://benchflow.inf.usi.ch
ESTIMATING THE COST FOR
EXECUTING BUSINESS PROCESSES
IN THE CLOUD
Vincenzo Ferme, Ana Ivanchikj, Cesare Pautasso
Faculty of Informatics
USI Lugano, Switzerland

69. BACKUP SLIDES
Vincenzo Ferme, Ana Ivanchikj, Cesare Pautasso
Faculty of Informatics
USI Lugano, Switzerland

70. Vincenzo Ferme
29
Published Work
[BTW ’15]
C. Pautasso, V. Ferme, D. Roller, F. Leymann, and M. Skouradaki. Towards workflow
benchmarking: Open research challenges. In Proc. of the 16th conference on
Database Systems for Business,Technology, and Web, BTW 2015, pages 331–350, 2015.
[SSP ’14]
M. Skouradaki, D. H. Roller, F. Leymann, V. Ferme, and C. Pautasso. Technical open
challenges on benchmarking workflow management systems. In Proc. of the 2014
Symposium on Software Performance, SSP 2014, pages 105–112, 2014.
[ICPE ’15]
M. Skouradaki, D. H. Roller, L. Frank, V. Ferme, and C. Pautasso. On the Road to
Benchmarking BPMN 2.0 Workflow Engines. In Proc. of the 6th ACM/SPEC
International Conference on Performance Engineering, ICPE ’15, pages 301–304, 2015.

71. Vincenzo Ferme
30
Published Work
[CLOSER ’15]
M. Skouradaki,V. Ferme, F. Leymann, C. Pautasso, and D. H. Roller. “BPELanon”: Protect
business processes on the cloud. In Proc. of the 5th International Conference on
Cloud Computing and Service Science, CLOSER 2015. SciTePress, 2015.
[SOSE ’15]
M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph
Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models.
In Proc. of the 9th International IEEE Symposium on Service-Oriented System
Engineering, SOSE 2015, 2015.
[BPM ’15]
V. Ferme, A. Ivanchikj, C. Pautasso. A Framework for Benchmarking BPMN 2.0
Workflow Management Systems. In Proc. of the 13th International Conference on
Business Process Management, BPM ’15, pages 251-259, 2015.

72. Vincenzo Ferme
31
Published Work
[BPMD ’15]
A. Ivanchikj,V. Ferme, C. Pautasso. BPMeter: Web Service and Application for Static
Analysis of BPMN 2.0 Collections. In Proc. of the 13th International Conference on
Business Process Management [Demo], BPM ’15, pages 30-34, 2015.
[ICPE ’16]
V. Ferme, and C. Pautasso. Integrating Faban with Docker for Performance
Benchmarking. In Proc. of the 7th ACM/SPEC International Conference on Performance
Engineering, ICPE ’16, 2016.
[CLOSER ’16]
V. Ferme, A. Ivanchikj, C. Pautasso., M. Skouradaki, F. Leymann. A Container-centric
Methodology for Benchmarking Workflow Management Systems. In Proc. of the
6th International Conference on Cloud Computing and Service Science, CLOSER 2016.
SciTePress, 2016.

73. Vincenzo Ferme
[CAiSE ’16]
M. Skouradaki, V. Ferme, C. Pautasso, F. Leymann, A. van Hoorn. Micro-Benchmarking
BPMN 2.0 Workflow Management Systems with Workflow Patterns . In Proc. of
the 28th International Conference on Advanced Information Systems Engineering, CAiSE
’16, 2016.
32
Published Work
[ICWE ’16]
C. Jürgen,V. Ferme, H.C. Gall. Using Docker Containers to Improve Reproducibility
in Software and Web Engineering Research. In Proc. of the 16th International
Conference on Web Engineering, 2016.
[ICWS ’16]
M. Skouradaki,V.Andrikopoulos, F. Leymann. Representative BPMN 2.0 Process Model
Generation from Recurring Structures. In Proc. of the 23rd IEEE International
Conference on Web Services, ICWS '16, 2016.

74. Vincenzo Ferme
33
Published Work
[SummerSOC ’16]
M. Skouradaki, T. Azad, U. Breitenbücher, O. Kopp, F. Leymann. A Decision Support
System for the Performance Benchmarking of Workflow Management Systems.
In Proc. of the 10th Symposium and Summer School On Service-Oriented Computing,
SummerSOC '16, 2016.
[OTM ’16]
M. Skouradaki, V. Andrikopoulos, O. Kopp, F. Leymann. RoSE: Reoccurring Structures
Detection in BPMN 2.0 Process Models Collections. In Proc. of On the Move to
Meaningful Internet Systems Conference, OTM '16, 2016. (to appear)
[BPM Forum ’16]
V. Ferme, A. Ivanchikj, C. Pautasso. Estimating the Cost for Executing Business
Processes in the Cloud. In Proc. of the 14th International Conference on Business
Process Management, BPM Forum ’16, 2016.

75. Vincenzo Ferme
34
Docker Performance
[IBM ’14]
W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An updated performance comparison
of virtual machines and Linux containers. IBM Research Report, 2014.
Although containers themselves have almost no overhead,
Docker is not without performance gotchas. Docker
volumes have noticeably better performance than files stored in
AUFS. Docker’s NAT also introduces overhead for workloads
with high packet rates. These features represent a tradeoff
between ease of management and performance and should
be considered on a case-by-case basis.
”
“Our results show that containers result in equal or better
performance than VMs in almost all cases.
“
”
BenchFlow Configures Docker for Performance by Default

76. Vincenzo Ferme
35
•We want to characterise the Workload Mix using Real-World process models
• Share your executable BPMN 2.0 process models, even anonymised
Process Models
•We want to characterise the Load Functions using Real-World behaviours
• Share your execution logs, even anonymised
Execution Logs
•We want to add more and more WfMSs to the benchmark
• Contact us for collaboration, and BenchFlow framework support
WfMSs
Call for Collaboration
WfMSs, process models, process logs

77. Vincenzo Ferme
36
Cloud WfMSs, business process optimisations, capacity planning
State of the Art of WfMSs in the Cloud
• Cloud WfMSs: emerging market
• BPaaS: started to be evaluated
• Business Process Execution Optimisations
• Cost-aware WfMSs

78. Vincenzo Ferme
36
Cloud WfMSs, business process optimisations, capacity planning
State of the Art of WfMSs in the Cloud
• Cloud WfMSs: emerging market
• BPaaS: started to be evaluated
• Business Process Execution Optimisations
• Cost-aware WfMSs
We Focus on Capacity Planning
with a Widely Used WfMSs

79. Vincenzo Ferme
37
BenchFlow Framework
system under test
Docker Engine

80. Vincenzo Ferme
37
BenchFlow Framework
system under test
Docker Engine
Containers

81. Vincenzo Ferme
37
BenchFlow Framework
system under test
Docker Engine
Docker Machine
provides
Containers

82. Vincenzo Ferme
37
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Machine
provides
Containers

83. Vincenzo Ferme
37
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Machine
provides
manages
Containers
Servers

84. Vincenzo Ferme
37
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Compose
Docker Machine
provides
manages
Containers
Servers

85. Vincenzo Ferme
37
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Compose
Docker Machine
provides
manages
Containers
Servers
SUT’s Deployment Conf.
deploys

86. Vincenzo Ferme
38
Server-side Data and Metrics Collection
WfMS
Start Workflow
asynchronous execution of workflows

87. Vincenzo Ferme
Users
D
A
B
C
Start
Application Server
Web
Service
Load Driver
DBMS
WfMS
Instance
Database
38
Server-side Data and Metrics Collection
WfMS
Start Workflow
asynchronous execution of workflows

88. Vincenzo Ferme
Users
D
A
B
C
Start
Application Server
Web
Service
Load Driver
DBMS
WfMS
Instance
Database
38
Server-side Data and Metrics Collection
WfMS
Start Workflow
End
asynchronous execution of workflows

89. Vincenzo Ferme
39
Server-side Data and Metrics Collection
monitors
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service

90. Vincenzo Ferme
39
Server-side Data and Metrics Collection
monitors
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
MONITOR

91. Vincenzo Ferme
40
Server-side Data and Metrics Collection
monitors
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
• CPU usage
• Database state
Examples of Monitors:Monitors’ Characteristics:
• RESTful services
• Lightweight (written in Go)
• As less invasive on the SUT as possible

92. Vincenzo Ferme
40
Server-side Data and Metrics Collection
monitors
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
CPU
Monitor
DB
Monitor
• CPU usage
• Database state
Examples of Monitors:Monitors’ Characteristics:
• RESTful services
• Lightweight (written in Go)
• As less invasive on the SUT as possible

93. Vincenzo Ferme
41
Server-side Data and Metrics Collection
collect data
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service

94. Vincenzo Ferme
41
Server-side Data and Metrics Collection
collect data
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Minio
Instance
Database
Analyses
COLLECTORS

95. Vincenzo Ferme
42
Server-side Data and Metrics Collection
collect data
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
• Container’s Stats (e.g., CPU usage)
• Database dump
• Applications Logs
Examples of Collectors:Collectors’ Characteristics:
• RESTful services
• Lightweight (written in Go)
• Two types: online and offline
• Buffer data locally

96. Vincenzo Ferme
42
Server-side Data and Metrics Collection
collect data
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Stats
Collector
DB
Collector
• Container’s Stats (e.g., CPU usage)
• Database dump
• Applications Logs
Examples of Collectors:Collectors’ Characteristics:
• RESTful services
• Lightweight (written in Go)
• Two types: online and offline
• Buffer data locally