Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Research
Dieter H. Roller, Frank Leymann, Marigianna Skouradaki,
Institute of Architecture of Application Systems
{skourad...
22
Research
© Marigianna Skouradaki
Motivation
*the appearance and company selection are random
33
Research
© Marigianna Skouradaki
Agenda
 Performance Factors
 Logistic Challenges on Benchmarking
 Technical Challen...
44
Research
© Marigianna Skouradaki
What is a Workflow Engine?
4.Navigator
3.Web Services
6.Application
Server
Workflow
En...
5
Logistic Challenges
66
Research
© Marigianna Skouradaki

Challenge: Process Model Collection
 More than 8.000 Process Models
77
Research
© Marigianna Skouradaki
Challenge: Process Model Collection -  Data and Timing Information
Load Distribution?
88
Research
© Marigianna Skouradaki
Challenge: Process Model Collection -  Data Flow and Timing Information
80%
20%
Proce...
99
Research
© Marigianna Skouradaki
Challenge: Synthesizing Benchmark Flows
BPMN 2.0
Process Models
Collection
Reflect Div...
1010
Research
© Marigianna Skouradaki
Challenge: What to Measure?
• Throughput
• Response Time
• Required Memory
• Power C...
11
Technical Challenges
1212
Research
© Marigianna Skouradaki
Challenge: Eliminate Impact of External Factors
Workflow
Engine
requestclient
respon...
1313
Research
© Marigianna Skouradaki
Challenge: Prevent System Overloading
Workflow
Engine
request
client
request
request...
1414
Research
© Marigianna Skouradaki
Challenge: Reliability, Recovery, Robustness
Workflow
Engine
request
client
request
...
1515
Research
© Marigianna Skouradaki
Challenge: System Internal Load Optimization
request
client
response
Workflow
Engine...
1616
Research
© Marigianna Skouradaki
Challenge: Long Running Processes
request
client
response
Workflow
Engine
3 months
B...
17
Project Status
1818
Research
© Marigianna Skouradaki
Collected Process Models
1960
3168
2177
608
21 427
Models (Non-Executable Files)
.pn...
© Marigianna Skouradaki 19
Research
Generation of Realistic Workload
BPMN 2.0
Process Models
Collection
Fragments
Reposito...
20
Research
Workflow
Engine
Enabling the Benchmark Execution
harness
Web
ServiceDBMS
1. Flexible Deployment
2. Flexible HW...
21
Research
1. Automatically deploy and start the benchmark
environment;
Enabling the Benchmark Execution
harness
Faban Dr...
2222
Research
© Marigianna Skouradaki
Conclusions & Future Work
BenchFlow Project:
http://www.iaas.uni-stuttgart.de/forsch...
Upcoming SlideShare
Loading in …5
×

Benchmarking Workflow Engines: Open Research Challenges - Presentation of BTW 2015

1,023 views

Published on

Workflow Management Systems (WfMS) provide platforms for delivering
complex service-oriented applications that need to satisfy enterprise-grade quality of
service requirements such as dependability and scalability. The performance of these
applications largely depends on the performance of the WfMS supporting them. Com-
paring the performance of different WfMSs and optimizing their configuration requires
that appropriate benchmarks are made available. In this position paper we make the
case for benchmarking the performance of WfMSs that are compliant with the Busi-
ness Process Model and Notation 2.0 standard and explore most of the challenges that
one must tackle when constructing such benchmarks

by Pautasso C., Ferme V., Roller H.D., Leymann F., Skouradaki M.

Published in: Software
  • Be the first to comment

Benchmarking Workflow Engines: Open Research Challenges - Presentation of BTW 2015

  1. 1. Research Dieter H. Roller, Frank Leymann, Marigianna Skouradaki, Institute of Architecture of Application Systems {skouradaki, dieter.h.roller,leymann}@iaas.uni-stuttgart.de Cesare Pautasso, Vincenzo Ferme Faculty of Informatics University of Lugano Switzerland {firstname.lastname@usi.ch} Towards Workflow Engine Benchmarking: Open Research Challenges
  2. 2. 22 Research © Marigianna Skouradaki Motivation *the appearance and company selection are random
  3. 3. 33 Research © Marigianna Skouradaki Agenda  Performance Factors  Logistic Challenges on Benchmarking  Technical Challenges on Benchmarking  Project Status  Conclusions & Future Work
  4. 4. 44 Research © Marigianna Skouradaki What is a Workflow Engine? 4.Navigator 3.Web Services 6.Application Server Workflow Engine 5.Instance Database 2.Users & IT tools … 1.Process Models
  5. 5. 5 Logistic Challenges
  6. 6. 66 Research © Marigianna Skouradaki  Challenge: Process Model Collection  More than 8.000 Process Models
  7. 7. 77 Research © Marigianna Skouradaki Challenge: Process Model Collection -  Data and Timing Information Load Distribution?
  8. 8. 88 Research © Marigianna Skouradaki Challenge: Process Model Collection -  Data Flow and Timing Information 80% 20% Process Logs Time for Completion
  9. 9. 99 Research © Marigianna Skouradaki Challenge: Synthesizing Benchmark Flows BPMN 2.0 Process Models Collection Reflect Diversity Usage Patterns of BPMN 2.0 Compliant with Workflow Engines Benchmark Models General vs. Domain Specific
  10. 10. 1010 Research © Marigianna Skouradaki Challenge: What to Measure? • Throughput • Response Time • Required Memory • Power Consumption Workflow Engine request Workflow Engine response • Network Traffic • SLA violations • Failure Rate • Recovery Time • … client
  11. 11. 11 Technical Challenges
  12. 12. 1212 Research © Marigianna Skouradaki Challenge: Eliminate Impact of External Factors Workflow Engine requestclient response monitoring
  13. 13. 1313 Research © Marigianna Skouradaki Challenge: Prevent System Overloading Workflow Engine request client request request Adjust delay rate between requests! request request
  14. 14. 1414 Research © Marigianna Skouradaki Challenge: Reliability, Recovery, Robustness Workflow Engine request client request request request request Control and measure failure?
  15. 15. 1515 Research © Marigianna Skouradaki Challenge: System Internal Load Optimization request client response Workflow Engine Throughput?
  16. 16. 1616 Research © Marigianna Skouradaki Challenge: Long Running Processes request client response Workflow Engine 3 months Benchmark in reasonable time?
  17. 17. 17 Project Status
  18. 18. 1818 Research © Marigianna Skouradaki Collected Process Models 1960 3168 2177 608 21 427 Models (Non-Executable Files) .pnml .bpmn2 .bpmn .yawl .bpel .EPC Total: 8363  IBM Industry Models  BPM Academic Initiative  Research Projects
  19. 19. © Marigianna Skouradaki 19 Research Generation of Realistic Workload BPMN 2.0 Process Models Collection Fragments Repository Workload Mix Graph Matching Selection Criteria Composition Criteria 80% 20%
  20. 20. 20 Research Workflow Engine Enabling the Benchmark Execution harness Web ServiceDBMS 1. Flexible Deployment 2. Flexible HW Resources 3. Frozen Initial Condition Faban Drivers Docker Containers Servers A B C D Load Functions Vincenzo Ferme
  21. 21. 21 Research 1. Automatically deploy and start the benchmark environment; Enabling the Benchmark Execution harness Faban Drivers 3. Determine when the benchmark ends;Faban + Workflow Engine DBMS A B C D MONITOR Instance Database COLLECTORS 2. Automatically deploy the workload mix; Vincenzo Ferme 4. Collect the execution and process logs.
  22. 22. 2222 Research © Marigianna Skouradaki Conclusions & Future Work BenchFlow Project: http://www.iaas.uni-stuttgart.de/forschung/projects/benchflow.php Benchmark long running processes? Throughput in asynchronous processes execution? Optimal delay rate in requests? Release the first prototype for Workload Mix Synthesizer First experiments with KPIs definition and computation Collect more processes Collect process execution logs

×