Be the first to like this
As opposed to databases for which established benchmarks have been driving the advancement of the field since a long time, workflow engines still lack a well-accepted benchmark that allows to give a fair comparison of their performance. In this talk we discuss how BenchFlow addresses the main challenges related to benchmarking these complex middleware systems at the core of business process automation and service composition solutions. In particular, we look at how to define suitable workloads, representative performance metrics, and how to fully automate the execution of the performance experiments. Concerning the automation of experiments execution and analysis, we designed and implemented the BenchFlow framework. The framework automates the deployment of WfMS packaged within Docker containers. This way the initial configuration and conditions for each experiment can be precisely controlled and reproduced. Moreover, the BenchFlow framework supports heterogeneous WfMS APIs by providing an extensible plugin mechanism. During the benchmark execution it automatically deploys a set of BPMN models and invokes them according to parametric load functions specified declaratively. It automatically collects performance and resource consumption data, both on the driver and servers during the experiment as well as extracting them from the WfMS database afterwards. In addition to latency, throughput and resource utilisation, we compute multiple performance metrics that characterise the WfMS performance at the engine level, at the process level and at the BPMN construct level. To ensure reliability and improve usefulness of the obtained results, we automatically compute descriptive statistics and perform statistical tests to asses the homogeneity of results obtained from different repetitions of the same experiment.
The talk will also present experimental results obtained while benchmarking popular open source engines, using workflow patterns as a micro-benchmark.