SlideShare a Scribd company logo
1 of 56
Download to read offline
Hands-on Webinar
Camunda BPM 7.2
Performance and Scalability
Daniel Meyer
 Process Engine Expert
 Technical Project Lead
@meyerdan | daniel.meyer@camunda.com
Bernd Rücker
 10+ years experience with workflow and Java
 Co-Founder of Camunda
 Evangelist & Head of Consulting
@berndruecker | bernd.ruecker@camunda.com
Your speakers today
Performance is a difficult topic
 It always depends
−On hardware
−On software environment (OS, Java, App Server, Database, …)
−On Service Tasks in the process
−On network topology (e.g. remote database, web services, …)
−On concurrent requests, database load, …
 There is no simple answer to performance
 But we always succeed – in each and every real-life situation
−Handling millions of process instances / day
−Handling more than 1.000 process instances / second
−Handling thousands of parallel users
Performance is a difficult topic
We are much faster than competition
see http://camunda.com/landing/whitepaper-camunda-jbpm/
In our tests, Camunda‘s
throughput was 10x –
30x higher than with
JBoss jBPM.
1. Understand basic engine architecture
2. Understand influence parameters on performance
3. Discuss performance improvement approaches
4. See example figures / measurements
5. Discuss future scenarios (e.g. sharding, NoSQL, …)
What we do today
Basic Engine Architecture
We use
Optimistic
Locking
Runtime vs. History Database
Runtime database schema
Learning #1:
The architecture it damn simple –
and the bottleneck is not the
process engine!
Biggest influence on Performance
Database Delegation Code
Call Service
Clustering via shared database
Learning #2:
All state is in the database so
clustering gets really easy.
camunda scales!
More on this later…
„But what can I do if performance IS a problem?“
1. Tasklist
2. (History) Queries
3. Job Execution
Typical Areas of performance issues
 Process/Task Variables
−Show in list
−Use in Search/Filter
 Support for Pagination
 Big number of users accessing the tasklist very often
Implementation challenge
 Provide a generic database schema
 Complex data types are serialized – no SQL-JOIN possible
 Variables are stored in one row per variable – multiple SQL-JOINs
might be required
 Some customers use 10-30 variables
Tasklist Requirements
 Add Process Variables optimized (and only used) for Queries
−Extract attributes
−Combine variables to work with LIKE
 Use own queries
−Native – if you want to improve the WHERE
−Custom – if you want to SELECT multiple information at once
 Own TaskInfo or ProcessInstanceInfo entities
−Persisted as MyBatis or JPA entities
−Combine all attributes – allow to query tasks without (or with one)
JOIN only
−Synchronisation via Listener – or use ProcessInstanceInfo as single
source
Solution Approaches: Tasklist
Example
Customer
- customerId
- company
- …
Your DB camunda
PROCESS_VARIABLES
customerId
...
searchField
4711
...
4711#camunda#Berlin#...
1
2
Native
Query:
3
Custom
Query:
4
Java API –
results are
camunda „Task“
entities
Own MyBatis
mapping – result
can be anything.
Called via
custom code.
Example
TaskInfo
- taskId
- customerId
- companyName
- contractId
- productName
- …
Your DB
camunda
PROCESS_VARIABLES
customerId
contractId
productId
4711
0815
42
5
TaskInfo Entity
(or ProcessInstanceInfo)
 The challenge:
−Indexes cost space and performance in writing data
−We provide a generic database schema without knowing what you
exactly do with it
−We constantly work on the right balance of too many and too less
indexes
 What you can do:
−Check indexes and slow query log
−Add index where appropriate for your situation (perfectly OK with
us, you do not loose support!)
−As Enterprise Customer you can always discuss/validate changes
with support
 Example: create index PROC_DEF_ID_END_TIME ON
ACT_HI_PROCINST (PROC_DEF_ID_,END_TIME_)
(History) Queries
You can also customize history
Custom History
(e.g.
ElasticSearch)
Different History Levels:
- NONE
- ACTIVITY
- AUDIT
- FULL
- CUSTOM (own Filter written
in Java, e.g. „only variable
X“, „not process Y“, …)
Example for custom log level:
https://github.com/camunda/camunda-
bpm-examples/tree/master/process-
engine-plugin/custom-history-level
Job Execution
 Asynchronous Continuation involve Jobs
 Jobs are stored in the database
 Job Executor can be configured
−Number of Worker Threads
−Number of Jobs fetched with one database query
−Size of in-memory Queue
−Lock Time, Retry Behavior, …
 Job Execution can be distributed over a Cluster
 Optimizing is not a straight forward task, hard to give general advise
 If you need to improve: Measure and benchmark configurations in
your environment!
Job Execution
The good news: We did big performance improvements in Camunda
BPM 7.2!
 Improved First Level Cache (throughput increased by up to 90% if
async Service Tasks are executed in a row)
 Improved locking to have less Optimistic Lock Exceptions and more
Jobs acquired per Acquisition. Results in bigger Clusters getting
possible.
Job Execution in Camunda BPM 7.2
Recap:
 Added log level “CUSTOM” for History
 First Level Cache
 Job Executor Acquisition Locking
Plus:
 Added flush ordering (comparable to Hibernate) to minimize risk of
deadlocks
Summary: Performance Improvements in 7.2
Learning #3:
All performance challenges can be
solved.
This is
AWESOME!
Recommendation: Measure! No guessing.
camunda engine
Process
Application
External
Load
Generator
e.g. JMeter,
HP Load Runner,
CURL, …
REST
„close to production“
environment
- Measure
- JobExecutor Horizontal Scalability
- Impact of 1st level cache reuse
- Improvements Version 7.1.0 vs. Version 7.2.0
- Environment: Amazon AWS Cloud (EC2 & RDS)
Benchmark
Benchmark Setup
Client
Process Engine
Node 1
Process Engine
Node 2
Process Engine
Node 3
Process Engine
Node 4
Start Process
Instance (Rest API)
Database
(Postgres)
https://github.com/meyerdan/ec2-benchmark
EC2 m3.xlarge
(Intel Xeon E5-2670 v2,
4 core, 15 GiB Memory)
EC2 m3.xlarge
(Intel Xeon E5-2670 v2,
4 core, 15 GiB Memory)
EC2 db.m3.xlarge
(Intel Xeon E5-2670 v2,
4 core, 15 GiB Memory)
Provisioned using Docker
EC2
Benchmark Setup - The process
- All service tasks „Async“
- 1st service task creates 5 variables
- Variables are read by subsequent service tasks
 Throughput in terms of transactions / second
 No absolute Numbers 
Benchmark Results
Benchmark Results (1)
Benchmark Results (1)
Benchmarks Results (2)
Benchmarks Results
Cache Off Cache On
Amazon RDS Metrics
Benchmarks Results
Cache Off
Cache On
Amazon RDS Metrics
What about true
Horizontal Scalability?
What is Horizontal Scalability?
Scale up the number of transactions executed by adding more
processing nodes to the system. [*]
[*] http://en.wikipedia.org/wiki/Scalability#Horizontal_and_vertical_scaling (Adapted)
Horizontal Scalability
transactions /
sec
nodes
The current Situation
Scale number of Process Engine Nodes (JVMs)
Up to a certain point
Limited possibilities for scaling the shared
relational Database. In a sense this can
only be scaled “up”, not “out”.
Shared
Relational
Database
Process
Engine
Process
Engine
Process
Engine
Which way to go?
Distributed
Datastore
Process
Engine
Process
Engine
Process
Engine
Distributed Datastore.
Use a database which is itself a
distributed system and can be
scaled horizontally.
- Apache Cassandra,
- Apache HBase,
- Distributed Caches
(Hazelcast, …)
- ...
Sharding and partitioning.
Distribute the state over multiple
Datastores.
- Multiple instances of
PostgreSQL
- Each “DB” is a Mongo DB
shard
- No “DB” at all: use a
filesystem journal?
- ...
Key Difference: on the right hand side, the process engine itself is “distributed”
in the sense that it is aware of the distribution and sharding.
The problem with Distributed Datastores
(In the context of process engines)
1. Consistency guarantees offered by these databases (eventual consistency, ACID vs.
BASE, ...) often do not match the requirements of BPMN process execution. See:
conflicting concurrent transactions:
a. Racing incoming signals (E.g.: Two Messages targeting the same event instance arrive at the
same time)
b. Joins & Synchronization (E.g.: Gateways, Multi Instance, ...)
c. Cancel Activity instance (E.g.: Interrupting Message Boundary Event)
1. Data Representation and Network Latency / Overhead: Process instance state is
composite:
a. Token state / active activity instances
b. Variables
c. Task Information, …
Challenge is to find a data representation which does not lead to distribution of the state of a
single process instance across the cluster while still supporting the required access patterns.
2. Significant differences between individual technologies while there are no
industry standards in place yet. (Different with SQL).
Sharding => Distributed yet Local
Scale horizontally...
Each “shard / node” maitains its state
locally
Partitioning workflow instance state
- Each process instance lives inside a single shard /
partition
=> local data consistency easy to guarantee,
=> easy to access efficiently
=> Support range of different persistence engines
(Relational Database, Non-Relational Databases, …)
Proces
s
Engine
Flexible Architecture
...
Reality @
zalando 2014
Proces
s
Engine
Proces
s
Engine
The simplest case
A single process engine node
running on top of a
conventional database.
A medium Scenario
Horizontally scale on top of a
conventional database.
Massive Compute Cluster
500 Nodes ?
All of this should be possible with one
unified architecture!
No more Search!
The catch
“Find Process Instance
for order with ID 43543242”
??
???
Human Workflow (Build Task Lists)
History: Monitoring, Reporting, …
Message Correlation
When is „Search“ required?
Message Correlation
The Problem to solve
Workflow Instance State
for order with ID 435345
Incoming Message:
“customer cancelled Order
with ID 435345”
 Yes, but for non-workflow execution Use Cases
Use Search Index?
(A)sync
Updates
Search Index
(Near Realtime)
Tasklist
Queries,
Monitoring,...
Vision
HistoryTasksCore Process Execution
Signal / Cancel Activity Instance by Id
Correlate Message
Query for List of Tasks Monitoring,
Reports
Real Time, Strongly Consistent
Horizontally scalable through sharding
Multiple persistence technologies possible
Near Real Time, Eventually Consistent
Use best technology for the Job.
Async Event Stream
But still...
HistoryTasksCore Process Execution
Signal / Cancel Activity Instance by Id
Correlate Message
Query for List of Tasks Monitoring,
Reports
In the simplest case!
Learning #4:
You can do true horizontal
clustering with the engine which
exists today!
There is no need for No-SQL
persistence in the core engine.
Learning #5:
Camunda is really damn smart :-)
Camunda BPM Performance is already awesome
However: We are continuously improving
performance
There are strategies to solve specific performance
challenges
There is no limit in scalability
Summary
Start now!
Open Source Edition
• Download:
www.camunda.org
• Docs, Tutorials etc.
• Forum
• Meetings
Enterprise Edition
• Trial:
www.camunda.com
• Additional Features
• Support, Patches etc.
• Consulting, Training
http://camunda.com/bpm/consultation/
info@camunda.com | US +1.415.800.3908 | DE +49 30 664040 900
Thank you!
Questions?

More Related Content

What's hot

Camunda BPM 7.13 Webinar
Camunda BPM 7.13 WebinarCamunda BPM 7.13 Webinar
Camunda BPM 7.13 Webinar
camunda services GmbH
 
Data Power Architectural Patterns - Jagadish Vemugunta
Data Power Architectural Patterns - Jagadish VemuguntaData Power Architectural Patterns - Jagadish Vemugunta
Data Power Architectural Patterns - Jagadish Vemugunta
floridawusergroup
 

What's hot (20)

Camunda BPM 7.13 Webinar
Camunda BPM 7.13 WebinarCamunda BPM 7.13 Webinar
Camunda BPM 7.13 Webinar
 
IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction
 
Developing event-driven microservices with event sourcing and CQRS (phillyete)
Developing event-driven microservices with event sourcing and CQRS (phillyete)Developing event-driven microservices with event sourcing and CQRS (phillyete)
Developing event-driven microservices with event sourcing and CQRS (phillyete)
 
MySQL SQL Tutorial
MySQL SQL TutorialMySQL SQL Tutorial
MySQL SQL Tutorial
 
Demystifying AuthN/AuthZ Using OIDC & OAuth2
Demystifying AuthN/AuthZ Using OIDC & OAuth2Demystifying AuthN/AuthZ Using OIDC & OAuth2
Demystifying AuthN/AuthZ Using OIDC & OAuth2
 
Deploy MySQL e Performance Tuning - 3º Zabbix Meetup do Interior
Deploy MySQL e Performance Tuning - 3º Zabbix Meetup do InteriorDeploy MySQL e Performance Tuning - 3º Zabbix Meetup do Interior
Deploy MySQL e Performance Tuning - 3º Zabbix Meetup do Interior
 
IBM Cloud Integration Platform Introduction - Integration Tech Conference
IBM Cloud Integration Platform Introduction - Integration Tech ConferenceIBM Cloud Integration Platform Introduction - Integration Tech Conference
IBM Cloud Integration Platform Introduction - Integration Tech Conference
 
MySQL Shell - the best DBA tool !
MySQL Shell - the best DBA tool !MySQL Shell - the best DBA tool !
MySQL Shell - the best DBA tool !
 
MySQL Slow Query log Monitoring using Beats & ELK
MySQL Slow Query log Monitoring using Beats & ELKMySQL Slow Query log Monitoring using Beats & ELK
MySQL Slow Query log Monitoring using Beats & ELK
 
Software architecture for high traffic website
Software architecture for high traffic websiteSoftware architecture for high traffic website
Software architecture for high traffic website
 
Distributed Lock Manager
Distributed Lock ManagerDistributed Lock Manager
Distributed Lock Manager
 
Deep Dive on Amazon Aurora
Deep Dive on Amazon AuroraDeep Dive on Amazon Aurora
Deep Dive on Amazon Aurora
 
Install Redis on Oracle Linux
Install Redis on Oracle LinuxInstall Redis on Oracle Linux
Install Redis on Oracle Linux
 
Camunda BPM at bpmNEXT 2016
Camunda BPM at bpmNEXT 2016Camunda BPM at bpmNEXT 2016
Camunda BPM at bpmNEXT 2016
 
Redis introduction
Redis introductionRedis introduction
Redis introduction
 
ELK, a real case study
ELK,  a real case studyELK,  a real case study
ELK, a real case study
 
[124]네이버에서 사용되는 여러가지 Data Platform, 그리고 MongoDB
[124]네이버에서 사용되는 여러가지 Data Platform, 그리고 MongoDB[124]네이버에서 사용되는 여러가지 Data Platform, 그리고 MongoDB
[124]네이버에서 사용되는 여러가지 Data Platform, 그리고 MongoDB
 
Redis cluster
Redis clusterRedis cluster
Redis cluster
 
MariaDB Performance Tuning and Optimization
MariaDB Performance Tuning and OptimizationMariaDB Performance Tuning and Optimization
MariaDB Performance Tuning and Optimization
 
Data Power Architectural Patterns - Jagadish Vemugunta
Data Power Architectural Patterns - Jagadish VemuguntaData Power Architectural Patterns - Jagadish Vemugunta
Data Power Architectural Patterns - Jagadish Vemugunta
 

Viewers also liked

Viewers also liked (20)

Order Processing at Scale: Zalando at Camunda Community Day
Order Processing at Scale: Zalando at Camunda Community DayOrder Processing at Scale: Zalando at Camunda Community Day
Order Processing at Scale: Zalando at Camunda Community Day
 
Camunda BPM at Zalando: Order Processing at scale
Camunda BPM at Zalando: Order Processing at scaleCamunda BPM at Zalando: Order Processing at scale
Camunda BPM at Zalando: Order Processing at scale
 
Camunda Docker
Camunda DockerCamunda Docker
Camunda Docker
 
Camunda BPM in DACH
Camunda BPM in DACHCamunda BPM in DACH
Camunda BPM in DACH
 
DMN in Camunda BPM 7.6
DMN in Camunda BPM 7.6DMN in Camunda BPM 7.6
DMN in Camunda BPM 7.6
 
Camunda BPM in Distributed Applications - Allianz Indonesia
Camunda BPM in Distributed Applications - Allianz IndonesiaCamunda BPM in Distributed Applications - Allianz Indonesia
Camunda BPM in Distributed Applications - Allianz Indonesia
 
Camunda BPM 7.2 - English
Camunda BPM 7.2 - EnglishCamunda BPM 7.2 - English
Camunda BPM 7.2 - English
 
Sneak Preview: Camunda Optimize
Sneak Preview: Camunda OptimizeSneak Preview: Camunda Optimize
Sneak Preview: Camunda Optimize
 
Camunda BPM 7.4 - What can you expect from the next release?
Camunda BPM 7.4 - What can you expect from the next release?Camunda BPM 7.4 - What can you expect from the next release?
Camunda BPM 7.4 - What can you expect from the next release?
 
CMMN in Camunda BPM 7.6
CMMN in Camunda BPM 7.6CMMN in Camunda BPM 7.6
CMMN in Camunda BPM 7.6
 
camunda for developer-friendly BPM
camunda for developer-friendly BPMcamunda for developer-friendly BPM
camunda for developer-friendly BPM
 
Case study: Camunda BPM in PwC project
Case study: Camunda BPM in PwC projectCase study: Camunda BPM in PwC project
Case study: Camunda BPM in PwC project
 
Camunda bpm 7.0-release-webinar-en
Camunda bpm 7.0-release-webinar-enCamunda bpm 7.0-release-webinar-en
Camunda bpm 7.0-release-webinar-en
 
camunda BPM + Apache Camel
camunda BPM + Apache Camelcamunda BPM + Apache Camel
camunda BPM + Apache Camel
 
Programacion de una tienda virtual en Grails
Programacion de una tienda virtual en GrailsProgramacion de una tienda virtual en Grails
Programacion de una tienda virtual en Grails
 
SFScon16 - Edmund Schöpf: "Camunda BPM in Banking"
SFScon16 - Edmund Schöpf: "Camunda BPM in Banking"SFScon16 - Edmund Schöpf: "Camunda BPM in Banking"
SFScon16 - Edmund Schöpf: "Camunda BPM in Banking"
 
camunda latest features and roadmap
camunda latest features and roadmapcamunda latest features and roadmap
camunda latest features and roadmap
 
Introducing Workflow Architectures Using Grails - Greach 2015
Introducing Workflow Architectures Using Grails - Greach 2015Introducing Workflow Architectures Using Grails - Greach 2015
Introducing Workflow Architectures Using Grails - Greach 2015
 
JBoss Middleware 및 Remoting 프로젝트 소개
JBoss Middleware 및 Remoting 프로젝트 소개JBoss Middleware 및 Remoting 프로젝트 소개
JBoss Middleware 및 Remoting 프로젝트 소개
 
Developer-Friendly BPM
Developer-Friendly BPMDeveloper-Friendly BPM
Developer-Friendly BPM
 

Similar to Camunda BPM 7.2: Performance and Scalability (English)

NZSPC 2013 - Ultimate SharePoint Infrastructure Best Practices Session
NZSPC 2013 - Ultimate SharePoint Infrastructure Best Practices SessionNZSPC 2013 - Ultimate SharePoint Infrastructure Best Practices Session
NZSPC 2013 - Ultimate SharePoint Infrastructure Best Practices Session
Michael Noel
 
20061122 JBoss-World Experiences with JBoss jBPM
20061122 JBoss-World Experiences with JBoss jBPM20061122 JBoss-World Experiences with JBoss jBPM
20061122 JBoss-World Experiences with JBoss jBPM
camunda services GmbH
 
Data Con LA 2019 - MetaConfig driven FeatureStore with Feature compute & Serv...
Data Con LA 2019 - MetaConfig driven FeatureStore with Feature compute & Serv...Data Con LA 2019 - MetaConfig driven FeatureStore with Feature compute & Serv...
Data Con LA 2019 - MetaConfig driven FeatureStore with Feature compute & Serv...
Data Con LA
 

Similar to Camunda BPM 7.2: Performance and Scalability (English) (20)

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
 
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedInHadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
 
L21 scalability
L21 scalabilityL21 scalability
L21 scalability
 
Handling Data in Mega Scale Systems
Handling Data in Mega Scale SystemsHandling Data in Mega Scale Systems
Handling Data in Mega Scale Systems
 
NZSPC 2013 - Ultimate SharePoint Infrastructure Best Practices Session
NZSPC 2013 - Ultimate SharePoint Infrastructure Best Practices SessionNZSPC 2013 - Ultimate SharePoint Infrastructure Best Practices Session
NZSPC 2013 - Ultimate SharePoint Infrastructure Best Practices Session
 
AWS Webinar 23 - Getting Started with AWS - Understanding total cost of owner...
AWS Webinar 23 - Getting Started with AWS - Understanding total cost of owner...AWS Webinar 23 - Getting Started with AWS - Understanding total cost of owner...
AWS Webinar 23 - Getting Started with AWS - Understanding total cost of owner...
 
J2EE Batch Processing
J2EE Batch ProcessingJ2EE Batch Processing
J2EE Batch Processing
 
Online Datastage training
Online Datastage trainingOnline Datastage training
Online Datastage training
 
Ops Jumpstart: MongoDB Administration 101
Ops Jumpstart: MongoDB Administration 101Ops Jumpstart: MongoDB Administration 101
Ops Jumpstart: MongoDB Administration 101
 
20061122 JBoss-World Experiences with JBoss jBPM
20061122 JBoss-World Experiences with JBoss jBPM20061122 JBoss-World Experiences with JBoss jBPM
20061122 JBoss-World Experiences with JBoss jBPM
 
MetaConfig driven FeatureStore : MakeMyTrip | Presented at Data Con LA 2019 b...
MetaConfig driven FeatureStore : MakeMyTrip | Presented at Data Con LA 2019 b...MetaConfig driven FeatureStore : MakeMyTrip | Presented at Data Con LA 2019 b...
MetaConfig driven FeatureStore : MakeMyTrip | Presented at Data Con LA 2019 b...
 
Data Con LA 2019 - MetaConfig driven FeatureStore with Feature compute & Serv...
Data Con LA 2019 - MetaConfig driven FeatureStore with Feature compute & Serv...Data Con LA 2019 - MetaConfig driven FeatureStore with Feature compute & Serv...
Data Con LA 2019 - MetaConfig driven FeatureStore with Feature compute & Serv...
 
Azure and cloud design patterns
Azure and cloud design patternsAzure and cloud design patterns
Azure and cloud design patterns
 
Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Caching for Microservices Architectures: Session II - Caching Patterns
Caching for Microservices Architectures: Session II - Caching PatternsCaching for Microservices Architectures: Session II - Caching Patterns
Caching for Microservices Architectures: Session II - Caching Patterns
 
JMP401: Masterclass: XPages Scalability
JMP401: Masterclass: XPages ScalabilityJMP401: Masterclass: XPages Scalability
JMP401: Masterclass: XPages Scalability
 
Continuent Tungsten - Scalable Saa S Data Management
Continuent Tungsten - Scalable Saa S Data ManagementContinuent Tungsten - Scalable Saa S Data Management
Continuent Tungsten - Scalable Saa S Data Management
 
RavenDB overview
RavenDB overviewRavenDB overview
RavenDB overview
 
Giga Spaces Data Grid / Data Caching Overview
Giga Spaces Data Grid / Data Caching OverviewGiga Spaces Data Grid / Data Caching Overview
Giga Spaces Data Grid / Data Caching Overview
 

More from camunda services GmbH

More from camunda services GmbH (20)

Using Camunda on Kubernetes through Operators
Using Camunda on Kubernetes through OperatorsUsing Camunda on Kubernetes through Operators
Using Camunda on Kubernetes through Operators
 
Predictive Process Monitoring in Camunda
Predictive Process Monitoring in CamundaPredictive Process Monitoring in Camunda
Predictive Process Monitoring in Camunda
 
Camunda Product Update – The present and the future of Process Automation
Camunda Product Update – The present and the future of Process AutomationCamunda Product Update – The present and the future of Process Automation
Camunda Product Update – The present and the future of Process Automation
 
Tips on how to build Camunda Run for production
Tips on how to build Camunda Run for productionTips on how to build Camunda Run for production
Tips on how to build Camunda Run for production
 
Process Driven Customer Interaction
Process Driven Customer InteractionProcess Driven Customer Interaction
Process Driven Customer Interaction
 
Exploring Automation in Government
Exploring Automation in GovernmentExploring Automation in Government
Exploring Automation in Government
 
The Pulse of Process Automation
The Pulse of Process AutomationThe Pulse of Process Automation
The Pulse of Process Automation
 
Blitzumfrage zur aktuellen Nutzung von Prozessautomatisierung in Unternehmen
Blitzumfrage zur aktuellen Nutzung von Prozessautomatisierung in UnternehmenBlitzumfrage zur aktuellen Nutzung von Prozessautomatisierung in Unternehmen
Blitzumfrage zur aktuellen Nutzung von Prozessautomatisierung in Unternehmen
 
Webinar - A Developer's Quick Start Guide to Open Source Process Automation U...
Webinar - A Developer's Quick Start Guide to Open Source Process Automation U...Webinar - A Developer's Quick Start Guide to Open Source Process Automation U...
Webinar - A Developer's Quick Start Guide to Open Source Process Automation U...
 
Extending human workflow preparing people and processes for the digital era w...
Extending human workflow preparing people and processes for the digital era w...Extending human workflow preparing people and processes for the digital era w...
Extending human workflow preparing people and processes for the digital era w...
 
[Webinar] Camunda Optimize Release 3.0
[Webinar] Camunda Optimize Release 3.0[Webinar] Camunda Optimize Release 3.0
[Webinar] Camunda Optimize Release 3.0
 
Webinar: Monitoring & Orchestrating Your Microservices Landscape using Workfl...
Webinar: Monitoring & Orchestrating Your Microservices Landscape using Workfl...Webinar: Monitoring & Orchestrating Your Microservices Landscape using Workfl...
Webinar: Monitoring & Orchestrating Your Microservices Landscape using Workfl...
 
Process Automation Forum, Processautomatisierung neu gedacht für das digitale...
Process Automation Forum, Processautomatisierung neu gedacht für das digitale...Process Automation Forum, Processautomatisierung neu gedacht für das digitale...
Process Automation Forum, Processautomatisierung neu gedacht für das digitale...
 
Process Automation Forum Zurich, finnova AG Bankware
Process Automation Forum Zurich, finnova AG BankwareProcess Automation Forum Zurich, finnova AG Bankware
Process Automation Forum Zurich, finnova AG Bankware
 
Process Automation Forum Munich, Swiss Life
Process Automation Forum Munich, Swiss LifeProcess Automation Forum Munich, Swiss Life
Process Automation Forum Munich, Swiss Life
 
Process Automation Forum Vienna, A1 & J-IT
Process Automation Forum Vienna, A1 & J-ITProcess Automation Forum Vienna, A1 & J-IT
Process Automation Forum Vienna, A1 & J-IT
 
Process Automation Forum Vienna, Raiffeisen
Process Automation Forum Vienna, RaiffeisenProcess Automation Forum Vienna, Raiffeisen
Process Automation Forum Vienna, Raiffeisen
 
Process Automation Forum Düsseldorf, Provinzial Rheinland Versicherung AG
Process Automation Forum Düsseldorf, Provinzial Rheinland Versicherung AGProcess Automation Forum Düsseldorf, Provinzial Rheinland Versicherung AG
Process Automation Forum Düsseldorf, Provinzial Rheinland Versicherung AG
 
[Webinar] BPM Renaissance: 5 Tips to Thrive in a Cloud-Native World
[Webinar] BPM Renaissance: 5 Tips to Thrive in a Cloud-Native World[Webinar] BPM Renaissance: 5 Tips to Thrive in a Cloud-Native World
[Webinar] BPM Renaissance: 5 Tips to Thrive in a Cloud-Native World
 
[Webinar] Announcing the Camunda Cloud Public Beta - February 2020
[Webinar] Announcing the Camunda Cloud Public Beta - February 2020[Webinar] Announcing the Camunda Cloud Public Beta - February 2020
[Webinar] Announcing the Camunda Cloud Public Beta - February 2020
 

Recently uploaded

Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Marc Lester
 

Recently uploaded (20)

how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
The Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationThe Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test Automation
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
 
Reinforcement Learning – a Rewards Based Approach to Machine Learning - Marko...
Reinforcement Learning – a Rewards Based Approach to Machine Learning - Marko...Reinforcement Learning – a Rewards Based Approach to Machine Learning - Marko...
Reinforcement Learning – a Rewards Based Approach to Machine Learning - Marko...
 
5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand
 
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdfThe Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024
 
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
 
AI Hackathon.pptx
AI                        Hackathon.pptxAI                        Hackathon.pptx
AI Hackathon.pptx
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 

Camunda BPM 7.2: Performance and Scalability (English)

  • 1. Hands-on Webinar Camunda BPM 7.2 Performance and Scalability
  • 2. Daniel Meyer  Process Engine Expert  Technical Project Lead @meyerdan | daniel.meyer@camunda.com Bernd Rücker  10+ years experience with workflow and Java  Co-Founder of Camunda  Evangelist & Head of Consulting @berndruecker | bernd.ruecker@camunda.com Your speakers today
  • 3. Performance is a difficult topic
  • 4.  It always depends −On hardware −On software environment (OS, Java, App Server, Database, …) −On Service Tasks in the process −On network topology (e.g. remote database, web services, …) −On concurrent requests, database load, …  There is no simple answer to performance  But we always succeed – in each and every real-life situation −Handling millions of process instances / day −Handling more than 1.000 process instances / second −Handling thousands of parallel users Performance is a difficult topic
  • 5. We are much faster than competition see http://camunda.com/landing/whitepaper-camunda-jbpm/ In our tests, Camunda‘s throughput was 10x – 30x higher than with JBoss jBPM.
  • 6. 1. Understand basic engine architecture 2. Understand influence parameters on performance 3. Discuss performance improvement approaches 4. See example figures / measurements 5. Discuss future scenarios (e.g. sharding, NoSQL, …) What we do today
  • 7. Basic Engine Architecture We use Optimistic Locking
  • 10. Learning #1: The architecture it damn simple – and the bottleneck is not the process engine!
  • 11. Biggest influence on Performance Database Delegation Code Call Service
  • 13. Learning #2: All state is in the database so clustering gets really easy. camunda scales! More on this later…
  • 14. „But what can I do if performance IS a problem?“
  • 15. 1. Tasklist 2. (History) Queries 3. Job Execution Typical Areas of performance issues
  • 16.  Process/Task Variables −Show in list −Use in Search/Filter  Support for Pagination  Big number of users accessing the tasklist very often Implementation challenge  Provide a generic database schema  Complex data types are serialized – no SQL-JOIN possible  Variables are stored in one row per variable – multiple SQL-JOINs might be required  Some customers use 10-30 variables Tasklist Requirements
  • 17.  Add Process Variables optimized (and only used) for Queries −Extract attributes −Combine variables to work with LIKE  Use own queries −Native – if you want to improve the WHERE −Custom – if you want to SELECT multiple information at once  Own TaskInfo or ProcessInstanceInfo entities −Persisted as MyBatis or JPA entities −Combine all attributes – allow to query tasks without (or with one) JOIN only −Synchronisation via Listener – or use ProcessInstanceInfo as single source Solution Approaches: Tasklist
  • 18. Example Customer - customerId - company - … Your DB camunda PROCESS_VARIABLES customerId ... searchField 4711 ... 4711#camunda#Berlin#... 1 2 Native Query: 3 Custom Query: 4 Java API – results are camunda „Task“ entities Own MyBatis mapping – result can be anything. Called via custom code.
  • 19. Example TaskInfo - taskId - customerId - companyName - contractId - productName - … Your DB camunda PROCESS_VARIABLES customerId contractId productId 4711 0815 42 5 TaskInfo Entity (or ProcessInstanceInfo)
  • 20.  The challenge: −Indexes cost space and performance in writing data −We provide a generic database schema without knowing what you exactly do with it −We constantly work on the right balance of too many and too less indexes  What you can do: −Check indexes and slow query log −Add index where appropriate for your situation (perfectly OK with us, you do not loose support!) −As Enterprise Customer you can always discuss/validate changes with support  Example: create index PROC_DEF_ID_END_TIME ON ACT_HI_PROCINST (PROC_DEF_ID_,END_TIME_) (History) Queries
  • 21. You can also customize history Custom History (e.g. ElasticSearch) Different History Levels: - NONE - ACTIVITY - AUDIT - FULL - CUSTOM (own Filter written in Java, e.g. „only variable X“, „not process Y“, …) Example for custom log level: https://github.com/camunda/camunda- bpm-examples/tree/master/process- engine-plugin/custom-history-level
  • 23.  Asynchronous Continuation involve Jobs  Jobs are stored in the database  Job Executor can be configured −Number of Worker Threads −Number of Jobs fetched with one database query −Size of in-memory Queue −Lock Time, Retry Behavior, …  Job Execution can be distributed over a Cluster  Optimizing is not a straight forward task, hard to give general advise  If you need to improve: Measure and benchmark configurations in your environment! Job Execution
  • 24. The good news: We did big performance improvements in Camunda BPM 7.2!  Improved First Level Cache (throughput increased by up to 90% if async Service Tasks are executed in a row)  Improved locking to have less Optimistic Lock Exceptions and more Jobs acquired per Acquisition. Results in bigger Clusters getting possible. Job Execution in Camunda BPM 7.2
  • 25. Recap:  Added log level “CUSTOM” for History  First Level Cache  Job Executor Acquisition Locking Plus:  Added flush ordering (comparable to Hibernate) to minimize risk of deadlocks Summary: Performance Improvements in 7.2
  • 26. Learning #3: All performance challenges can be solved.
  • 28. Recommendation: Measure! No guessing. camunda engine Process Application External Load Generator e.g. JMeter, HP Load Runner, CURL, … REST „close to production“ environment
  • 29. - Measure - JobExecutor Horizontal Scalability - Impact of 1st level cache reuse - Improvements Version 7.1.0 vs. Version 7.2.0 - Environment: Amazon AWS Cloud (EC2 & RDS) Benchmark
  • 30. Benchmark Setup Client Process Engine Node 1 Process Engine Node 2 Process Engine Node 3 Process Engine Node 4 Start Process Instance (Rest API) Database (Postgres) https://github.com/meyerdan/ec2-benchmark EC2 m3.xlarge (Intel Xeon E5-2670 v2, 4 core, 15 GiB Memory) EC2 m3.xlarge (Intel Xeon E5-2670 v2, 4 core, 15 GiB Memory) EC2 db.m3.xlarge (Intel Xeon E5-2670 v2, 4 core, 15 GiB Memory) Provisioned using Docker
  • 31. EC2
  • 32. Benchmark Setup - The process - All service tasks „Async“ - 1st service task creates 5 variables - Variables are read by subsequent service tasks
  • 33.  Throughput in terms of transactions / second  No absolute Numbers  Benchmark Results
  • 37. Benchmarks Results Cache Off Cache On Amazon RDS Metrics
  • 38. Benchmarks Results Cache Off Cache On Amazon RDS Metrics
  • 40. What is Horizontal Scalability? Scale up the number of transactions executed by adding more processing nodes to the system. [*] [*] http://en.wikipedia.org/wiki/Scalability#Horizontal_and_vertical_scaling (Adapted) Horizontal Scalability transactions / sec nodes
  • 41. The current Situation Scale number of Process Engine Nodes (JVMs) Up to a certain point Limited possibilities for scaling the shared relational Database. In a sense this can only be scaled “up”, not “out”. Shared Relational Database Process Engine Process Engine Process Engine
  • 42. Which way to go? Distributed Datastore Process Engine Process Engine Process Engine Distributed Datastore. Use a database which is itself a distributed system and can be scaled horizontally. - Apache Cassandra, - Apache HBase, - Distributed Caches (Hazelcast, …) - ... Sharding and partitioning. Distribute the state over multiple Datastores. - Multiple instances of PostgreSQL - Each “DB” is a Mongo DB shard - No “DB” at all: use a filesystem journal? - ... Key Difference: on the right hand side, the process engine itself is “distributed” in the sense that it is aware of the distribution and sharding.
  • 43. The problem with Distributed Datastores (In the context of process engines) 1. Consistency guarantees offered by these databases (eventual consistency, ACID vs. BASE, ...) often do not match the requirements of BPMN process execution. See: conflicting concurrent transactions: a. Racing incoming signals (E.g.: Two Messages targeting the same event instance arrive at the same time) b. Joins & Synchronization (E.g.: Gateways, Multi Instance, ...) c. Cancel Activity instance (E.g.: Interrupting Message Boundary Event) 1. Data Representation and Network Latency / Overhead: Process instance state is composite: a. Token state / active activity instances b. Variables c. Task Information, … Challenge is to find a data representation which does not lead to distribution of the state of a single process instance across the cluster while still supporting the required access patterns. 2. Significant differences between individual technologies while there are no industry standards in place yet. (Different with SQL).
  • 44. Sharding => Distributed yet Local Scale horizontally... Each “shard / node” maitains its state locally Partitioning workflow instance state - Each process instance lives inside a single shard / partition => local data consistency easy to guarantee, => easy to access efficiently => Support range of different persistence engines (Relational Database, Non-Relational Databases, …)
  • 45. Proces s Engine Flexible Architecture ... Reality @ zalando 2014 Proces s Engine Proces s Engine The simplest case A single process engine node running on top of a conventional database. A medium Scenario Horizontally scale on top of a conventional database. Massive Compute Cluster 500 Nodes ? All of this should be possible with one unified architecture!
  • 46. No more Search! The catch “Find Process Instance for order with ID 43543242” ?? ???
  • 47. Human Workflow (Build Task Lists) History: Monitoring, Reporting, … Message Correlation When is „Search“ required?
  • 48. Message Correlation The Problem to solve Workflow Instance State for order with ID 435345 Incoming Message: “customer cancelled Order with ID 435345”
  • 49.  Yes, but for non-workflow execution Use Cases Use Search Index? (A)sync Updates Search Index (Near Realtime) Tasklist Queries, Monitoring,...
  • 50. Vision HistoryTasksCore Process Execution Signal / Cancel Activity Instance by Id Correlate Message Query for List of Tasks Monitoring, Reports Real Time, Strongly Consistent Horizontally scalable through sharding Multiple persistence technologies possible Near Real Time, Eventually Consistent Use best technology for the Job. Async Event Stream
  • 51. But still... HistoryTasksCore Process Execution Signal / Cancel Activity Instance by Id Correlate Message Query for List of Tasks Monitoring, Reports In the simplest case!
  • 52. Learning #4: You can do true horizontal clustering with the engine which exists today! There is no need for No-SQL persistence in the core engine.
  • 53. Learning #5: Camunda is really damn smart :-)
  • 54. Camunda BPM Performance is already awesome However: We are continuously improving performance There are strategies to solve specific performance challenges There is no limit in scalability Summary
  • 55. Start now! Open Source Edition • Download: www.camunda.org • Docs, Tutorials etc. • Forum • Meetings Enterprise Edition • Trial: www.camunda.com • Additional Features • Support, Patches etc. • Consulting, Training http://camunda.com/bpm/consultation/ info@camunda.com | US +1.415.800.3908 | DE +49 30 664040 900