Copper: A high performance workflow engine

5,312 views

Published on

COPPER (COmmon Persistable Process Excecution Runtime) is an open-source high performance workflow engine, that persists the workflow instances (process) state into a database. So there is no limit to the runtime of a process. It can run for weeks, month or years. In addition, this strategy leads to crash safety.

A workflow can describe business processes for example, however any kind of use case is supported. The "modelling" language is Java, that has several advantages:

* with COPPER any Java developer is able to design workflows
* all Java developers like to use Java
* many Java libs can be integrated within COPPER
* many Java tools, like IDEs, can be used
* with COPPER your productivity will be increased when using a workflow engine
* using Java solutions will protect your investment
* COPPER is OpenSource under Apache Licence 2.0

Please visit copper-engine.org for details.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,312
On SlideShare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
63
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Copper: A high performance workflow engine

  1. 1. Common Persistable Process Execution Runtime Native JVM Workflow Engine http://www.copper-engine.org/
  2. 2. Short Profile High performance, lightweight workflow engine for Java Outstanding:  Java is the workflow description language! OpenSource Apache License Running in any container, e.g. Spring, JEE, ... Support for various RDBMS, currently  Oracle  MySQL  PostgreSQL  Apache DerbyDB
  3. 3. Why use Java for Workflow Design? Source: www.bpm-guide.de/bpmn
  4. 4. Why use Java for Workflow Design? Problems of graphical Process Modeling  Simple issues become more simple, complex issues more complex  The business process gets obscured as execution details slip in  The development process gets cumbersome  Too opaque for users, too unwieldy for developers
  5. 5. Why use Java for Workflow Design? Use the widely known Java language Utilize the complete range of Java features Use your favourite development environment Use all those highly elaborated Java tools for  editing workflows  workflow compilation, debugging and profiling  teamwork support Avoid team setup expenses because of additional languages, notations, tools and runtimes  many skilled Java professionals available
  6. 6. Core Workflow Engine Requirements Readable and reasonable workflow description Usually, workflows orchestrate multiple partner systems Generally, the lifetime of a workflow is long  from seconds, to hours and days, even months Conclusion:    Workflow instances have to survive Java process lifetime (persistence) A workflow engine has to cope with an unlimited number of workflows instances at the same time. Performance optimization with regard to throughput and latency
  7. 7. Why plain Java is not enough Straightforward workflow definition in pure Java public void execute(Process processData) { Contract contract = crmAdapter.getContractData(processData.getCustomerId()); if (contract.isPrepay()) sepAdapter.recharge(processData.getAmount()); else postpayInvoice.subtract(processData.getAmount()); smsAdapter.message(processData.getMSISDN(), "recharging successful"); } This is simple to read, but:  Every workflow instance occupies one Java thread  limited number of parallel workflow instances  A running Java thread cannot be persisted  no long running workflows, no crash safety
  8. 8. Try it asynchronously One Thread occupied per Workflow instance? Why not calling a partner system asynchronously?  public void execute(Process processData) { ResponseReference r = new ResponseReference(); Contract contract = null; synchronized (r) { crmAdapter.sendContractDataRequest(processData.getCustomerId(), r); r.wait(); contract = r.getContractData(); } … } But: r.wait() still blocks the thread...
  9. 9. Don't block the thread So, we try to avoid Object.wait: private String correlationId = null; public void execute(Process processData) { if (correlationId == null) { correlationId = … // create a GUID crmAdapter.sendContractDataRequest(processData.getCustomerId(), correlationId); // somehow register this workflow instance to wait for correlationId // execute is called again, when the response is available return; } else { Contract contract = crmAdapter.getResponse(correlationId); // continue to process the workflow … }} But: This approach is bad for the readability, especially with larger workflows
  10. 10. COPPER approach Substitute Object.wait public void execute(Process processData) { String correlationId = getEngine().createUUID(); crmAdapter.sendContractDataRequest(processData.getCustomerId(), correlationId); this.wait(WaitMode.ALL, 10000, correlationId); Contract contract = this.getAndRemoveResponse(correlationId); // continue to process the workflow … } Interrupt and Resume anywhere (within the workflow) Call stack is persisted and restored  Internally implemented by Bytecode Instrumentation
  11. 11. Some more features Crash recovery Change Management of Workflows  supports Versioning as well as Modification of workflows  hot workflow deployment Management & Monitoring via JMX Distributed Execution on multiple coupled engines enables  Load Balancing  Redundancy  High Availability (requires a high available DBMS, e.g. Oracle RAC) Fast and generic Audit Trail
  12. 12. COPPER Architecture COPPER runtime Workflow Definitions Database Workflow instances Queue Filesystem Overview over the main COPPER components, here for a persistent engine. In a transient engine, workflow istances and queues reside in the main memory.
  13. 13. COPPER Architecture explained ProcessingEngine    The main entity in the COPPER architecture, responsible for execution of workflow instances. Offers a Java API to launch workflow instances, notification of waiting workflow instances, etc. The engine supports transient or persistent workflows - this depends on the concrete configuration (both provided out-of-thebox) An engine is running in a single JVM process. A JVM process may host several engines.
  14. 14. COPPER Architecture explained Workflow Repository  encapsulates the storage and handling of workflow definitions (i.e. their corresponding Java files) and makes the workflows accessible to one or more COPPER processing engines.  Reads workflow definitions from the file system  Observes the filesystem for modified files --> hot deployment
  15. 15. Execution Animation invoke() wf:Workflow Input Channel id = 4711 data = foo newInstance() wf:Workflow inject dependencies COPPER runtime run(…) id = null data = null InputChannel Processor pool Remote Partner System Queue Workflow Repository Filesystem Correlation Map
  16. 16. Execution Animation Input Channel COPPER runtime InputChannel Processor pool Queue Workflow Repository wf:Workflow Filesystem Correlation Map id = 4711 data = foo dequeue() Remote Partner System
  17. 17. Execution Animation Input Channel COPPER runtime Serialize Java call stack and store it persistently InputChannel Processor pool Remote Partner System Queue Workflow Repository wf:Workflow Filesystem Correlation Map id = = foo data 4711 data = foo cid
  18. 18. Execution Animation Input Channel COPPER runtime Processor Thread is now free to process otherProcessor pool InputChannel workflows Remote Partner System Queue cid Workflow Repository Filesystem wf:Workflow id = 4711 Correlation Map data = foo data = foo
  19. 19. Execution Animation Input Channel COPPER runtime Retrieve persistent Java callstack and resume InputChannel Processor pool Remote Partner System Queue cid Workflow Repository Filesystem wf:Workflow id = 4711 Correlation Map data = foo response data = foo data
  20. 20. Execution Animation Input Channel COPPER runtime Retrieve persistent Java callstack and resume InputChannel Processor pool Queue cid Workflow Repository wf:Workflow Filesystem Correlation Map id = 4711 data = foo response data dequeue() Remote Partner System
  21. 21. Execution Animation Resume here Input Channel COPPER runtime InputChannel Processor pool Remote Partner System removeWorkflow() Queue continue processing Workflow Repository wf:Workflow Filesystem Correlation Map id = 4711 data = foo response data
  22. 22. Execution Animation Input Channel COPPER runtime InputChannel Processor Processing finished pool Queue Workflow Repository Filesystem Correlation Map Remote Partner System
  23. 23. COPPER Architecture explained Processor Pool  A named set of threads executing workflow instances  Configurable name and number of processing threads    Each processor pool owns a queue, containing the workflow instances ready for execution, e.g. after initial enqueue or wakeup a transient engine’s queue resides in memory a persistent engine’s queue resides in the database Supports changing the number of threads dynamically during runtime via JMX COPPER supports multiple processor pools, a workflow instance may change its processor pool at any time
  24. 24. COPPER Architecture explained COPPER runtime Short running tasks pay for the cost induced by long running tasks because Processor pool of thread pool saturation queue long running tasks (e.g. complex database query) short running tasks
  25. 25. COPPER Architecture explained COPPER runtime Processor pool long running tasks default queue
  26. 26. COPPER Architecture explained COPPER runtime Processor pool long running tasks Configurable thread pools help avoiding thread pool saturation for short running tasks default queue
  27. 27. COPPER Architecture explained Database Layer    Encapsulates the access to persistent workflow instances and queues Decoupling of the core COPPER components and the database Enables implementation of custom database layers, e.g. with application specific optimizations or for unsupported DBMS. Audit Trail  Simple and generic Audit Trail implementations  Log data to the database for tracebility and analysis
  28. 28. COPPER Architecture explained Batcher     Enables simple use of database batching/bulking, Collects single database actions (mostly insert, update, delete) and bundles them to a single batch, Usually increases the database throughput by a factor of 10 or more, Widely used by the COPPER database layer, but open for custom use.
  29. 29. COPPER Architecture explained COPPER runtime Queue wf:Workflow id TxnData = 0815 data = bar Correlation Map Database
  30. 30. COPPER Architecture explained COPPER runtime Queue wf:Workflow id TxnData = 0816 data = bar2 Correlation Map TxnData Database
  31. 31. COPPER Architecture explained COPPER runtime Queue wf:Workflow id TxnData = 0817 data = bar3 Correlation Map TxnData TxnData Database
  32. 32. COPPER Architecture explained COPPER runtime Queue Correlation Map TxnData TxnData TxnData JDBC.executeBatch() Database
  33. 33. COPPER Architecture explained COPPER runtime Continue processing workflows after database operations have been committed and results have Queue been sent back to the workflow instances Correlation Map Database
  34. 34. COPPER Open Source (Apache) Available for Java 6 and 7 http://www.copper-engine.org/
  35. 35.  Umfassendes Response-Handling  Early Responses möglich  Multiple Responses möglich (first oder all)  Beliebige CorreleationId
  36. 36.  Performance Zahlen

×