Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Designing process flows with #Baker

1,132 views

Published on

Baker is a library that reduces the effort to orchestrate (micro)service-based process flows. Using our intuitive scala/java dsl, developers can declare the orchestration logic in a recipe which is made out of interactions (system calls), ingredients (data) and events. A visual recipe as a graph is also generated so you can easily discuss the logic with other stakeholders, i.e. product owners, testers, etc... The recipe is compiled into a petrinet which is running in an akka actor, and probably in an akka cluster. It is already applied in ING to achieve less spaghetti orchestration code in apis, and more reusability, scalability and resilience.

Published in: Software
  • Login to see the comments

Designing process flows with #Baker

  1. 1. Bekir Oguz Merlijn van Ittersum Designing process flows with #Baker Amsterdam.Scala Meetup September 2017
  2. 2. How to bake a pizza? 2
  3. 3. How to bake a pizza? 3
  4. 4. How to bake a pizza? 4
  5. 5. Why?
  6. 6. It is hard to develop complex orchestration APIs 6
  7. 7. 7
  8. 8. We need a cleaner way to allow design changes in complex APIs 8 Example: initial design of the API A B C D E 1 2 3 4 5
  9. 9. A B C D E 1 2 3 4 5 We need a cleaner way to allow design changes in complex APIs 9 Example: API after one design change
  10. 10. A B C D E 1 2 3 4 5 We need a cleaner way to allow design changes in complex APIs 10 Example: API after ten design changes
  11. 11. We need a cleaner way to allow design changes in complex APIs 11 Example: API after ten design changes A B C D E 1 2 3 4 5
  12. 12. Open Kinder- / Jongerenrekening Open Groei Groter rekening Another Product Verify customer rules Verify customer rules Verify customer rules Register individual Register individual Register Individual Send a present Send a present Not applicable Send a message to the customer Send a message to the customer Send a message to the customer Open a payments account Open a savings account Open a payments account Enable billing Enable billing Assign package There is more need for reuse between teams 12
  13. 13. 13 What?
  14. 14. Baker is a Scala library that simplifies the development of complex orchestration logic. What? Baker! github.com/ing-bank/baker
  15. 15. Intermediate Languagevalidate compile execute Baker runtime
  16. 16. Recipe Event Ingredient Interaction
  17. 17. Defining ingredients 17 case class CustomerInfo(name: String, address: String, email: String) // ingredients val customerInfo = Ingredient[CustomerInfo]("customerInfo") val trackingId = Ingredient[String]("trackingId") val order = Ingredient[String]("order")
  18. 18. Defining events 18 // events val goodsShipped = Event("GoodsShipped", trackingId) val orderPlaced = Event("OrderPlaced", order) val paymentMade = Event("PaymentMade") val valid = Event("Valid") val sorry = Event("Sorry")
  19. 19. Defining interactions 19 // interactions val validateOrder = Interaction( name = "ValidateOrder", inputIngredients = order, output = FiresOneOfEvents(valid, sorry) ) val shipGoods = Interaction( name = "ShipGoods", inputIngredients = Ingredients(goods, customerInfo), output = FiresOneOfEvents(goodsShipped) )
  20. 20. Defining a recipe 20 // recipe val webShopRecipe: Recipe = Recipe("WebShop") .withInteractions( validateOrder, manufactureGoods .withRequiredEvents(valid, paymentMade), shipGoods, sendInvoice .withRequiredEvent(goodsShipped) ) .withSensoryEvents( customerInfoReceived, orderPlaced, paymentMade)
  21. 21. Compile the recipe 21 // compiles the recipe val compiledRecipe = RecipeCompiler.compileRecipe(webShopRecipe) // list of validation errors val errors: Seq[String] = compiledRecipe.validationErrors // .dot (graphviz notation) representation val visualization: String = compiledRecipe.getRecipeVisualization
  22. 22. Visualising the recipe 22
  23. 23. A Baker instance can be created from the valid recipe. It takes care of the execution of the recipe. The only thing left is to provide sensory events to Baker! Baker runtime 23
  24. 24. Initializing Baker instance 24 You can create a new Baker instance by providing the recipe and implementations of the interactions: val compiledRecipe = RecipeCompiler.compileRecipe(webShopRecipe) val validateOrderImpl = validateOrder implement { (order: String) => { // Some logic here valid.instance() // or maybe invalid event to be returned } } val implementations = Seq(validateOrderImpl, manufactureGoodsImpl, sendInvoiceImpl, shipGoodsImpl) val baker = new Baker(compiledRecipe, implementations)
  25. 25. For each client request you do the following: Firing events 25 val processId = UUID.randomUUID().toString baker.bake(processId) baker.handleEvent(processId, orderPlaced.instance(testOrder)) baker.handleEvent(processId, paymentMade.instance()) baker.handleEvent(processId, customerInfoReceived.instance(testCustomerInfoData)) val actualIngredients: Map[String, Any] = baker.getIngredients(processId) val actualEvents: Seq[RuntimeEvent] = baker.events(processId)
  26. 26. 26
  27. 27. 27
  28. 28. 28
  29. 29. 29
  30. 30. 30
  31. 31. 31
  32. 32. 32
  33. 33. 33
  34. 34. 34
  35. 35. 35
  36. 36. 36
  37. 37. 37
  38. 38. 38
  39. 39. 39
  40. 40. 40
  41. 41. 41
  42. 42. 1. It is hard to develop complex orchestration APIs 2. We need a cleaner way to allow design changes in complex APIs 3. There is more need for reuse between teams Does Baker help with the Why? 42
  43. 43. 1. It is hard to develop complex orchestration APIs With Baker, you think about what you want, not about the order. Baker can validate the recipe for you. Clear insight in the current model by visualising the recipe. 2. We need a cleaner way to allow design changes in complex APIs 3. There is more need for reuse between teams Does Baker help with the Why? 43
  44. 44. 1. It is hard to develop complex orchestration APIs With Baker, you think about what you want, not about the order. Baker can validate the recipe for you. Clear insight in the current model by visualising the recipe. 2. We need a cleaner way to allow design changes in complex APIs Baker links the interactions when compiling the recipe. If interactions are added, removed or changed, Baker will update the model accordingly. 3. There is more need for reuse between teams Does Baker help with the Why? 44
  45. 45. 1. It is hard to develop complex orchestration APIs With Baker, you think about what you want, not about the order. Baker can validate the recipe for you. Clear insight in the current model by visualising the recipe. 2. We need a cleaner way to allow design changes in complex APIs Baker links the interactions when compiling the recipe. If interactions are added, removed or changed, Baker will update the model accordingly. 3. There is more need for reuse between teams The possibility to share interactions, ingredients and events between recipes allows for much more reuse of code among teams. Does Baker help with the Why? 45
  46. 46. How?
  47. 47. Baker is powered by 47 PetriNet
  48. 48. Process logic is isolated in an Akka actor, this gives us the following benefits: • Event sourcing through persistent actors • Supporting many data stores, i.e. cassandra • Distributing actors with Cluster Sharding Akka 48
  49. 49. On creation of the Baker instance, the recipe is compiled into a Petri net. A Petri net is a mathematical model that can be used for reasoning about concurrency. This Petri net is then used to determine when & what interactions to execute. PetriNet 49
  50. 50. Execution Semantics: Token passing 50 t1 t2 t3
  51. 51. Execution Semantics: Token passing 51 t1 t2 t3
  52. 52. Execution Semantics: Token passing 52 t1 t2 t3
  53. 53. Execution Semantics: Token passing 53 t1 t2 t3
  54. 54. Execution Semantics: Concurrency 54 t1 t3 t4 t5 t2
  55. 55. Execution Semantics: Concurrency 55 t1 t3 t4 t5 t2 Choice
  56. 56. Execution Semantics: Concurrency 56 t1 t3 t4 t5 t2 Parallel split
  57. 57. Execution Semantics: Concurrency 57 t1 t3 t4 t5 t2 Parallel paths
  58. 58. Execution Semantics: Concurrency 58 t1 t3 t4 t5 t2 Parallel paths
  59. 59. Execution Semantics: Concurrency 59 t1 t3 t4 t5 t2 synchronization
  60. 60. • Simple, uses only a few concepts and components • Exact mathematical definition of execution semantics • Intuitive graphical notation • Well-developed mathematical theory for process analysis Advantages 60
  61. 61. Reachability 61 ? A B
  62. 62. Unreachable code 62 A B
  63. 63. Detect cycles 63
  64. 64. Path traversal time 64 Average traversal time: 10 seconds A B
  65. 65. Identifying bottlenecks 65 Takes 70% of the time A B
  66. 66. Path prediction 66 83% 17% A B
  67. 67. 67 Features Recipe Runtime
  68. 68. 68 Features Recipe Developer friendly Java/Scala dsl Runtime
  69. 69. 69 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Runtime
  70. 70. 70 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime
  71. 71. 71 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model
  72. 72. 72 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model Persisted state through event sourcing
  73. 73. 73 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model Persisted state through event sourcing Data encryption
  74. 74. 74 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model Persisted state through event sourcing Data encryption Automatic retries on technical failures
  75. 75. 75 Features Recipe Developer friendly Java/Scala dsl Compile-time recipe validation Type safety Runtime Execution of the recipe, powered by akka and petrinet model Persisted state through event sourcing Data encryption Automatic retries on technical failures Parallel execution of paths
  76. 76. Property Based Testing 76
  77. 77. Property Based Testing 77 val recipeGen: Gen[Recipe] = for { name <- nameGen sensoryEvents <- Gen.listOf(eventGen) suchThat (_.nonEmpty) interactions <- interactionsGen(sensoryEvents) suchThat (_.nonEmpty) } yield Recipe(name) //turn the lists into var args .withSensoryEvents(sensoryEvents: _*) .withInteractions(interactions.toList: _*)
  78. 78. Property Based Testing 78 forAll(recipeGen) { recipe => val validations = ValidationSettings( allowCycles = false, allowNonExecutableInteractions = false) val compiledRecipe = RecipeCompiler.compileRecipe(recipe, validations) if (compiledRecipe.validationErrors.nonEmpty) { logRecipeStats(recipe) logCompiledRecipeStats(compiledRecipe) } compiledRecipe.validationErrors.isEmpty }
  79. 79. Questions?

×