Open XKE - POC d'une architecture distribuée de calculs financiers par Xavier Bucchiotty
•
0 likes•1,343 views
Recommended
More Related Content
What's hot
What's hot (18)
Viewers also liked
Viewers also liked (7)
Similar to Open XKE - POC d'une architecture distribuée de calculs financiers par Xavier Bucchiotty
Similar to Open XKE - POC d'une architecture distribuée de calculs financiers par Xavier Bucchiotty (20)
More from Publicis Sapient Engineering
More from Publicis Sapient Engineering (20)
Recently uploaded
Recently uploaded (20)
Open XKE - POC d'une architecture distribuée de calculs financiers par Xavier Bucchiotty
- 1. Système distribué de calculs financiers Par Xavier Bucchiotty
- 4. Step 1 Step 2 Step 3 Step 4 Stream API Iteratees Akka actor Akka cluster
- 5. Use case Financial debt management
- 6. CAUTION
- 8. initial = 1000 € duration = 5 years fixed interets rate = 5% Date 1000 € Amort Interests Outstanding 2013-01-01 200 € 50 € 800 € 2014-01-01 200 € 40 € 600 € 2015-01-01 200 € 30 € 400 € 2016-01-01 200 € 20 € 200 € 2017-01-01 200 € 10 € 0 €
- 9. initial = 1000 € duration = 5 years fixed interets rate = 5% Date 1000 € Amort Interests Outstanding 2013-01-01 200 € 50 € 800 € 2014-01-01 200 € 40 € 600 € 2015-01-01 200 € 30 € 400 € 2016-01-01 200 € 20 € 200 € 2017-01-01 200 € 10 € 0 € date = last date + (1 year)
- 10. initial = 1000 € duration = 5 years fixed interets rate = 5% Date 1000 € Amort Interests Outstanding 2013-01-01 200 € 50 € 800 € 2014-01-01 200 € 40 € 600 € 2015-01-01 200 € 30 € 400 € 2016-01-01 200 € 20 € 200 € 2017-01-01 200 € 10 € 0 € amort = initial / duration
- 11. initial = 1000 € duration = 5 years fixed interets rate = 5% Date 1000 € Amort Interests Outstanding 2013-01-01 200 € 50 € 800 € 2014-01-01 200 € 40 € 600 € 2015-01-01 200 € 30 € 400 € 2016-01-01 200 € 20 € 200 € 2017-01-01 200 € 10 € 0 € outstanding = last oustanding - amort
- 12. initial = 1000 € duration = 5 years fixed interets rate = 5% Date 1000 € Amort Interests Outstanding 2013-01-01 200 € 50 € 800 € 2014-01-01 200 € 40 € 600 € 2015-01-01 200 € 30 € 400 € 2016-01-01 200 € 20 € 200 € 2017-01-01 200 € 10 € 0 € interests = last outstanding * rate
- 13. val f = (last: Row) => new Row { def date = last.date + (1 year) def amortization = last amortization def outstanding = last.outstanding - amortization def interests = last.outstanding * fixedRate }
- 17. case class Loan( ... ) { def first: Row def f:(Row => Row) def rows = Stream.iterate(first)(f) .take(duration) }
- 18. case class Portfolio(loans: Seq[Loan]) { def rows = loans.stream.flatMap(_.rows) }
- 19. Date Amort Interests Total paid 2013-01-01 200 € 40 € 240 € 2015-01-01 200 € 30 € 230 € 200 € 20 € 220 € 2017-01-01 200 € 10 € 210 € 2013-01-01 200 € 50 € 250 € 2014-01-01 200 € 40 € 240 € 2015-01-01 200 € 30 € 230 € 2016-01-01 200 € 20 € 220 € 2017-01-01 200 € 10 € 210 € 2013-01-01 200 € 50 € 250 € 2014-01-01 Loan 3 250 € 2016-01-01 Loan 2 50 € 2014-01-01 Loan 1 200 € 200 € 40 € 240 € 2015-01-01 200 € 30 € 230 € 2016-01-01 200 € 20 € 220 € 2017-01-01 200 € 10 € 210 € Total 3450 €
- 20. // Produce rows val totalPaid = portfolio.rows // Transform rows to amount .map(row => row.interests + row.amortization) //Consume amount .foldLeft(0 EUR)(_ + _)
- 21. // Produce rows val totalPaid = portfolio.rows // Transform rows to amount .map(row => row.interests + row.amortization) type RowProducer = Iterable[Row] type RowTransformer[T] = (Row=>T) //Consume amount .foldLeft(0 EUR)(_ + _) type AmountConsumer[T] = (Iterable[Amount]=>T)
- 22. RowProducer (Iterable[Row]) //Loan Stream.iterate(first)(f) take duration //Porfolio loans => loans flatMap (loan => loan.rows) + on demand computation - sequential computation
- 23. RowTransformer (Row => T) object RowTransformer { val totalPaid = (row: Row) => row.interests + row.amortization } + function composition - type limited to «map»
- 24. AmountConsumer (Iterable[Amount] => T) object AmountConsumer { def sum = (rows: Iterable[Amount]) => rows.foldLeft(Amount(0, EUR))(_ + _) } + function composition - synchronism
- 25. Step 1 Stream API 5000 loans 50 rows ~ 560 ms
- 26. Pros Cons On demand computation Sequential computation Function composition Synchronism Transformation limited to «map»
- 27. Step 2 Iteratees
- 28. Integrating Play iteratees libraryDependencies ++= Seq( "com.typesafe.play" %% "play-iteratees" % "2.2.0-RC2" )
- 30. Enumerator Iteratees are immutable Input Status Asynchronous by design Type safe Iteratee
- 32. case class Loan(initial: Amount, duration: Int, rowIt: RowIt) { def rows(implicit ctx: ExecutionContext) = Enumerator.enumerate( Stream.iterate(first)(f).take(duration) ) } Data producer
- 33. case class Portfolio(loans: Seq[Loansan]) { def rows(implicit ctx: ExecutionContext) = Enumerator.interleave(loans.map(_.rows)) } producers can be combined
- 35. Iteratee Consumer as a state machine
- 37. object Input { case class El[+E](e: E) case object Empty case object EOF }
- 39. object Step { case class Done[+A, E] (a: A, remaining: Input[E]) case class Cont[E, +A] (k: Input[E] => Iteratee[E, A]) case class Error[E] (msg: String, input: Input[E]) }
- 40. Enumerator Status Continue Iteratee Input El(...) def step = ... val count = 1 computes Iteratee def step = ... val count = 0
- 41. Enumerator Status Done Iteratee Input def step = ... val count = 1 EOF computes Iteratee def step = ... val count = 1
- 42. Enumerator Status Error Input Iteratee El(...) def step = ... val error = "Runtime Error" computes Iteratee def step = ... val count = 1
- 43. val last: RowConsumer[Option[Row]] = { def step(last: Option[Row]): K[Row,Option[Row]]= { case Input.Empty => Cont(step(last)) case Input.EOF => Done(last, Input.EOF) case Input.El(e) => Cont(step(Some(e))) } Cont(step(Option.empty[Row])) }
- 44. object AmountConsumer { val sum: AmountConsumer[Amount] = (rows: Iterable[Amount]) => rows.foldLeft(Amount(0, EUR))(_ + _) }
- 45. object AmountConsumer { val sum: AmountConsumer[Amount] = Iteratee.fold[Amount, Amount] (Amount(0, EUR))(_ + _) }
- 46. import RowTransformer.totalPaid import AmountConsumer.sum val totalPaidComputation: Future[Amount] = portfolio.rows.run(sum)
- 47. import RowTransformer.totalPaid import AmountConsumer.sum val totalPaidComputation: Future[Amount] = portfolio.rows |>>> sum
- 51. object RowTransformer { val totalPaid = Enumeratee.map[Row](row => row.interests + row.amortization ) } Data transformation
- 52. def until(date: DateMidnight) = Enumeratee.filter[Row]( row => !row.date.isAfter(date) ) Data filtering
- 53. type RowProducer = Iterable[Row] type RowTransformer[T] = (Row=>T) type AmountConsumer[T] = (Iterable[Amount]=>T) type RowProducer = Enumerator[Row] type RowTransformer[T] = Enumeratee[Row, T] type AmountConsumer[T] = Iteratee[Amount, T]
- 54. Futures are composable map, flatMap, filter onComplete, onSuccess, onError, recover
- 55. // Produce rows val totalPaidComputation: Future[Amount] = portfolio.rows &> totalPaid |>>> sum // Blocking the thread to wait for the result val totalPaid = Await.result( totalPaidComputation, atMost = defaultTimeout) totalPaid should equal(3480 EUR)
- 56. We still have function composition and prepares the code for asynchronism
- 57. RowProducer //Loan Enumerator.enumerate( Stream.iterate(first)(f).take(duration) ) //Porfolio Enumerator.interleave(loans.map(_.rows)) + on demand computation + parallel computation
- 58. RowTransformer val totalPaid = Enumeratee.map[Row](row => row.interests + row.amortization ) + Function composition + map, filter, ...
- 59. AmountConsumer def sum = Iteratee.fold[Amount, Amount] (Amount(0, EUR))(_ + _) + Function composition + Asynchronism
- 60. Step 1 Step 2 Stream API Iteratees 5000 loans 50 rows 5000 loans 50 rows ~ 560 ms ~ 3500 ms ?
- 61. simple test complex test Thread.sleep((Math.random() * 1000) % 2) toLong)
- 62. Step 1 Step 2 Stream API Iteratees 5000 loans 50 rows 5000 loans 50 rows ~ 560 ms ~ 3500 ms with pause with pause ~ 144900 ms ~ 157285 ms ?
- 63. Cost of using this implementation of iteratees is greater than gain of interleaving for such small operations
- 66. Step 1 Step 2 Stream API Iteratees 5000 loans 50 rows 5000 loans 50 rows ~ 560 ms ~ 4571 ms with pause with pause ~ 144900 ms ~ 39042 ms
- 67. Pros Cons On demand computation Sequential computation Function composition Synchronism Transformation limited to «map»
- 68. Pros On demand computation Function composition Sequential computation Synchronism Cons
- 69. Pros Cons On demand computation No error management Function composition No elasticity Parallel computation No resilience Asynchronism
- 71. Integrating Akka libraryDependencies ++= Seq( "com.typesafe.akka" %% "akka-actor" % "2.2.0" )
- 72. Actors are objects They communicate with each other by messages asynchronously
- 73. class Backend extends Actor { def receive = { case Compute(loan) => sender.tell( msg = loan.stream.toList, sender = self) } } case class Compute(loan: Loan)
- 74. case class Loan def rows(implicit calculator: ActorRef, ctx: ExecutionContext) = { val responseFuture = ask(calculator,Compute(this)) val rowsFuture = responseFuture .mapTo[List[Row]] rowsFuture.map(Enumerator.enumerate(_)) ) } }
- 75. val system = ActorSystem.create("ScalaIOSystem") val calculator = system.actorOf(Props[Backend] .withRouter( RoundRobinRouter(nrOfInstances = 10) ) ,"calculator") }
- 76. Supervision val simpleStrategy = OneForOneStrategy() { case _: AskTimeoutException => Resume case _: RuntimeException => Escalate } system.actorOf(Props[Backend] ... .withSupervisorStrategy(simpleStrategy)), "calculator")
- 79. Actor System Routee 1 Router Routee 2 Routee 3
- 80. RowProducer //Loan ask(calculator,Compute(this)) .mapTo[List[Row]] .map(Enumerator.enumerate(_)) //Porfolio Enumerator.interleave(loans.map(_.rows)) + parallel computation - on demand computation
- 81. RowTransformer val totalPaid = Enumeratee.map[Row](row => row.interests + row.amortization ) + Nothing changed
- 82. AmountConsumer def sum = Iteratee.fold[Amount, Amount] (Amount(0, EUR))(_ + _) + Nothing changed
- 83. Step 1 Step 2 Step 3 Stream API Iteratees Akka actor 5000 loans 50 rows 5000 loans 50 rows 5000 loans 50 rows ~ 560 ms ~ 4571 ms ~ 4271 ms with pause with pause with pause ~ 144900 ms ~ 39042 ms ~ 40882 ms
- 84. Pros Cons On demand computation No error management Function composition No elasticity Parallel computation No resilience Asynchronism
- 85. Pros Cons No error management On demand computation Function composition No elasticity Parallel computation No resilience Asynchronism
- 86. Pros Cons Error management No on demand computation Function composition No elasticity Parallel computation No resilience Asynchronism
- 88. Integrating Akka Cluster libraryDependencies ++= Seq( "com.typesafe.akka" %% "akka-cluster" % "2.2.0" )
- 89. Cluster Router ClusterRouterConfig Can create actors on different nodes of the cluster Role Local actors or not Control number of actors per node per system
- 90. Cluster Router AdaptiveLoadBalancingRouter Collect metrics (CPU, HEAP, LOAD) via JMX or Hyperic Sigar and make load balancing
- 91. val calculator = system.actorOf(Props[Backend] .withRouter( RoundRobinRouter(nrOfInstances = 10) ) ,"calculator") } val calculator = system.actorOf(Props[Backend] .withRouter(ClusterRouterConfig( local = localRouter, settings = clusterSettings) ) , "calculator") }
- 92. Actor System Routee 3 Actor System Routee 1 Routee 4 Elasticity Router Routee 5 Routee 3 Routee 6 Actor System
- 93. application.conf cluster { seed-nodes = [ "akka.tcp://ScalaIOSystem@127.0.0.1:2551", "akka.tcp://ScalaIOSystem@127.0.0.1:2552" ] auto-down = on }
- 94. Actor System Routee 3 Actor System Routee 1 Routee 4 Resilience Router Routee 5 Routee 3 Routee 6 Actor System
- 96. RowTransformer val totalPaid = Enumeratee.map[Row](row => row.interests + row.amortization ) + Nothing changed
- 97. AmountConsumer def sum = Iteratee.fold[Amount, Amount] (Amount(0, EUR))(_ + _) + Nothing changed
- 98. Pros Cons Error management No on demand computation Function composition No elasticity Parallel computation No resilience Asynchronism
- 99. Pros Error management Function composition Parallel computation Asynchronism No elasticity No resilience Cons No on demand computation
- 100. Pros Cons Error management No on demand computation Function composition Network serialization Parallel computation Asynchronism Elasticity Resilience
- 101. Step 1 Step 2 Step 3 Step 4 Stream API Iteratees Akka actor Akka cluster 5000 loans 50 rows 5000 loans 50 rows 5000 loans 50 rows 5000 loans 50 rows ~ 560 ms ~ 4571 ms ~ 4271 ms ~ 6213 ms with pause with pause with pause with pause ~ 144900 ms ~ 39042 ms ~ 40882 ms ~ 77957 ms 1 node / 2 actors
- 102. Step 1 Step 2 Step 3 Step 4 Stream API Iteratees Akka actor Akka cluster 5000 loans 50 rows 5000 loans 50 rows 5000 loans 50 rows 5000 loans 50 rows ~ 560 ms ~ 4571 ms ~ 4271 ms ~ 5547 ms with pause with pause with pause with pause ~ 144900 ms ~ 39042 ms ~ 40882 ms ~ 39695 ms 2 nodes / 4 actors
- 103. Conclusion
- 104. Step 1 Step 2 Step 3 Step 4 Stream API Iteratees Akka actor Akka cluster powerful library elegant API error management elasticity low memory enable asynchronism and parallelism control on parallel execution via configuration resilience performance when single threaded monitoring
- 105. It’s all about trade-off
- 106. But do you really need distribution?
- 107. Hot subject Recet blog post from «Mandubian» for Scalaz stream machines and iteratees [1] Recent presentation from «Heather Miller» for spores (distribuables closures) [2] Recent release of Scala 2.10.3 and performance optimization of Promise Release candidate of play-iteratee module with performance optimization Lots of stuff in the roadmap of Akka cluster 2.3.0
- 108. Hot subject [1] : http://mandubian.com/2013/08/21/playztream/ [2] : https://speakerdeck.com/heathermiller/on-pickles-and-sporesimproving-support-for-distributed-programming-in-scala