Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011

2,317 views

Published on

Erlang and Scala have a very complementary set of strengths and weaknesses for building large scale distributed systems. Cliff shows how to build polyglot distributed systems using Scalang, a library that allows actors written in Scala to perform remote messaging with both Erlang and other Scala nodes. Presented at Boundary's Pizza, Beer, & Tech Talks meetup on October 6, 2011.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,317
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011

  1. 1. Building Polyglot Systems With Scalang Cliff Moon @moonpolysoftThursday, September 22, 2011
  2. 2. Why Scala and Erlang? • Complementary sets of features. • Compatible type systems. • Separates concerns.Thursday, September 22, 2011
  3. 3. meter meter meter Collector streaker meter meter meter meter meter meter meter meter Collector scylla meter REST REST REST REST Boundary ArchitectureThursday, September 22, 2011
  4. 4. First Revision • JInterface and wrappers. • Cumbersome code. • Wrong level of abstraction. • Not performant.Thursday, September 22, 2011
  5. 5. JInterface Wrappers def task(m : OtpErlangObject, mbox : OtpMbox) = m match { case ETuple(EAtom("cluster"), pid : Pid, ref : Ref) =>Thursday, September 22, 2011
  6. 6. Scalang • Correctness of behavior. • Performance. • Simplicity.Thursday, September 22, 2011
  7. 7. Proc Proc Proc Jetlang Delivery Codecs Netty NIO Sockets Scalang ArchitectureThursday, September 22, 2011
  8. 8. Using ScalangThursday, September 22, 2011
  9. 9. Node //make sure epmd is running val node = Node(“scala@localhost.local”, “cookie”)Thursday, September 22, 2011
  10. 10. Processes class Echo(ctx : ProcessContext) extends Process(ctx) { override def onMessage(msg : Any) = msg match { case (pid : Pid, m : String) => pid ! m } } val echoPid = node.spawn[Echo](“echo_server”)Thursday, September 22, 2011
  11. 11. Sending Messages • Send messages to a Pid. • Send messages to a local registered name. • Send messages to a remote registered name.Thursday, September 22, 2011
  12. 12. Sending Messages - in process aPid ! ‘do_something ‘regname ! ‘do_another_thing (‘regname, ‘erl@localhost.local) ! ‘do_something_elseThursday, September 22, 2011
  13. 13. Sending Messages - outside proc node.send(aPid, ‘do_something) node.send(‘regname, ‘do_another_thing) node.send( (‘regname, ‘erl@localhost.local), ‘do_something_else)Thursday, September 22, 2011
  14. 14. Error Handling • Uncaught exceptions in process code. • Implemented via bi-directional links. • Link breakage triggers exit notification. • Links work both intra and inter - node. • Not pre-emptive on the Scalang side.Thursday, September 22, 2011
  15. 15. Error Handling class ExitHandler(ctx : ProcessContext) extends Process(ctx) { override def trapExit(from : Pid, msg : Any) { log.warn(“exit notification from %s”, from) } }Thursday, September 22, 2011
  16. 16. Type Mappings From Erlang To Scala From Scala To Erlang Small Integer Int Byte Small Integer Integer Int Int Integer Float Double Long Small Bignum Boolean Boolean Double Float Atom Symbol Symbol Atom Reference Reference Reference Reference Port Port Port Port Pid Pid Pid Pid Small Tuple Tuple Fun Fun Large Tuple BigTuple String String String String List List List List BigInteger Large Bignum Binary ByteBuffer Array[Byte] Binary Small Bignum Long ByteBuffer Binary Large Bignum BigInt BitString Bitstring Fun Fun Tuple Tuple Bitstring Bitstring BigTuple TupleThursday, September 22, 2011
  17. 17. Rich Type Mappings • Invoked for tagged tuples. • User-supplied decode logic. • Avoids performance penalty of reflective instantiation.Thursday, September 22, 2011
  18. 18. Rich Type Mappings (‘struct, List( {struct, [ (‘key, value), {key, Value}, (‘key2, value2), {key2, Value2}, ... )) ... ]}Thursday, September 22, 2011
  19. 19. Rich Type Mappings object StructFactory extends TypeFactory { def createType(name : Symbol, arity : Int, reader : TermReader) : Any = { reader.mark (name, arity) match { case (‘struct,2) => Some(readMap(reader)) case _ => reader.reset; None } } }Thursday, September 22, 2011
  20. 20. Rich Type Mappings val node = Node(name,cookie,NodeConfig( typeFactory = StructFactory))Thursday, September 22, 2011
  21. 21. Services • Initialization parameters. • Built in call and cast support. • Transparent interoperability with gen_server.Thursday, September 22, 2011
  22. 22. Services • handleCall - gen_server:call/2 - request & respose • handleCast - gen_server:cast/2 - request • handleInfo - Raw message received.Thursday, September 22, 2011
  23. 23. Services case class EchoArgs(name : Symbol) class Echo(ctx : ServiceContext[EchoArgs]) extends Service(ctx) { val EchoArgs(name) = ctx.args override def handleCall(from : (Pid,Reference), req : Any) : Any = { (name, req) } }Thursday, September 22, 2011
  24. 24. Anonymous Processes node.spawn { mbox => val msg = mbox.receive // do some long running work node.send(‘master, (results, mbox.self) ) }Thursday, September 22, 2011
  25. 25. Runtime Metrics • Message meters per connection. • Execution histograms per process. • Serialization time histograms. • Message queue size gauges. • Internal thread pool metrics.Thursday, September 22, 2011
  26. 26. Runtime MetricsThursday, September 22, 2011
  27. 27. Performance Tuning • ThreadPoolFactory - pluggable thread pool implementations. • Elastic thread pools from overlock. • Threads tuned to max(8, cpu_count * 2). • Beware of starvation.Thursday, September 22, 2011
  28. 28. Thread Pools • Boss pool - Initial connection and accept handling. • Worker pool - Non blocking reads and writes. • Actor pool - Process callbacks. • Batch Executor - Per-process execution logic.Thursday, September 22, 2011
  29. 29. Thread Pools val node = Node(name,cookie,NodeConfig( poolFactory = MyThreadPools))Thursday, September 22, 2011
  30. 30. Process PatternsThursday, September 22, 2011
  31. 31. Deferred Reply def handleCall(from : (Pid, Reference), msg : Any) : Any = { ctx.node.spawn { mbox => // long running work reply(from, response) } ‘noreply }Thursday, September 22, 2011
  32. 32. State Machine class Stateful(ctx : ProcessContext) extends Process(ctx) { @volatile var state = ‘a override def onMessage(msg : Any) = (msg, state) match { case (‘event, ‘a) => ... } }Thursday, September 22, 2011
  33. 33. Worker Pools class StatelessWorker(ctx : ProcessContext) extends Process(ctx) { def onMessage(msg : Any) = msg match { ... } } //experimental! node.spawn[StatelessWorker](reentrant = true)Thursday, September 22, 2011
  34. 34. Supervision Trees class TaskMaster(ctx : ProcessContext) extends Process(ctx) { def onMessage(msg : Any) = { //distribute work to workers } override def handleExit(worker : Pid, reason : Any) { //remove worker from pool, or restart } }Thursday, September 22, 2011
  35. 35. Admin Endpoint class Admin(ctx : ServiceContext[Args]) extends Service(ctx) { def handleCall(from: (Pid,Reference), req : Any) = req match { case (‘reassign, node : Symbol) => //reassign message stream to new node } }Thursday, September 22, 2011
  36. 36. Admin Endpoint ok = gen_server:call({admin, backend@data01}, {reassign, cruncher@data02}).Thursday, September 22, 2011
  37. 37. Demo!Thursday, September 22, 2011
  38. 38. Conclusion • Wrap services in Scalang actors. • Throw out your RPC codegen tools. • Embrace a mesh topology. • Let Erlang and Scala do the things they are good at.Thursday, September 22, 2011
  39. 39. Questions? https://github.com/boundary/scalang https://github.com/boundary/overlock Thank you.Thursday, September 22, 2011

×