Your SlideShare is downloading. ×
0
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Cliff Moon - Building Polyglot Distributed Systems with Scalang, Boundary Tech Talks October 6, 2011

2,048

Published on

Erlang and Scala have a very complementary set of strengths and weaknesses for building large scale distributed systems. Cliff shows how to build polyglot distributed systems using Scalang, a library …

Erlang and Scala have a very complementary set of strengths and weaknesses for building large scale distributed systems. Cliff shows how to build polyglot distributed systems using Scalang, a library that allows actors written in Scala to perform remote messaging with both Erlang and other Scala nodes. Presented at Boundary's Pizza, Beer, & Tech Talks meetup on October 6, 2011.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,048
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Building Polyglot Systems With Scalang Cliff Moon @moonpolysoftThursday, September 22, 2011
  • 2. Why Scala and Erlang? • Complementary sets of features. • Compatible type systems. • Separates concerns.Thursday, September 22, 2011
  • 3. meter meter meter Collector streaker meter meter meter meter meter meter meter meter Collector scylla meter REST REST REST REST Boundary ArchitectureThursday, September 22, 2011
  • 4. First Revision • JInterface and wrappers. • Cumbersome code. • Wrong level of abstraction. • Not performant.Thursday, September 22, 2011
  • 5. JInterface Wrappers def task(m : OtpErlangObject, mbox : OtpMbox) = m match { case ETuple(EAtom("cluster"), pid : Pid, ref : Ref) =>Thursday, September 22, 2011
  • 6. Scalang • Correctness of behavior. • Performance. • Simplicity.Thursday, September 22, 2011
  • 7. Proc Proc Proc Jetlang Delivery Codecs Netty NIO Sockets Scalang ArchitectureThursday, September 22, 2011
  • 8. Using ScalangThursday, September 22, 2011
  • 9. Node //make sure epmd is running val node = Node(“scala@localhost.local”, “cookie”)Thursday, September 22, 2011
  • 10. Processes class Echo(ctx : ProcessContext) extends Process(ctx) { override def onMessage(msg : Any) = msg match { case (pid : Pid, m : String) => pid ! m } } val echoPid = node.spawn[Echo](“echo_server”)Thursday, September 22, 2011
  • 11. Sending Messages • Send messages to a Pid. • Send messages to a local registered name. • Send messages to a remote registered name.Thursday, September 22, 2011
  • 12. Sending Messages - in process aPid ! ‘do_something ‘regname ! ‘do_another_thing (‘regname, ‘erl@localhost.local) ! ‘do_something_elseThursday, September 22, 2011
  • 13. Sending Messages - outside proc node.send(aPid, ‘do_something) node.send(‘regname, ‘do_another_thing) node.send( (‘regname, ‘erl@localhost.local), ‘do_something_else)Thursday, September 22, 2011
  • 14. Error Handling • Uncaught exceptions in process code. • Implemented via bi-directional links. • Link breakage triggers exit notification. • Links work both intra and inter - node. • Not pre-emptive on the Scalang side.Thursday, September 22, 2011
  • 15. Error Handling class ExitHandler(ctx : ProcessContext) extends Process(ctx) { override def trapExit(from : Pid, msg : Any) { log.warn(“exit notification from %s”, from) } }Thursday, September 22, 2011
  • 16. Type Mappings From Erlang To Scala From Scala To Erlang Small Integer Int Byte Small Integer Integer Int Int Integer Float Double Long Small Bignum Boolean Boolean Double Float Atom Symbol Symbol Atom Reference Reference Reference Reference Port Port Port Port Pid Pid Pid Pid Small Tuple Tuple Fun Fun Large Tuple BigTuple String String String String List List List List BigInteger Large Bignum Binary ByteBuffer Array[Byte] Binary Small Bignum Long ByteBuffer Binary Large Bignum BigInt BitString Bitstring Fun Fun Tuple Tuple Bitstring Bitstring BigTuple TupleThursday, September 22, 2011
  • 17. Rich Type Mappings • Invoked for tagged tuples. • User-supplied decode logic. • Avoids performance penalty of reflective instantiation.Thursday, September 22, 2011
  • 18. Rich Type Mappings (‘struct, List( {struct, [ (‘key, value), {key, Value}, (‘key2, value2), {key2, Value2}, ... )) ... ]}Thursday, September 22, 2011
  • 19. Rich Type Mappings object StructFactory extends TypeFactory { def createType(name : Symbol, arity : Int, reader : TermReader) : Any = { reader.mark (name, arity) match { case (‘struct,2) => Some(readMap(reader)) case _ => reader.reset; None } } }Thursday, September 22, 2011
  • 20. Rich Type Mappings val node = Node(name,cookie,NodeConfig( typeFactory = StructFactory))Thursday, September 22, 2011
  • 21. Services • Initialization parameters. • Built in call and cast support. • Transparent interoperability with gen_server.Thursday, September 22, 2011
  • 22. Services • handleCall - gen_server:call/2 - request & respose • handleCast - gen_server:cast/2 - request • handleInfo - Raw message received.Thursday, September 22, 2011
  • 23. Services case class EchoArgs(name : Symbol) class Echo(ctx : ServiceContext[EchoArgs]) extends Service(ctx) { val EchoArgs(name) = ctx.args override def handleCall(from : (Pid,Reference), req : Any) : Any = { (name, req) } }Thursday, September 22, 2011
  • 24. Anonymous Processes node.spawn { mbox => val msg = mbox.receive // do some long running work node.send(‘master, (results, mbox.self) ) }Thursday, September 22, 2011
  • 25. Runtime Metrics • Message meters per connection. • Execution histograms per process. • Serialization time histograms. • Message queue size gauges. • Internal thread pool metrics.Thursday, September 22, 2011
  • 26. Runtime MetricsThursday, September 22, 2011
  • 27. Performance Tuning • ThreadPoolFactory - pluggable thread pool implementations. • Elastic thread pools from overlock. • Threads tuned to max(8, cpu_count * 2). • Beware of starvation.Thursday, September 22, 2011
  • 28. Thread Pools • Boss pool - Initial connection and accept handling. • Worker pool - Non blocking reads and writes. • Actor pool - Process callbacks. • Batch Executor - Per-process execution logic.Thursday, September 22, 2011
  • 29. Thread Pools val node = Node(name,cookie,NodeConfig( poolFactory = MyThreadPools))Thursday, September 22, 2011
  • 30. Process PatternsThursday, September 22, 2011
  • 31. Deferred Reply def handleCall(from : (Pid, Reference), msg : Any) : Any = { ctx.node.spawn { mbox => // long running work reply(from, response) } ‘noreply }Thursday, September 22, 2011
  • 32. State Machine class Stateful(ctx : ProcessContext) extends Process(ctx) { @volatile var state = ‘a override def onMessage(msg : Any) = (msg, state) match { case (‘event, ‘a) => ... } }Thursday, September 22, 2011
  • 33. Worker Pools class StatelessWorker(ctx : ProcessContext) extends Process(ctx) { def onMessage(msg : Any) = msg match { ... } } //experimental! node.spawn[StatelessWorker](reentrant = true)Thursday, September 22, 2011
  • 34. Supervision Trees class TaskMaster(ctx : ProcessContext) extends Process(ctx) { def onMessage(msg : Any) = { //distribute work to workers } override def handleExit(worker : Pid, reason : Any) { //remove worker from pool, or restart } }Thursday, September 22, 2011
  • 35. Admin Endpoint class Admin(ctx : ServiceContext[Args]) extends Service(ctx) { def handleCall(from: (Pid,Reference), req : Any) = req match { case (‘reassign, node : Symbol) => //reassign message stream to new node } }Thursday, September 22, 2011
  • 36. Admin Endpoint ok = gen_server:call({admin, backend@data01}, {reassign, cruncher@data02}).Thursday, September 22, 2011
  • 37. Demo!Thursday, September 22, 2011
  • 38. Conclusion • Wrap services in Scalang actors. • Throw out your RPC codegen tools. • Embrace a mesh topology. • Let Erlang and Scala do the things they are good at.Thursday, September 22, 2011
  • 39. Questions? https://github.com/boundary/scalang https://github.com/boundary/overlock Thank you.Thursday, September 22, 2011

×