Lambda jam 2013

  • 837 views
Uploaded on

Slides from my Lambda Jam 2013 talk: Distributed Programming with Erlang, Riak Core, and Riak Pipe

Slides from my Lambda Jam 2013 talk: Distributed Programming with Erlang, Riak Core, and Riak Pipe

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
837
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. DISTRIBUTED SYSTEMS: ERLANG, RIAK CORE, RIAK PIPE A few clues, a few rants, and a call to action 1Tuesday, July 9, 13
  • 2. ABOUT ME • Technical Evangelist for Basho • @macintux • jdaily@basho.com • http://tinyurl.com/jrdlj13 2Tuesday, July 9, 13
  • 3. ZERG.ERLANGONXEN.COM 3Tuesday, July 9, 13
  • 4. THOU SHALT LOOK AT ERLANG John’s first commandment of infrastructure software 4Tuesday, July 9, 13
  • 5. 5Tuesday, July 9, 13
  • 6. 6Tuesday, July 9, 13
  • 7. The Erlang virtual machine provides fault-tolerance features and a highly scalable architecture while the Erlang programming language keeps the required source code small and easy to follow. It’s important to understand that few programming languages can provide real fault-tolerance with scalability. In fact, I’d say Erlang is roughly alone in this regard. MICHAELTRUOG 7Tuesday, July 9, 13
  • 8. WHY DISTRIBUTED PROGRAMMING IS HARD • Clocks 8Tuesday, July 9, 13
  • 9. WHY DISTRIBUTED PROGRAMMING IS HARD • Clocks • Latency 9Tuesday, July 9, 13
  • 10. WHY DISTRIBUTED PROGRAMMING IS HARD • Clocks • Latency • Lost messages 10Tuesday, July 9, 13
  • 11. WHY DISTRIBUTED PROGRAMMING IS HARD • Clocks • Latency • Lost messages • Servers break 11Tuesday, July 9, 13
  • 12. 12Tuesday, July 9, 13
  • 13. CAP LIGHTNINGTALK 13Tuesday, July 9, 13
  • 14. 14Tuesday, July 9, 13
  • 15. 15Tuesday, July 9, 13
  • 16. 16Tuesday, July 9, 13
  • 17. 17Tuesday, July 9, 13
  • 18. 18Tuesday, July 9, 13
  • 19. /CAP LIGHTNINGTALK /memes 19Tuesday, July 9, 13
  • 20. • Key/value store DYNAMO & RIAK 20Tuesday, July 9, 13
  • 21. • Key/value store • Consistent hashing DYNAMO & RIAK 21Tuesday, July 9, 13
  • 22. • Key/value store • Consistent hashing • Granular compute engines DYNAMO & RIAK 22Tuesday, July 9, 13
  • 23. • Key/value store • Consistent hashing • Granular compute engines • Replication DYNAMO & RIAK 23Tuesday, July 9, 13
  • 24. 24Tuesday, July 9, 13
  • 25. 25Tuesday, July 9, 13
  • 26. The spirit of the cloud is ‘every node can die’, writing the administration software for a cloud expecting the administration nodes to stay alive is hypocritical. Yet just saying ‘hey the API will be down if you lose the controlling node’ kind of sucks so it comes down having a distributed system either with stateless nodes and a distributed database or with distributed nodes and a local database. HEINZ GIES, PROJECT FIFO 26Tuesday, July 9, 13
  • 27. 27Tuesday, July 9, 13
  • 28. 28Tuesday, July 9, 13
  • 29. RIAK PIPE Like UNIX pipes, but not! 29Tuesday, July 9, 13
  • 30. RIAK PIPE COMPONENTS • Pipeline (client, fittings, sink) • Fitting (specification of behavior, partitioner) • Coordinator (one/fitting/cluster) • Worker process (one/fitting/vnode, implements a behavior) 30Tuesday, July 9, 13
  • 31. RIAK PIPE PROCESSING 1. Client submits a pipeline specification 2. Coordinators are spawned from last to first 3. When input arrives at a vnode, if no worker for that fitting exists yet the appropriate coordinator will be asked for the fitting specification 4. The last fitting sends its output to the sink, which is typically the client 31Tuesday, July 9, 13
  • 32. RIAK PIPE DEBUGGING • Log and trace messages are optionally sent to the sink • riak_pipe:status will return a list of fittings; each item is a list of worker details: node ID, state, queue length, idle time, etc • Fittings can define a validator function to be invoked on inputs before processing • Intermediate outputs can be forwarded directly to sink 32Tuesday, July 9, 13
  • 33. THE END IS NIGH • Riak 1.4 • RICON • Meetups 33Tuesday, July 9, 13