This document discusses patterns in distributed computing. It outlines some of the key hurdles in distributed systems like asynchrony, locality, failure, and Byzantine faults. It then summarizes common distributed algorithms like leader election, group membership, and consensus. It presents two Ruby libraries, TokenWorker and StaticQueueWorker, that implement leader election and work distribution patterns. It also briefly outlines the Paxos consensus algorithm but notes that a full implementation is very challenging. The document advocates for practical approaches over purely formal solutions and emphasizes starting with reasonable reliability requirements.

1. Patterns in Distributed
Computing
Mike Perham
http://mikeperham.com

2. Me?
DataFabric

3. seti@home
distributed.net
folding@home

4. They have nothing
to do with us.

5. Say Hello to
Production
App1 App2 App3
DB

6. You have a Recurring Task

7. Cron? Which machine?

8. Add a dedicated slice?
App1 App2 App3
Jobs
DB

9. Problem:
Scaling past One Machine
App1 App2 App3
DB

10. Assumption:
Our solution is a redundant
set of processes

11. Let’s get Formal

12. Difficulties
Asynchrony
Locality
Failure
Byzantine?

13. Hurdle:
Group Membership
App1 App2 App3

14. Group Membership
Truly Anonymous is impossible
Algorithm scalability?
10 nodes = 100 messages = 20kb
1000 nodes = 1,000,000 messages = 200mb!
Can run often!

15. Now that everyone is in the
same room...

16. Hurdle:
Consensus
the process of agreeing on one
result among a group of participants.
(wikipedia)

17. Leader Election

18. Leader Election

19. Leader Election
Breaking symmetry
Performance vs Reliability

20. Consensus
Once we have consensus, everything else
follows
But at what cost?

21. Cold Reality:
This stuff is
unpredictable, hard to
test, full of nasty edge
cases.

22. So what do we do?

23. Get Real!
Start making Trade-offs.

24. What are your actual
reliability requirements?

25. Reliability Scale

26. Formally
Correct
Code
Reliability Scale

27. Formally
My
Correct
Code
Code
Reliability Scale

28. Formally
My
Correct
Code
Code
(and probably
yours too) Reliability Scale

29. Formally
My
Correct
Code Memcached Code
(and probably
yours too) Reliability Scale

30. Single Point of Failure?
Oh noes!

31. Politics
http://github.com/mperham/politics
a Ruby library providing utilities and
algorithms for solving common distributed
computing problems.

32. Problem
Database frequently scanned for events of
interest, don’t want to create event dupes

33. TokenWorker
Leader Election via Memcached

34. TokenWorker Notes
M processes, 1 becomes leader for the given
time period and performs the work
Fault tolerant, not scalable
As reliable as memcached
The trick is the memcached::add API

35. 4 module Aggregator
5 class Engine
6 include Politics::TokenWorker
7
8 def initialize
9 register_worker 'dash-aggregator', :iteration_length =>
60.seconds, :servers => memcached_servers
10 end
11
12 def start
13 process do
14 MetricAggregator.new.aggregate
15 end
16 end

36. See TokenWorker ScreenCast

37. Problem
Have a known space to scan regularly, divisible
into N parts
Have 12 databases which need to be scanned
for events every two minutes

38. StaticQueueWorker
Peer-to-Peer Work Coordination

39. StaticQueueWorker Notes
Work is divided into N buckets
M-1 processes can be working on those N
buckets concurrently
Scalable and fault tolerant
Peers discover via Bonjour and communicate via
DRb

40. 16 module Politics
17 class QueueWorkerExample
18 include Politics::StaticQueueWorker
19 TOTAL_BUCKETS = 20
20
21 def initialize
22 register_worker ‘queue-example', TOTAL_BUCKETS, :iteration_length =>
60, :servers => memcached_servers
23 end
24
25 def start
26 process_bucket do |bucket|
27 puts "PID #{$$} processing bucket #{bucket}/#{TOTAL_BUCKETS} at
#{Time.now}..."
28 sleep 1.5
29 end
30 end

41. See StaticQueueWorker ScreenCast

42. What if you need the good
stuff, with no SPOF?

43. Paxos
a family of protocols for solving consensus
in a network of unreliable processors.
(wikipedia)

44. Paxos
The Holy Grail of Distributed Algorithms
Google has spent tens of man-years on their C+
+ implementation.
“Paxos Made Live”

45. Paxos
Phase 1:
C
Prepare
A B

46. Paxos
Phase 1:
C
Prepare
24
A B
24

47. Paxos
Phase 1:
C
Prepare
24
“Ok”
“Ok”
A B
24

48. Paxos
Phase 2:
C
Accept
24/A
A B
24/A

49. Paxos
Phase 2:
C
Accept
24/A
“Ok”
“Ok”
A B
24/A

50. Paxos Notes
Happy path is easy.
Recovering from failures is hard.
Slow, because we need consensus.
Started implementing, quickly stopped.

51. The Future
Plan to continue work on Paxos, if there is
interest
Other ideas for utilities around distributed
computing themes, contact me

52. Summary
Formal
Group Membership and Consensus
Informal
TokenWorker and StaticQueueWorker
Paxos

53. EOF

54. github.com/mperham/politics
mikeperham.com
mperham@gmail.com