Supporting the growth of Uber's product, business and systems necessitated a new way to think about its dispatching systems. Availability is taken extremely seriously. Manual operational intervention should be kept to a minimum. And scaling is a continuous exercise and should be possible with little effort. With these tenets in mind, we developed Ringpop. Ringpop is a library that brings cooperation and coordination to distributed applications. In this talk, you'll hear about gossip protocols, consistent hashing and the fun ways we applied Ringpop to distributed systems programming! Join me!

1. EXPLORATIONS IN
COOPERATIVE
DISTRIBUTED
SYSTEMS
WITH UBER’S RINGPOP

2. WHAT IS RINGPOP?
HIGH-LEVEL FACTS
It is a library written in Node.js and Go.
It is...
“...scalable, fault-tolerant application layer sharding.”
"...a library that brings cooperation and coordination to distributed applications.”
“...a hash ring.”
It is open source.

3. BUT WHAT IS UBER?
REALTIME DISPATCH ENGINEERING
Mobile API, match-maker, trip orchestrator
Platform for marketplaces
Highly interactive, real-time
Data locality
Long-running transactions

4. NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
ROUTING LAYER DISPATCH LAYER
LONDONNEWYORK
ORIGINAL DISPATCH ARCHITECTURE
UBER REAL-TIME DISPATCHING
0 1 2 3

5. NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
ROUTING LAYER
ORIGINAL DISPATCH ARCHITECTURE
UBER REAL-TIME DISPATCHING
0 1 2 3
DISPATCH LAYER
LONDONNEWYORK

6. NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
ROUTING LAYER
ORIGINAL DISPATCH ARCHITECTURE
UBER REAL-TIME DISPATCHING
0 1 2 3
DISPATCH LAYER
LONDONNEWYORK

7. NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
ROUTING LAYER DISPATCH LAYER
LONDONNEWYORK
ORIGINAL DISPATCH ARCHITECTURE
UBER REAL-TIME DISPATCHING
0 1 2 3

8. NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NGINX
HAPROXY
NODE.JS
TWEMPROXY
REDIS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
NODE.JS
NODE.JS
REDIS
NODE.JS
NODE.JS
ROUTING LAYER DISPATCH LAYER
LONDONNEWYORK
ORIGINAL DISPATCH ARCHITECTURE
UBER REAL-TIME DISPATCHING
0 1 2 3

9. EVOLUTION OF THINKING AT UBER
UBER REAL-TIME DISPATCHING
Scaling our
organization,
product and
systems
100x scale
Availability
over
consistency Self-healing
No “master” No monolith
Flexibility and
safety

10. ENTER RINGPOP
MEMBERSHIP PROTOCOL, CONSISTENT HASHING,
REQUEST ROUTING

11. INST. A INST. B INST. C
DATABASE DATABASE
DISCRETE APPLICATION INSTANCES
ENTER RINGPOP: MEMBERSHIP PROTOCOL
FE1 FE2 FE3 FE4 FE5
FRONT-END
APPLICATION
STORAGE

12. COOPERATIVE APPLICATION INSTANCES
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A
INST. B
INST. C
DATABASE DATABASE
FE1 FE2 FE3 FE4 FE5
FRONT-END
APPLICATION
STORAGE

13. MEMBERSHIP PROTOCOL
RINGPOP

14. SWIM GOSSIP PROTOCOL
ENTER RINGPOP: MEMBERSHIP PROTOCOL
PING
INST. A INST. B
INST. A
INST. B
INST. C
INST. A
INST. B
INST. C
INST. C
INST. A
INST. B
INST. C
PING PING
Membership list

15. ENTER RINGPOP: MEMBERSHIP PROTOCOL
PING
INST. A INST. B
INST. A
INST. B
INST. C
INST. A
INST. B
INST. C
INST. C
INST. A
INST. B
INST. C
FAILURE DETECTION

16. ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. B
INST. A
INST. B
INST. C
INST. A
INST. B
INST. C
INST. C
INST. A
INST. B
INST. C
PING-REQ PING
AN INDIRECT PING

17. ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. B
INST. A
INST. B
INST. C
INST. A
INST. B
INST. C
INST. C
INST. A
INST. B
INST. C
PING-REQ
DECLARE INST. B SUSPECT

18. ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. B
INST. A
INST. B
INST. C
INST. A
INST. B
INST. C
INST. C
INST. A
INST. B
INST. C
PING
INST. B
PIGGYBACK MEMBERSHIP UPDATES

19. ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. B
INST. A
INST. B
INST. C
INST. A
INST. B
INST. C
INST. C
INST. A
INST. B
INST. C
PING
INST. B
INFECTION-STYLE DISSEMINATION

20. ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. B
INST. A
INST. B
INST. C
INST. A
INST. B
INST. C
INST. C
INST. A
INST. B
INST. C
PING
DETECTING A FAILURE

21. CREATING A CLUSTER
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A
INST. A
INST. C
INST. B
SENDS JOIN
SENDS JOIN

22. INST. B STARTS UP
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A
INST. A
INST. C
SENDS JOIN
SENDS JOIN
INST. B INST. B

23. PIGGYBACK MEMBERSHIP ON JOIN
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A
INST. A
INST. C
INST. B INST. B
SENDS JOIN
RESPONDS TO JOIN
INST. B

24. “A” APPLIES UPDATE FROM “B”
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A
INST. A
INST. C
INST. B INST. BINST. B
SENDS JOIN
RESPONDS TO JOIN
INST. B

25. “A” PIGGYBACKS UPDATE
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A
INST. A
INST. C
INST. B INST. BINST. B
SENDS JOIN
SENDS PING
INST. A

26. “B” APPLIES “A” UPDATE
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A
INST. A
INST. C
INST. B INST. BINST. B
SENDS JOIN
SENDS PING
INST. A
INST. A

27. CONSISTENT HASHING
RINGPOP

28. ENTER RINGPOP: CONSISTENT HASHING
START WITH A KEYSPACE AND A RING
232
-1

29. ENTER RINGPOP: CONSISTENT HASHING
hash(“INST. A”);
hash(“INST. B”);
hash(“INST. C”);
HASH APPLICATION INSTANCES

30. ENTER RINGPOP: CONSISTENT HASHING
INST. B
INST. C
DIVIDE UP THE KEYSPACE
INST. A

31. ENTER RINGPOP: CONSISTENT HASHING
ASSIGN OWNERSHIP “USER1”
hash(“USER1”);
INST. A
INST. B
INST. C

32. ENTER RINGPOP: CONSISTENT HASHING
ASSIGN OWNERSHIP “USER5”
hash(“USER5”);
INST. B
INST. C
INST. A

33. ENTER RINGPOP: CONSISTENT HASHING
INST. A
INST. B
INST. C
ASSIGN OWNERSHIP “USER8”, “USER4”
hash(“USER4”);
hash(“USER8”);

34. ENTER RINGPOP: CONSISTENT HASHING
LOSING CAPACITY
hash(“USER1”);
hash(“USER5”);
hash(“USER4”);
hash(“USER8”);
INST. A
INST. B
INST. C

35. ADDING CAPACITY
ENTER RINGPOP: CONSISTENT HASHING
INST. D
INST. A
INST. B
INST. C
hash(“USER1”);
hash(“USER5”);
hash(“USER4”);
hash(“USER8”);

36. APPLICATION LAYER MIDDLEWARE
ENTER RINGPOP: REQUEST ROUTING
HTTP / THRIFT / ETC
ROUTING
HASH RING
BUSINESS LOGIC
MEMBERSHIP
INST. A
{RINGPOP
}PROCESS
APPLICATION
FRONT-END
STORAGE

37. PROGRAMMING RINGPOP
INSTANTIATE, BOOTSTRAP, LOOKUP

38. A TYPICAL WEB APP
PROGRAMMING RINGPOP

39. INSTANTIATING RINGPOP
PROGRAMMING RINGPOP

40. BOOTSTRAPPING RINGPOP
PROGRAMMING RINGPOP

41. RING LOOKUPS
PROGRAMMING RINGPOP

42. APPLICATIONS
RINGPOP

43. GEOSPATIAL INDEX
APPLICATIONS OF RINGPOP
REPLICA 2
REPLICA 1
OWNER
INST. D
DISPATCH
UPDATE DRIVER
LOCATION
UPDATE
UPDATE

44. WORK DELEGATION
APPLICATIONS OF RINGPOP
BACKUP 2
BACKUP 1
LEADER
BACKUP 3
POLL DB FOR WORK

45. APPLICATIONS OF RINGPOP
RINGPOP IN PRODUCTION
Geospatial sharding
Work delegation
Server-side push / long-polling
Caching
Aggregation
Mailboxes
Database
Service Discovery and Routing

46. LESSONS LEARNED
RINGPOP

47. Verifying correctness.
Scaling and Failing.
DEVELOPMENT AND STAGING
LESSONS LEARNED

48. Convergence
Cross-pollination
Flappy nodes
Hard to forget
Slow start times
Anti-entropy
Tooling
Backwards compatibility
PRODUCTION
LESSONS LEARNED

49. CONCLUSION
RINGPOP

50. DynamoDB
by Amazon
Riak
by Basho
Serf
by Hashicorp
Cassandra
by Apache
Orleans
by Microsoft
Akka
by Typesafe

51. GRAZIE!
Presented by Jeff Wolski <wolski@uber.com>
Uber is hiring.
Come work with me in our Amsterdam office!