This document discusses using ZooKeeper to automatically handle Redis failover. ZooKeeper is an open-source tool that provides primitives for building distributed applications and handles tasks like leader election and quorum management. The presenter describes how his redis_failover Ruby gem uses ZooKeeper to monitor Redis servers, detect failures, and automatically inform clients so they reconnect to the new master, preventing downtime during a failover. Several companies already use this approach with redis_failover to make their Redis infrastructure more robust and fault-tolerant.

1. Handling Redis failover
with ZooKeeper
Ryan LeCompte
BaRuCo Lightning Talk
September 2012

2. Me
• Rubyist at Spreecast in San Francisco
• redis_failover, method_locator,
synchronizable
• http://github.com/ryanlecompte
• lecompte@gmail.com
• @ryanlecompte

3. Typical Redis Setup

4. Master Dies
:-(

5. What happens to your data and
connected clients?
• Resque Jobs?
• Sidekiq Jobs?
• Cached pages?
• Cached stats?
• Rubygems?

6. Scramble to manually
reconfigure slave as master

7. Update all Redis clients to
talk to the new master

8. Restart production services
so they can see the new config

9. What do we really
want to do?
NOTHING

10. Failover should be safe
and automatic

11. Let’s talk about
ZooKeeper

12. Open source Apache
Project (originally from
Yahoo)

13. Provides a set of simple
primitives to help you
build powerful
distributed applications

14. Automatically handles
network partitions &
quorum management

15. Replicated, Highly
Available, Ordered,
Battle-proven

16. Used by Yahoo, Netflix,
Rackspace, Hadoop,
Neo4j, etc

17. How can I use
ZooKeeper in my
architecture?

18. Distributed
Configuration
(update thread pool sizes &
other settings without
requiring a server restart)

19. Distributed Locking
(only one process should be
able to do something at a time)

20. Priority Queues

21. Leader Election, Group
Membership & more

22. Practical application:
Using ZooKeeper to
build automatic
failover for Redis

23. redis_failover
http://github.com/ryanlecompte/redis_failover

24. Provides a smart “failover-
aware” client that wraps
existing Ruby client for
Redis

25. Clients automatically
connect to ZooKeeper
and register for watch
notifications

26. Node Managers monitor
Redis servers for availability
and automatically inform
clients of failover via
ZooKeeper

27. Architecture

28. RedisFailover::Client
Example
# create new client with list of ZooKeeper servers
client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:
2182,localhost:2183')
# write operations automatically routed to master
client.set(‘foo’, ‘bar’)
# read operations automatically routed to a slave
client.smembers(‘data’)

29. Companies using
redis_failover
• Papertrail
• Backupify
• MediaPiston
• IDT Corporation
• others

30. Consider incorporating
ZooKeeper into your overall
architecture.
Utilize the primitives it gives you for
solving your next distributed
programming challenge.

31. Don’t let your Redis servers
go unmanaged in production.
Consider using
redis_failover to sleep
better at night.

32. Thanks for listening!
http://github.com/ryanlecompte/redis_failover
Questions? Ask me at the beach!