Taming Pythons with ZooKeeper

1,582 views

Published on

Concurrency is hard. Consistency in distributed systems is hard. And then the whole thing should be highly-available and error resilient.

Fear not, there are good news: There exists an awesome tool called ZooKeeper to help you with this. There even exists a plethora of Python libraries for it, but how to know what to use and how?

This talk will walk you through ZooKeeper and how to use it with Python. We’ll be focusing on what I think is the most prominient ZooKeeper library out there for Python: Kazoo.

You’ll see how to do things in ZooKeeper and how to implement them using Kazoo. We’ll also peek in to the recipes Kazoo offers, and if we have enough time, touch a real life application we’ve used Kazoo and ZooKeeper to build at Spotify.

Published in: Technology, Self Improvement
  • Be the first to comment

Taming Pythons with ZooKeeper

  1. 1. June 29, 2013 Taming Pythons with ZooKeeper Wednesday, 3 July 13
  2. 2. $ whoami Wednesday, 3 July 13
  3. 3. Jyrki Pulliainen @nailor jyrki@spotify.com Wednesday, 3 July 13
  4. 4. Wednesday, 3 July 13
  5. 5. ZooKeeper? Wednesday, 3 July 13 - This talk is about ZooKeeper. - Preaching Java software in Python conference. I can think of more healthier things to do too. - On side note, our Product Owner is really damn good at this game
  6. 6. ZooKeeper! Wednesday, 3 July 13 - Apache project, Yahoo 2007 - Consistency & Partition tolerance - Filesystem like, actually can be viewed as a trie, store data in directories too - In memory, limits the dataset you can store. Maximum zookeeper node, znode, data size 1M
  7. 7. The Tao of ZooKeeper 7 Wednesday, 3 July 13 Orderly, Reliable, Efficient, Timely, Contention free, ambition free
  8. 8. Wednesday, 3 July 13
  9. 9. June 29, 2013 One library for the enterprise Curator, from Netflix Wednesday, 3 July 13
  10. 10. June 29, 2013 Seven for the pythonistas gevent-zookeeper zkpython zc.zk pykeeper twitter’s zookeeper library zoop txzookeeper Wednesday, 3 July 13 - gevent-zookeeper -> Spotify - zkpython segfaults - Others have somewhat OK implementations, but lack core features
  11. 11. One Library to rule them all* * unless you are running twisted Wednesday, 3 July 13 - txzookeeper still valid for twisted
  12. 12. Kazoo Wednesday, 3 July 13 - Origins from the Nimbus project - Ben Bangert as the Sauron of ZooKeeper - Not Frodo, that bastard wanted to destroy the perfectly good ring. - All Python, including the protocol. No more segfaults!
  13. 13. CRUD ZooKeeper CRUD Let’s talk about KaZoo 13 Wednesday, 3 July 13 Create, Read, Update, Delete All basic operations available as async too, but we’ll focus on the synchronous use
  14. 14. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() TextText Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
  15. 15. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() TextText zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181') Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
  16. 16. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() Text zk = KazooClient(hosts='127.0.0.1:2181/namespace,127.0.0.2:2181') Text zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181') Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
  17. 17. Create zk.create('/europython', b'2013') Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
  18. 18. Create zk.create('/europython', b'2013') zk.create('/europython/jyrki', ephemeral=True) Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
  19. 19. Create zk.create('/europython', b'2013') zk.create('/europython/jyrki', ephemeral=True) zk.create('/europython/sequential', sequence=True) Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
  20. 20. Read Text zk.get('/europython') Wednesday, 3 July 13
  21. 21. Read Text zk.get('/europython') zk.exists('/europython/jyrki') Wednesday, 3 July 13
  22. 22. Read Text zk.get('/europython') zk.exists('/europython/jyrki') zk.get_children('/europython') Wednesday, 3 July 13
  23. 23. Update & Delete zk.set_data('/europython/jyrki', b'nervous') Wednesday, 3 July 13
  24. 24. Update & Delete zk.set_data('/europython/jyrki', b'nervous') zk.delete('/europython/jyrki') Wednesday, 3 July 13
  25. 25. Wednesday, 3 July 13
  26. 26. Distributed locks? Barrier? Semaphores? Counters? Elections? Wednesday, 3 July 13
  27. 27. Textfrom kazoo.recipe import <your-favourite-thing> Wednesday, 3 July 13 Of course it does not have everything from curator
  28. 28. Want to know when things change? Wednesday, 3 July 13
  29. 29. watchers Wednesday, 3 July 13
  30. 30. Text zk.exists('/europython/wine', watch=callback) zk.get('/europython/wine', watch=callback) zk.get_children('/europython/dinners', watch=callback) Wednesday, 3 July 13
  31. 31. from kazoo.recipe.watchers import DataWatch, ChildWatch @DataWatch('/path/to/node') def data_callback(data, stat): # ... do_something @ChildWatch('/path/to/node') def child_callback(children): # ... do_something Wednesday, 3 July 13
  32. 32. TESTS Wednesday, 3 July 13
  33. 33. What if something goes WRONG? Wednesday, 3 July 13
  34. 34. sys.exit()Wednesday, 3 July 13
  35. 35. Stand back! It’s time for real life example Wednesday, 3 July 13
  36. 36. Wednesday, 3 July 13
  37. 37. SEARCHED FOR THE NEWEST JUSTIN BIEBER Encryption keys were not available to play it. Wednesday, 3 July 13
  38. 38. SUMMARY Wednesday, 3 July 13
  39. 39. Guess what, we’re hiring Wednesday, 3 July 13
  40. 40. Thank You! jyrki@spotify.com @nailor Wednesday, 3 July 13

×