Taming Pythons with ZooKeeper
•
5 likes•2,036 views
Concurrency is hard. Consistency in distributed systems is hard. And then the whole thing should be highly-available and error resilient. Fear not, there are good news: There exists an awesome tool called ZooKeeper to help you with this. There even exists a plethora of Python libraries for it, but how to know what to use and how? This talk will walk you through ZooKeeper and how to use it with Python. We’ll be focusing on what I think is the most prominient ZooKeeper library out there for Python: Kazoo. You’ll see how to do things in ZooKeeper and how to implement them using Kazoo. We’ll also peek in to the recipes Kazoo offers, and if we have enough time, touch a real life application we’ve used Kazoo and ZooKeeper to build at Spotify.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Taming Pythons with ZooKeeper
Similar to Taming Pythons with ZooKeeper (20)
Recently uploaded
Recently uploaded (20)
Taming Pythons with ZooKeeper
- 1. June 29, 2013 Taming Pythons with ZooKeeper Wednesday, 3 July 13
- 2. $ whoami Wednesday, 3 July 13
- 4. Wednesday, 3 July 13
- 5. ZooKeeper? Wednesday, 3 July 13 - This talk is about ZooKeeper. - Preaching Java software in Python conference. I can think of more healthier things to do too. - On side note, our Product Owner is really damn good at this game
- 6. ZooKeeper! Wednesday, 3 July 13 - Apache project, Yahoo 2007 - Consistency & Partition tolerance - Filesystem like, actually can be viewed as a trie, store data in directories too - In memory, limits the dataset you can store. Maximum zookeeper node, znode, data size 1M
- 7. The Tao of ZooKeeper 7 Wednesday, 3 July 13 Orderly, Reliable, Efficient, Timely, Contention free, ambition free
- 8. Wednesday, 3 July 13
- 9. June 29, 2013 One library for the enterprise Curator, from Netflix Wednesday, 3 July 13
- 10. June 29, 2013 Seven for the pythonistas gevent-zookeeper zkpython zc.zk pykeeper twitter’s zookeeper library zoop txzookeeper Wednesday, 3 July 13 - gevent-zookeeper -> Spotify - zkpython segfaults - Others have somewhat OK implementations, but lack core features
- 11. One Library to rule them all* * unless you are running twisted Wednesday, 3 July 13 - txzookeeper still valid for twisted
- 12. Kazoo Wednesday, 3 July 13 - Origins from the Nimbus project - Ben Bangert as the Sauron of ZooKeeper - Not Frodo, that bastard wanted to destroy the perfectly good ring. - All Python, including the protocol. No more segfaults!
- 13. CRUD ZooKeeper CRUD Let’s talk about KaZoo 13 Wednesday, 3 July 13 Create, Read, Update, Delete All basic operations available as async too, but we’ll focus on the synchronous use
- 14. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() TextText Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
- 15. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() TextText zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181') Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
- 16. First we need to connect! Text from kazoo.client import KazooClient zk = KazooClient(hosts='127.0.0.1:2181') zk.start() # ... zk.stop() Text zk = KazooClient(hosts='127.0.0.1:2181/namespace,127.0.0.2:2181') Text zk = KazooClient(hosts='127.0.0.1:2181,127.0.0.2:2181') Wednesday, 3 July 13 Easy to connect to one host, multiple host, with namespace.... ZooKeeper supports connection namespacing! Bonus: get notified when the connection state changes
- 17. Create zk.create('/europython', b'2013') Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
- 18. Create zk.create('/europython', b'2013') zk.create('/europython/jyrki', ephemeral=True) Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
- 19. Create zk.create('/europython', b'2013') zk.create('/europython/jyrki', ephemeral=True) zk.create('/europython/sequential', sequence=True) Wednesday, 3 July 13 Adding nodes easy, helpers exist for recursive creation Ephemeral ZK feature, session + heartbeats, can’t have children! Incremental: guaranteed ever increasing 10 digit number in node name
- 23. Update & Delete zk.set_data('/europython/jyrki', b'nervous') Wednesday, 3 July 13
- 24. Update & Delete zk.set_data('/europython/jyrki', b'nervous') zk.delete('/europython/jyrki') Wednesday, 3 July 13
- 25. Wednesday, 3 July 13
- 27. Textfrom kazoo.recipe import <your-favourite-thing> Wednesday, 3 July 13 Of course it does not have everything from curator
- 28. Want to know when things change? Wednesday, 3 July 13
- 29. watchers Wednesday, 3 July 13
- 30. Text zk.exists('/europython/wine', watch=callback) zk.get('/europython/wine', watch=callback) zk.get_children('/europython/dinners', watch=callback) Wednesday, 3 July 13
- 31. from kazoo.recipe.watchers import DataWatch, ChildWatch @DataWatch('/path/to/node') def data_callback(data, stat): # ... do_something @ChildWatch('/path/to/node') def child_callback(children): # ... do_something Wednesday, 3 July 13
- 32. TESTS Wednesday, 3 July 13
- 33. What if something goes WRONG? Wednesday, 3 July 13
- 34. sys.exit()Wednesday, 3 July 13
- 35. Stand back! It’s time for real life example Wednesday, 3 July 13
- 36. Wednesday, 3 July 13
- 37. SEARCHED FOR THE NEWEST JUSTIN BIEBER Encryption keys were not available to play it. Wednesday, 3 July 13
- 38. SUMMARY Wednesday, 3 July 13
- 39. Guess what, we’re hiring Wednesday, 3 July 13