How to create your in-memory KV store
•Download as PPTX, PDF•
0 likes•1,014 views
In this talk we will try to create an in-memory distributed KV store using riak_core library.
Recommended
Recommended
More Related Content
What's hot
What's hot (8)
Similar to How to create your in-memory KV store
Similar to How to create your in-memory KV store (20)
More from Gianluca Padovani
More from Gianluca Padovani (14)
Recently uploaded
Recently uploaded (20)
How to create your in-memory KV store
- 1. Cook your KV How to create your in memory KV store
- 2. GPad Born to be a developer with an interest in distributed system. I develop using many languages like C++, C#, js and ruby. I had fallen in love with functional programming, especially with elixir, erlang. Twitter: https://twitter.com/gpad619 Github: https://github.com/gpad/ Medium: https://medium.com/@gpad CTO & founder of coders51
- 3. Schedule What is riak_core? How to implement a simple distributed command How to implement a command that involves all nodes If we had more time, we would talk about: What happens when I remove a node? How to handle a node crash? Conflict-resolution http://github.com/gpad/no_slides http://medium.com/@GPad/
- 4. What is riak_core? From readme page on github: Riak Core is the distributed systems framework that forms the basis of how Riak distributes data and scales. More generally, it can be thought of as a toolkit for building distributed, scalable, fault-tolerant applications.
- 5. What is riak_core? DO It manages the addition and removal of nodes from the cluster It manages how the requests should be routed inside the cluster It manages how to upgrade cluster having nodes with various capabilities It manages information cluster-wide etc ... DON’T It doesn’t help to store data It isn’t DB centric It doesn’t care what type of data do you use (more or less) It’ doesn’t care what type of computation you do ... etc ...
- 6. How does a riak_core cluster work? Every node is equivalent (masterless) Any client can connect to any node of the cluster The node executes the work or forwards it to correct node Consistent Hashing / VNode
- 7. How to partition data
- 8. How to partition data Node 1Physical node Virtual node
- 9. How to partition data Every virtual node is an Erlang process Every virtual node is an gen_fsm Riak_core defines a behaviour to implement the vnode
- 10. Riak_core_vnode behaviour Define the functions that a module should implement How to manage a command How to manage a coverage command Etc ...
- 11. Implement a distributed command
- 12. Implement a distributed command
- 13. Implement a distributed command
- 14. Implement a distributed command
- 15. Implement a distributed command
- 16. Implement a distributed command
- 17. Implement a distributed command
- 18. Distributed command - Demo!
- 19. All nodes Can we get only one key at time from the cluster? Is it possible to execute a command on all nodes? What do you mean for all nodes?
- 20. All nodes
- 21. All nodes
- 22. All nodes
- 23. All nodes
- 24. All nodes
- 25. All nodes
- 26. All nodes
- 27. All nodes - VNode
- 28. All nodes - Service
- 29. All nodes - Supervisor
- 30. All nodes - Demo!
- 31. What we have done... Create an in-memory KV store that: Can be used from every nodes Can execute a command on a specific node Can execute a command on the entire cluster
- 32. What we have done... Main concepts: Consistent hashing Virtual Node vs Physical Node Preference List
- 33. What we have done... Next concepts: Handoff - Manage the addition and removal of nodes Manage the crash of a node through replications How to use conflict-resolution strategy http://github.com/gpad/no_slides http://medium.com/@GPad/
- 34. Thank you!
- 35. Q&A
Editor's Notes
- On node 1 NoSlides.Service.put(:gpad, 42) NoSlides.Service.put(:joe, 1) NoSlides.Service.put(:asdf, 45) NoSlides.Service.put(:gino, 666) NoSlides.Service.get(:gpad) NoSlides.Service.get(:joe) On node 2: NoSlides.Service.get(:gpad) NoSlides.Service.get(:joe) Remove node … :riak_core.leave On node 1: NoSlides.Service.get(:gpad)