Voldemort

1,810 views

Published on

Voldemort is a distributed data store that is designed as a key-value store used by LinkedIn for high-scalability storage.

Published in: Technology
  • Be the first to comment

Voldemort

  1. 1. Project Voldemort A distributed database. Presented To: Sir Tariq Mehmood Presented By: Fasiha Ikram Aniqa Naeem
  2. 2. Voldemort Voldemort is a distributed data store that is designed as a key-value store used by LinkedIn for high-scalability storage. It is named after the fictional Harry Potter villain Lord Voldemort.
  3. 3. there are job titles, job openings for people, Groups and companies that offerings jobs. Big Data Variety velocity volum e Need frequently read write LinkedIn Big Data Problem
  4. 4. Voldemort Scale Both • The amount of data we can store (write) • The number of requests for that data (read)
  5. 5. Why Not Using Hadopp Naturally the only way to do this is to spread both the load and the data across many servers. 1. Need to find a way to split the data so that all servers have different data 2. Need to find a way to handle server failures without interrupting service 3. HBase still be write-heavy (due to horizontal partitioning and use of SSTables, which are write optimized)
  6. 6. Why voldemort • Data is automatically replicated over multiple servers. • Data is automatically partitioned so each server contains only a subset of the total data • Provides tunable consistency (strict quorum or eventual consistency) • Server failure is handled transparently • Pluggable Storage Engines -- BDB-JE, MySQL, Read-Only
  7. 7. Why voldemort • Pluggable serialization -- Protocol Buffers, Thrift, Avro and Java Serialization • Data items are versioned to maximize data integrity in failure scenarios without compromising availability of the system • Each node is independent of other nodes with no central point of failure or coordination
  8. 8. Why voldemort • Good single node performance: you can expect 10- 20k operations per second depending on the machines, the network, the disk system, and the data replication factor • Support for pluggable data placement strategies to support things like distribution across data centers that are geographically far apart.
  9. 9. Voldemort Storage Engines  Trivial to integrate new persistence mechanisms with Voldemort  2 Classes: Config(data) & Storage Engine(servers info)  3 Operations: put(k, v), get(k), delete(k)  Complication: k is Versioned<Key>
  10. 10. Architecture of Queue
  11. 11. Key-Value Storage • To enable high performance and availability it allow only very simple key-value data access. • Both keys and values can be complex compound objects including lists or maps, but none-the-less the only supported queries are effectively the following: value = store.get(key) store.put(key, value) store.delete(key)
  12. 12. Query execution • Voldemort supports hashtable semantics, so a single value can be modified at a time and retrieval is by primary key. • This makes distribution across machines particularly easy since everything can be split by the primary key.
  13. 13. Consistent Hashing Mechanism • In order to effectively Scaling , the data in Voldemort is split-up in such a way that each item is stored on multiple Servers. • For retrieving data first figure out which is the correct server to use. This partitioning is done via a consistent hashing mechanism that let’s any server calculate the location of data without doing any expensive look ups
  14. 14. Detecting Failure • Voldemort set an SLA (service level agreement) for the requests and ban servers who cannot meet their SLA (this could be because they are down, because requests are timing out). • Servers that violate this SLA get banned for a short period of time, after which they attempt to restore them.
  15. 15. Dealing With Failure  Since each value is stored in multiple places it is possible that one of these servers will not get updated (say because it is crashed when the update occurs).  To solve this problem Voldemort uses a data versioning mechanism called Vector Clocks.  This data versioning allows the servers to detect stale data when it is read and repair it.
  16. 16. Comparison to Hbase databases Query language Architecture Database Model Replication Issues Voldemort API calls Big Unordered Map Key-value NoSQL Distributed data structure Topology Aware Routing Strategies Not Satisfyin g ACID Properti es. Hbase API calls REST XML Thrift Big Multi- dimensional Sorted Map HDFS Master- slave/Master -master replication Master Slave Which Is Not Highly Availabl e
  17. 17. Acid Properties Voldemort Hbase Atomicity Yes Yes Consistency Yes Yes Isolation No Yes Durability Yes Yes Locking model Optimistic Locking MVCC Free for commercial use Yes Yes
  18. 18. Industry Implementation  Linkdin founder of voldemort  shopping cart used by Gilt Groupe, which is powered by Voldemort.
  19. 19. Use Case  High-Performance Key-Value Store (Amazon Dynamo clone)  treats the key‐value store as an API and adds an in‐memory caching layer, which means that you can plug into the back end that makes the most sense for your particular needs.
  20. 20. Pros • only efficient queries are possible, very predictable performance. • easy to distribute across a cluster. • clean separation of storage and logic. • The storage layer is completely mockable so development and unit testing can be done against a throw-away in-memory storage system without needing a real cluster.
  21. 21. Cons • no complex query filters • no foreign key constraints • no triggers • No built-in support for “multiple data center”- aware routing (there must be 1 copy of each key in at least one data center)
  22. 22. Conclusion • It is basically just a big, distributed, persistent, fault-tolerant hash table. • The redundancy of storage makes the system more resilient to server failure. Since each value is stored N times, you can tolerate as many as N – 1 machine failures without data loss.
  23. 23. Refrences 1. http://blog.linkedin.com/2009/04/01/project-voldemort-part-ii-how-it-works/ 2. http://blog.linkedin.com/2009/03/20/project-voldemort-scaling-simple-storage-at- linkedin/ 3. http://en.wikipedia.org/wiki/Voldemort_(distributed_data_store) 4. http://highscalability.com/product-project-voldemort-distributed-database 5. http://www.dummies.com/how-to/content/using-pluggable-storage-with-nosql.html 6. http://vschart.com/compare/project-voldemort/vs/hbase 7. http://www.project-voldemort.com/voldemort/design.html

×