Bhupeshbansal bigdata
•Download as PPT, PDF•
0 likes•675 views
The document discusses big data challenges and solutions. It describes how specialized systems like Hadoop are more efficient than relational databases for large-scale data. It provides examples of open source projects that can be used for tasks like storage, search, streaming data, and batch processing. The document also summarizes the design of the Voldemort distributed key-value store and how it was inspired by Dynamo and Memcached.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Bhupeshbansal bigdata
Similar to Bhupeshbansal bigdata (20)
Recently uploaded
Recently uploaded (20)
Bhupeshbansal bigdata
- 1. Big Data and Me Bhupesh Bansal Feb 3, 2012
- 2. Relational Model Architecture Reference : http:// www.slideshare.net / adorepump / voldemort-nosql
- 3. Linkedin 2006 Reference : http://www.slideshare.net/linkedin/linked-in-javaone-2008-tech-session-comm
- 5. Specialized Systems Architecture Reference : http:// www.slideshare.net / adorepump / voldemort-nosql
- 6. Linkedin 2007 Reference : http://www.slideshare.net/linkedin/linked-in-javaone-2008-tech-session-comm
- 8. Batch Driven Architecture Reference : http:// www.slideshare.net / bhupeshbansal /hadoop-user-group-jan2010
- 9. Motivation I : Big Data 02/06/12 Reference : algo2.iti.kit.edu/.../fopraext/index.html
- 10. Motivation II: Data Driven Features
- 11. Motivation III: Makes Money 02/06/12 Proprietary & Confidential
- 12. Motivation IV: Big Data is cool 02/06/12
- 13. Reference : http:// www.slideshare.net / BenSiscovick /the-business-of-big-data-ia-ventures-8577588
- 28. How Voldemort was born ? Reference : 1) http://www.slideshare.net/bhupeshbansal/hadoop-user-group-jan2010 2) http://www.slideshare.net/adorepump/voldemort-nosql
- 40. What do we use Hadoop/Voldemort for ? Proprietary & Confidential 02/06/12
- 41. Batch Driven Architecture Reference : http:// www.slideshare.net / bhupeshbansal /hadoop-user-group-jan2010
- 42. Data Flow Driven Architecture Reference : http:// sna-projects.com /blog/2011/08/ kafka /
- 43. Questions
Editor's Notes
- Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- Statistical learning as the ultimate agile development tool (Peter Norvig), “business logic” through data rather than code
- No Joins Across data domains due to APIs Within data domains due to performance Natural operation: getAll(id…) Latency: if you want to call 30 services on your main pages, they better be quick (30 * 20ms = 600ms)
- - Strong Consistency: all clients see the same view, even in presence of updates - High Availability: all clients can find some replica of the data, even in the presence of failures Partition-tolerance: the system properties hold even when the system is partitioned high availability : Mantra for websites Better to deal with inconsistencies, because their primary need is to scale well to allow for a smooth user experience.
- Hashing .. Why do we need it ?? Basic problem : Clients need to know which data is where ?? Many ways of solving it Central configuration Hashing Linear hashing works : issue is when cluster is dynamic ?? KeyHash –node IDmapping change for a lot of entries When you add new slots Consistent hashing : preserves key –Node mapping for most of the keys and only change the minimal amount needed How to do it ?? Number of partitions ---------------------------- Arbitrary , each node is allocated many partitions (better load balancing and fault tolerance) Few hundreds to few thousands .. Key partition mapping is fixed and only ownership of partitions can change
- Give example of read and writes with vector clocks Pros and cons vs paxos and 2pc User can supply strategy for handling cases where v1 and v2 are not comparable.
- Fancy way of doing Optimistic locking
- Very simple APIS NO Range Scans .. . No iterator on KeySet / Entry SET : Very hard to fix performance Have plans to provide such an iterator
- Explain about partitions Make things fast by removing slow things, not by tuning HTTP client not performant Separate caching layer
- Transfer time: 30 minutes Can max out a gb network, so be careful
- Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- Questions, comments, etc