HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir, StumbleUpon
Report
Share
•0 likes•3,766 views
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir, StumbleUpon
•0 likes•3,766 views
Report
Share
Download to read offline
This session is a case study of how we used our already existing HBase cluster as content addressable storage for BLOBs. We will discuss how we wrote a CAS implementation using HBase as the backend, Scala and Finagle as the application and using caching reverse proxies (i.e. Varnish in our case) for serving BLOBs at scale. The talk will dicuss why content addressable storage is the right pattern for many web use cases, how to foster an already existing HBase cluster for better usage of possibly underutilized resources, and operational gotchas to store and serve BLOBs from HBase at scale.
Recommended
More Related Content
What's hot
What's hot(20)
Viewers also liked
Viewers also liked(20)
Similar to HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir, StumbleUpon
Similar to HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir, StumbleUpon(20)
More from Cloudera, Inc.
More from Cloudera, Inc.(20)
Recently uploaded
Recently uploaded(20)
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir, StumbleUpon
- 1. Content Addressable Storages for fun and profit.
- 2. Berk D. Demir @bddemir
- 3. I break and fix things at @StumbleUpon.
- 4. Problem.
- 5. Serve lots of static assets with low latency and high availability.
- 6. Understanding data.
- 7. A lot. 100 million. Small. 19 kilobytes. Frequent. Updates.
- 8. BLOBs don’t change. They get replaced. We want to keep all without duplicating.
- 9. Content Addressable Store
- 10. Store immutable content and authoritatively address it with a cryptographic hash.
- 11. We had ideas.
- 12. Very bad ideas.
- 13. Very bad. Shared Storage, i.e., NFS.
- 14. Bad ideas.
- 15. Bad. AWS S3, RS Cloud Files, ... Distributed: AFS, Gluster HDFS (Oh my!)
- 16. Bad ideas. Take 2
- 17. o_O Write a distributed, fault tolerant, replicating, multi datacenter, fast, CAS for BLOBs.
- 18. Reimplementing a lot of things is generally not a good sign.
- 20. HBase Distributed, Fault tolerant, Replicating, Multi datacenter, Fast.
- 21. Immutable rows with compact keys, separated into different column families based on their access patterns.
- 22. One table to rule them all. m: d: MD5 16 bytes Metadata BLOB (SHA-1 20 bytes) 9 bytes Many bytes MAX_FILESIZE => 20G, VERSION => 1, BLOCKCACHE => true, BLOOMFILTER => ROW Pre-split into 512 regions at table creation time.
- 23. Scala, Finagle, asynchbase, Varnish
- 24. HTTP has a lot to offer.
- 25. Verbs GET HEAD PUT DELETE GET /KwIEec5utYGrKmzXYLgFzg HTTP/1.1 Host: b9.sustatic.com
- 26. Headers Cache-Control: max-age=<1 year> Last-Modified: <cell timestamp> Content-MD5: <row key: base64> Content-Disposition: attachment; filename=su.xpi
- 27. HBase and HTTP are the perfect tools to build simple, reliable, fast data services.
- 28. Get excited and build things!
- 29. Thanks.
- 30. Like the design of this slide deck? Direct your positive feedback to Coda Hale (@coda)