HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN

1,462 views

Published on

In this talk, you'll learn how Rocket Fuel has developed various HBase access patterns and multi-tenancy scenarios and the role of DeathStar, an in-house solution built on top of Apache Slider and YARN. We'll cover how we use a single YARN cluster to host multiple smaller and highly customized HBase clusters, and how dynamic provisioning and elastic scaling are made possible in this model.

Published in: Software
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,462
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN

  1. 1. DeathStar Easy, Dynamic, Multi-tenant HBase via YARN Ishan Chhabra, Nitin Aggarwal Rocketfuel Inc.
  2. 2. In a not so distant past…
  3. 3. 1000 node cluster
  4. 4. Rogue Applications
  5. 5. Cannot customize per application
  6. 6. Hard to capacity plan or support new applications
  7. 7. The Common Solution: Separate Clusters
  8. 8. Non uniform network usage Different DFSs’, leading to lot of copying of data Low cluster utilization High lead time for new applications
  9. 9. Run HBase on YARN Built on top of Slider Solution: DeathStar
  10. 10. Hangar App Cluster 3 Provisioning Model App Cluster 2 App Cluster 1
  11. 11. (grid/deathstar): $ git commit Capacity planning and configuration discussion Create simple JSON config As applications mature from hangar to their cluster
  12. 12. Dynamic Cluster: Make API call to start, stop and scale cluster Static Cluster: Good to go
  13. 13. Strict Isolation
  14. 14. Common HDFS Layer Bulkload MapReduce over snapshots
  15. 15. Dynamic config and cluster size changes
  16. 16. Fits into organization’s capacity planning model
  17. 17. Clusters out of thin air
  18. 18. Hot swap a new cluster (human error / corruption) Easier HBase version upgrades and testing Temporary scale up for backfill “Dynamic” enables interesting use cases
  19. 19. Key Challenges and Solutions
  20. 20. Another failure mode Taken care of by auto restarts RM HA in the works
  21. 21. Early Days: Bugs Slider did not acknowledge container allocations correctly Fixed scheduled for 0.8: SLIDER-828
  22. 22. Not easily reproducible, still debugging Zombie Regionservers Early Days: Bugs
  23. 23. Long running apps a secondary use case Logging, an unsolved problem Store logs on local disks, considering ELK
  24. 24. YARN/Slider lack certain scheduling constraints At most x instances per node for spread and availability Custom patch in-house
  25. 25. Rolling restarts for config changes Unsolved. SLIDER-226
  26. 26. Data Locality Locality Anti-Locality
  27. 27. Metrics Reporting Custom hadoop metrics OpenTSDB reporter App name passed via config
  28. 28. Conclusion: Is it for me?
  29. 29. Conclusion: Is it worth it?
  30. 30. Thank you! Questions? Reach us at: ishan@rocketfuel.com naggarwal@rocketfuel.com
  31. 31. Are you sure you want to scroll down?
  32. 32. Really? Dungeons and Dragons await…
  33. 33. Key Insight: HBase Multi-Tenancy and Access Patterns
  34. 34. HBase Service Online Operational Store
  35. 35. HBase Data Pipeline 1 Mutable Materialized View Stream 1 Data Pipeline 2 Data Pipeline 3 Stream 2
  36. 36. HBase Prep Stage Transient Cache Stage 1 Stage 2 Stage 3

×