HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN

•

Jun. 05, 2015

• •

DeathStar
Easy, Dynamic, Multi-tenant HBase via
YARN
Ishan Chhabra, Nitin Aggarwal
Rocketfuel Inc.

In a not so distant
past…

1000 node cluster

Next SlideShares

Upcoming SlideShare

Architectural Overview of MapR's Apache Hadoop Distribution

Architectural Overview of MapR's Apache Hadoop Distribution

Loading in …3

×

HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN

Jun. 05, 2015

• •

In this talk, you'll learn how Rocket Fuel has developed various HBase access patterns and multi-tenancy scenarios and the role of DeathStar, an in-house solution built on top of Apache Slider and YARN. We'll cover how we use a single YARN cluster to host multiple smaller and highly customized HBase clusters, and how dynamic provisioning and elastic scaling are made possible in this model.

In this talk, you'll learn how Rocket Fuel has developed various HBase access patterns and multi-tenancy scenarios and the role of DeathStar, an in-house solution built on top of Apache Slider and YARN. We'll cover how we use a single YARN cluster to host multiple smaller and highly customized HBase clusters, and how dynamic provisioning and elastic scaling are made possible in this model.

More Related Content

An Army of Davids: How Markets and Technology Empower Ordinary People to Beat Big Media, Big Government, and Other Goliaths Glenn Reynolds

World Wide Mind: The Coming Integration of Humanity, Machines, and the Internet Michael Chorost

Emergence: The Connected Lives of Ants, Brains, Cities, and Software Steven Johnson

Tubes: A Journey to the Center of the Internet Andrew Blum

The Impulse Economy: Understanding Mobile Shoppers and What Makes Them Buy Gary Schwartz

Hamlet's BlackBerry: A Practical Philosophy for Building a Good Life in the Digital Age William Powers

In the Plex: How Google Thinks, Works, and Shapes Our Lives Steven Levy

Socialnomics: How Social Media Transforms the Way We Live and Do Business Erik Qualman

Public Parts: How Sharing in the Digital Age Improves the Way We Work and Live Jeff Jarvis

The Nature of the Future: Dispatches from the Socialstructed World Marina Gorbis

Talking Back to Facebook: The Common Sense Guide to Raising Kids in the Digital Age James P. Steyer

The Thank You Economy Gary Vaynerchuk

Blog Schmog: The Truth About What Blogs Can (and Can't) Do for Your Business Robert W. Bly

The End of Business As Usual: Rewire the Way You Work to Succeed in the Consumer Revolution Brian Solis

Our Final Invention: Artificial Intelligence and the End of the Human Era James Barrat

No Place to Hide: Edward Snowden, the NSA, and the U.S. Surveillance State Glenn Greenwald

Who Owns the Future? Jaron Lanier

The Social Life of Information: Updated, with a New Preface-Revised John Seely Brown

An Introduction to Information Theory: Symbols, Signals and Noise John R. Pierce

The Dark Net: Inside the Digital Underworld Jamie Bartlett

Cognitive Surplus: Creativity and Generosity in a Connected Age Clay Shirky

The History of the Future: Oculus, Facebook, and the Revolution That Swept Virtual Reality Blake J. Harris

Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are Seth Stephens-Davidowitz

Ten Arguments for Deleting Your Social Media Accounts Right Now Jaron Lanier

The New New Thing: A Silicon Valley Story Michael Lewis

New Dark Age: Technology and the End of the Future James Bridle

Blockchain Revolution: How the Technology Behind Bitcoin Is Changing Money, Business, and the World Don Tapscott

The Emperor's New Mind: Concerning Computers, Minds, and the Laws of Physics Roger Penrose

The Death of Expertise: The Campaign Against Established Knowledge and Why it Matters Tom Nichols

Alone Together: Why We Expect More from Technology and Less from Each Other Sherry Turkle

Algorithms to Live By: The Computer Science of Human Decisions Tom Griffiths

Kill All Normies: Online Culture Wars From 4Chan And Tumblr To Trump And The Alt-Right Angela Nagle

HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN

1. DeathStar Easy, Dynamic, Multi-tenant HBase via YARN Ishan Chhabra, Nitin Aggarwal Rocketfuel Inc.
2. In a not so distant past…
3. 1000 node cluster
4. Rogue Applications
5. Cannot customize per application
6. Hard to capacity plan or support new applications
7. The Common Solution: Separate Clusters
8. Non uniform network usage Different DFSs’, leading to lot of copying of data Low cluster utilization High lead time for new applications
9. Run HBase on YARN Built on top of Slider Solution: DeathStar
10. Hangar App Cluster 3 Provisioning Model App Cluster 2 App Cluster 1
11. (grid/deathstar): $ git commit Capacity planning and configuration discussion Create simple JSON config As applications mature from hangar to their cluster
12. Dynamic Cluster: Make API call to start, stop and scale cluster Static Cluster: Good to go
13. Strict Isolation
14. Common HDFS Layer Bulkload MapReduce over snapshots
15. Dynamic config and cluster size changes
16. Fits into organization’s capacity planning model
17. Clusters out of thin air
18. Hot swap a new cluster (human error / corruption) Easier HBase version upgrades and testing Temporary scale up for backfill “Dynamic” enables interesting use cases
19. Key Challenges and Solutions
20. Another failure mode Taken care of by auto restarts RM HA in the works
21. Early Days: Bugs Slider did not acknowledge container allocations correctly Fixed scheduled for 0.8: SLIDER-828
22. Not easily reproducible, still debugging Zombie Regionservers Early Days: Bugs
23. Long running apps a secondary use case Logging, an unsolved problem Store logs on local disks, considering ELK
24. YARN/Slider lack certain scheduling constraints At most x instances per node for spread and availability Custom patch in-house
25. Rolling restarts for config changes Unsolved. SLIDER-226
26. Data Locality Locality Anti-Locality
27. Metrics Reporting Custom hadoop metrics OpenTSDB reporter App name passed via config
28. Conclusion: Is it for me?
29. Conclusion: Is it worth it?
30. Thank you! Questions? Reach us at: ishan@rocketfuel.com naggarwal@rocketfuel.com
31. Are you sure you want to scroll down?
32. Really? Dungeons and Dragons await…
33. Key Insight: HBase Multi-Tenancy and Access Patterns
34. HBase Service Online Operational Store
35. HBase Data Pipeline 1 Mutable Materialized View Stream 1 Data Pipeline 2 Data Pipeline 3 Stream 2
36. HBase Prep Stage Transient Cache Stage 1 Stage 2 Stage 3