Let's Talk Operations! (Hadoop Summit 2014)
Report
Allen WittenauerFollow
Jun. 16, 2014•0 likes•1,754 views
Let's Talk Operations! (Hadoop Summit 2014)
Jun. 16, 2014•0 likes•1,754 views
Download to read offline
Report
Technology
These are the introductory slides I used (in some form or another) for the Let's Talk Operations! sessions for the 2014 Hadoop Summits. No video for this one!
Allen WittenauerFollow
Recommended
Deploying Grid Services Using Apache HadoopAllen Wittenauer
Hadoop 24/7Allen Wittenauer
Next Generation Hadoop OperationsOwen O'Malley
August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...Yahoo Developer Network
March 2011 HUG: Scaling HadoopYahoo Developer Network
Accelerating Shuffle: A Tailor-Made RDMA Solution for Apache Spark with Yuval...Spark Summit
More Related Content
What's hot
Hadoop in Practice (SDN Conference, Dec 2014)Marcel Krcah
Real-time searching of big data with Solr and HadoopRogue Wave Software
Top 10 lessons learned from deploying hadoop in a private cloudRogue Wave Software
All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...Ernie Souhrada
Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014NoSQLmatters
File ContextHadoop User Group
Viewers also liked
Apache Hadoop Shell RewriteAllen Wittenauer
Apache Hadoop for System AdministratorsAllen Wittenauer
Apache Yetus: Intro to Precommit for HBase ContributorsAllen Wittenauer
Apache Yetus: Helping Solve the Last Mile ProblemAllen Wittenauer
Hadoop Operations at LinkedInAllen Wittenauer
Hadoop Performance at LinkedInAllen Wittenauer
Similar to Let's Talk Operations! (Hadoop Summit 2014)
VoltDB and Erlang - Tech planet 2012Eonblast
Introduction to Cassandra and CQL for Java developersJulien Anguenot
Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...Ceph Community
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...smallerror
Fixing_Twitterliujianrong
Fixing twitterRoger Xia
Recently uploaded
V3Cube Gojek Clone - Rebrand With SuperiorityV3cube
Salesforce Miami User Group Event - 3rd QuarterSkyPlanner
Product Research Presentation-Maidy Veloso.pptxMaidyVeloso
Navigating the FutureOnBoard
Framing Few Shot Knowledge Graph Completion with Large Language ModelsMODUL Technology GmbH
Uber Clone Script - Keys to Understanding the Ride Hailing IndustryeSiteWorld TechnoLabs Pvt. Ltd.
Let's Talk Operations! (Hadoop Summit 2014)
- 1. Let’s Talk Operations! Allen Wittenauer!
- 3. Twitter: @_a__w_ Email: aw @ apache.org!
- 4. How many individual grids should I have?
- 5. One big grid Grid per project • Pros! • Lower ops overhead! • One location for all data! • Cons ! • Dev and Prod on one system • Pros! • Capacity planning per project! • Cons ! • More headcount to maintain! • Multiple copies of data! • Data ingress is a mess
- 7. ETL Dev Prod Base ETL Pull Event Feeds Database Feeds Base ETL Pull Base ETL Pull Post-Processed Data
- 9. How do I solve some common distcp issues?
- 10. • Common issues! • Version incompatibilities! • Network bandwidth consumption! ! • Some tricks! • Use WebHDFS! • All modern versions support it! • Read and write in both directions! • Create a separate queue with hard limits! • Pull from larger, push from smaller
- 11. Q&A Allen Wittenauer Twitter: @_a__w_ Email: aw @ apache.org
- 12. Bonus Slide!
- 13. 20 GB /, ... 200 GB task space (rest) HDFS • root partitioning ! ! ! ! ! • non-root partitioning 5 GB swap 200 GB task space (rest) HDFS