Hadoop Release Plan Feb17

1,533 views
1,462 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,533
On SlideShare
0
From Embeds
0
Number of Embeds
619
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Fleshed out Context object API include JobClient
  • Hadoop Release Plan Feb17

    1. 1. Hadoop Release Plans February 17th, 2010 Owen O'Malley, Yahoo! Hadoop Team
    2. 2. Yahoo and 0.20 <ul><li>Yahoo back ported a lot of features from trunk into Yahoo’s 0.20 </li></ul><ul><ul><ul><li>Back port features to support Yahoo needs </li></ul></ul></ul><ul><ul><ul><li>Keep our branch in Git and publish to GitHub </li></ul></ul></ul><ul><ul><ul><li>Has been deployed on all 25,000 nodes </li></ul></ul></ul><ul><ul><ul><li>Improved Capacity scheduler </li></ul></ul></ul><ul><ul><ul><li>Run tasks as users </li></ul></ul></ul><ul><ul><ul><li>Currently have Yahoo 0.20.8 deployed and Yahoo 0.20.9 being tested </li></ul></ul></ul><ul><li>I submitted Apache 0.20.2 rc2 today. </li></ul>
    3. 3. The current state of Hadoop 0.21 <ul><li>Branched on 19 Sep 2009 </li></ul><ul><li>Has a couple of big ticket items </li></ul><ul><ul><ul><li>HDFS sync/flush/append </li></ul></ul></ul><ul><ul><ul><li>Improved MapReduce schedulers </li></ul></ul></ul><ul><ul><ul><li>Run tasks as user </li></ul></ul></ul><ul><li>Lots of blockers still open 3 + 5 + 20 </li></ul><ul><li>Some missing back ports </li></ul><ul><ul><ul><li>libhdfs is still in MapReduce! </li></ul></ul></ul><ul><li>Lots of grunt work left to do </li></ul>
    4. 4. Yahoo and Security <ul><li>Yahoo needs Hadoop Security </li></ul><ul><ul><ul><li>Finish development this month </li></ul></ul></ul><ul><ul><ul><li>Deploy first integration cluster in April </li></ul></ul></ul><ul><ul><ul><li>Deploy to production in August </li></ul></ul></ul><ul><ul><ul><li>Can’t afford to slip dates </li></ul></ul></ul><ul><ul><ul><li>Building on Yahoo 0.20 was lower risk </li></ul></ul></ul><ul><li>Created a new branch Yahoo 0.20.100 </li></ul><ul><li>We will publish to GitHub. </li></ul>
    5. 5. Going Forward <ul><li>For Yahoo, Hadoop 0.21 will be very stale by the time it could be deployed. </li></ul><ul><li>For our customers, moving to 0.21 isn’t worth the cost in incompatibility. </li></ul><ul><li>Yahoo is planning on jumping straight from 0.20 with security to 0.22. </li></ul><ul><li>Is anyone outside of Yahoo interested enough in 0.21 to get it released? </li></ul>

    ×