• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
20080529dublinpt1
 

20080529dublinpt1

on

  • 3,397 views

One in a series of presentations given at the IBM Cloud Computing Center in Dublin.

One in a series of presentations given at the IBM Cloud Computing Center in Dublin.

Statistics

Views

Total Views
3,397
Views on SlideShare
3,388
Embed Views
9

Actions

Likes
4
Downloads
93
Comments
0

4 Embeds 9

http://www.linkedin.com 4
http://www.tumblr.com 2
http://www.techgig.com 2
http://algebrahub.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    20080529dublinpt1 20080529dublinpt1 Presentation Transcript

    • The Hadoop Community Jeff Hammerbacher Manager, Data May 28 - 29, 2008
    • Who is Using Hadoop? ▪ Mentioned previously: Yahoo!, Powerset, Quantcast, Last.fm, Autodesk ▪ A9.com ▪ Build Amazon’s product search indices ▪ Session analytics ▪ FIM ▪ Log analysis and machine learning ▪ Wikia Search ▪ 125 nodes ▪ Veoh, NetSeer, Krugle, Rapleaf, Joost, New York Times ▪ For more, check out http://wiki.apache.org/hadoop/PoweredBy
    • Hadoop in the ASF ▪ Began life as subproject of Lucene ▪ Now a top level project: http://hadoop.apache.org ▪ HBase now a subproject of Hadoop ▪ Pig and Mahout are related projects ▪ Zookeeper and virtual cluster management project in Apache soon ▪ Apache Software Foundation provides organizational and legal support ▪ Membership in ASF is by invitation only ▪ ASF Members elect a Board of Directors ▪ Each top level project has a Project Management Committee (PMC) ▪ Each PMC has a VP who is an officer of the ASF appointed by the Board ▪ The VP of the Hadoop PMC is Owen O’Malley of Yahoo
    • The Hadoop PMC Name Organization Andrzej Bialecki Getopt Doug Cutting Yahoo! Dhruba Borthakur Facebook Enis Soztutar Agmlab Jim Kellerman Powerset Nigel Daley Yahoo! Owen O’Malley Yahoo! Michael Stack Powerset Christophe Taton INRIA Tom White Independent Consultant
    • Apache Infrastructure for Hadoop ▪ Web site: Apache Forrest ▪ Wiki: MoinMoin ▪ Version Control: Subversion ▪ API Documentation: JavaDoc ▪ Bug Tracking: JIRA ▪ Continuous Build Server: Hudson ▪ IRC Channel: #hadoop on irc.freenode.org ▪ And of course, the mailing lists: ▪ core-user@hadoop.apache.org ▪ core-dev@hadoop.apache.org
    • Contributing to Hadoop ▪ Get comfortable with available documentation on the website ▪ Read through the wiki ▪ Browse the mailing list archives ▪ Dig into the JIRA! ▪ Open source bug tracking software from Atlassian ▪ “Issues”: Bugs, feature requests, documentation requests ▪ Issues categorized by “component” and “version” ▪ “Workflow”: Issue as FSM; each state is a “status” ▪ http://www.atlassian.com/software/jira/docs/latest/introduction.html
    • Contributing to Hadoop More About JIRA ▪ Every Issue has a unique, numbered Key ▪ Type ▪ Status ▪ Priority ▪ Assignee ▪ Reporter ▪ Votes ▪ Watchers
    • Contributing to Hadoop More About Ticket Classification ▪ Status: Open, In Progress, Reopened, Resolved, Closed, or Patch Available ▪ Priorities: Blocker, Critical, Major, Minor, Trivial ▪ Type: Bug, Improvement, New Feature, Task ▪ Voting on an issue means you actively want to see it fixed ▪ Watching an issue means you can passively track progress
    • Contributing to Hadoop More About JIRA ▪ Title, Created Time, Updated Time, Component, Affect/Fix Version, Links/ Sub-Tasks, Description, Comments
    • Contributing to Hadoop Filters and the Issue Navigator ▪ You can view related Issues via the Issue Navigator ▪ “Filter” determines what is shown in the Navigator ▪ Common Filters on the right-hand side of main login ▪ Outstanding, Assigned to Me, Reported by Me, Resolved Recently, Added Recently, Updated Recently, Most Important ▪ “Most Important” Filter just sorts by Issue Priority ▪ I’d recommend the “... Recently” and “Most Important” Filters first ▪ Can also click “Find Issues” on top nav to build your own Filters
    • The JIRA Issue Navigator
    • The JIRA Issue Navigator Creating a New Filter
    • Contributing to Hadoop JIRA Reports and Release Notes ▪ Reports add a visualization component to Filters ▪ Most can be applied to any saved filter ▪ Some Reports have a chart configured ▪ Common Reports: ▪ Road Map ▪ Open Issues ▪ Popular Issues (based on number of Votes) ▪ To keep up with what’s new, the Release Notes are quite useful
    • Future Directions for Hadoop HDFS ▪ For 0.18 ▪ HADOOP-1700: Append to Files in HDFS ▪ Numerous blocking issues; hope to have code freeze by early June ▪ 8 Voters, 21 Watchers ▪ HADOOP-3022: Fast Cluster Restart ▪ HADOOP-1702: Reduce buffer copies when data is written to DFS ▪ HADOOP-3164: Use FileChannel.transferTo() when data is read from DN ▪ HADOOP-3058: Hadoop DFS to report more replication metrics ▪ HADOOP-3246: FTP client over HDFS
    • Future Directions for Hadoop HDFS ▪ Scalability ▪ Separate DFS into multiple volumes and have a NN per volume ▪ Manage volume metadata in Zookeeper ▪ Availability ▪ Mirroring ▪ Have Zookeeper manage metadata ▪ Backup and Recovery ▪ Synchronized global snapshot via ZFS or LVM ▪ http://wiki.apache.org/hadoop/HdfsFutures
    • Future Directions for Hadoop MapReduce ▪ For 0.18 ▪ HADOOP-544: Replace the job, tip and task ids with objects ▪ HADOOP-3245: Provide ability to persist running jobs ▪ HADOOP-3130: Shuffling takes too long to get the last map output ▪ HADOOP-3221: Need a quot;LineBasedTextInputFormatquot; ▪ HADOOP-3149: Supporting multiple outputs for M/R jobs ▪ HADOOP-2182: Input Split details for maps should be logged ▪ HADOOP-3226: Run combiner when merging spills from map output ▪ HADOOP-3227: Implement a binary input/output format for Streaming
    • Future Directions for Hadoop MapReduce ▪ Scheduling ▪ Factor job and task scheduling out of code to allow for testing different policies (HADOOP-3412) ▪ Augment JobTracker to be a resource manager and job scheduler ▪ Speculative Execution Policies ▪ Separate logic for Mapper and Reducer ▪ Break Reducer into more granular tasks ▪ Allow for execution across many different data sources ▪ for example, MySQL
    • Future Directions for Hadoop Other Interesting Tickets ▪ HADOOP-4: Tool to mount dfs on linux ▪ HADOOP-249: Improving Map -> Reduce performance and Task JVM reuse ▪ HADOOP-2510: Map-Reduce 2.0 ▪ HADOOP-2864: Improve the Scalability and Robustness of IPC ▪ HADOOP-2884: Refactor Hadoop package structure and source tree ▪ HADOOP-3366: Shuffle/Merge improvements ▪ HADOOP-3421: Requirements for a Resource Manager for Hadoop ▪ HADOOP-3444: Implementing a Resource Manager (V1) for Hadoop
    • Contributing to Hadoop Patch Submission ▪ http://wiki.apache.org/hadoop/HowToContribute ▪ Basically run “svn diff” on your checkout of trunk and write output to a “.patch” file, then attach it to the issue ▪ Hudson will pick up patch and apply to trunk ▪ Make sure to have tests and JavaDoc comments ▪ Performance regressions tested via DFSIO and GridMix benchmarks
    • Contributing to Hadoop Project Ideas ▪ http://wiki.apache.org/hadoop/ProjectSuggestions ▪ Testing, Tools, and Research ▪ Security ▪ Tools ▪ Performance monitoring and benchmarking ▪ Anomaly detection ▪ General system management
    • Contributing to Hadoop Project Ideas continued ▪ Performance ▪ Speculative execution policies ▪ Resource-aware task scheduling (instead of slot-based) ▪ Better failure detection algorithms ▪ Linear Algebra, Statistics, and Machine Learning ▪ SAS/R for massive data sets ▪ Vector and Matrix algebra libraries ▪ Common statistical functions: point estimation, hypothesis testing ▪ Model training and validation libraries
    • (c) 2008 Facebook, Inc. or its licensors.  quot;Facebookquot; is a registered trademark of Facebook, Inc.. All rights reserved. 1.0