Hadoop Release Plan Feb17
•
1 like•1,232 views
Yahoo plans to backport features from Hadoop trunk to their 0.20 branch to support their needs, which has been deployed on all 25,000 of their nodes. They are working on adding security features to 0.20 by August to meet their requirements, as moving to 0.21 would introduce incompatibilities. Yahoo will likely skip 0.21 and move directly to 0.22, and they question if there is enough external interest to release 0.21.
Report
Share
Report
Share
![Yahoo and 0.20 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Twitter Protobufs And Hadoop Hug 021709
The document discusses using Protocol Buffers at Twitter to store tweets in a way that is robust, efficient, amenable to large-scale analysis, and reusable. Protocol Buffers were chosen because they allow for splitting, efficient parsing, adding new fields, ignoring unused fields, small data size, and hierarchical structure - all important requirements for storing Twitter's large volume of tweet data.
Searching At Scale
The document discusses tools for managing large datasets and performing search operations at scale, including HDFS, Solr, and Katta. It provides guidance on using these tools for indexing, search latency, reliability, and updates. Specifically, it recommends using SolrRecordWriter to build indexes in HDFS, running replicated Solr indexes on Katta for fault tolerance, and using map reduce or full rebuilds to handle index updates. It also provides configuration details for a 12 node Katta cluster with Solr front ends.
Ordered Record Collection
The document discusses different approaches to sorting in MapReduce frameworks over time. It describes Hadoop versions between 0.10-0.22, where sorting was handled by buffering records in memory, spilling to disk when thresholds were exceeded, and merging the spilled files. Later versions improved by distributing the sorting work across maps and making the memory footprint more predictable.
File Context
The document summarizes the motivation and design of the new File system API in Hadoop, including FileContext and AbstractFileSystem. It aims to separate the user API layer from the implementation layer, introduce FileContext as a focus point on multiple file systems, and clean up some issues with the old FileSystem API. The new APIs provide a first-class URI namespace, simplify file system implementation, and allow client configuration to come from the target server.
Hadoop and Voldemort @ LinkedIn
The document discusses Project Voldemort, a distributed key-value storage system developed at LinkedIn. It provides an overview of Voldemort's motivation and features, including high availability, horizontal scalability, and consistency guarantees. It also describes LinkedIn's use of Voldemort and Hadoop for applications like event logging, online lookups, and batch processing of large datasets.
Hadoop Record Reader In Python
This document discusses building a Hadoop Record Reader in Python. It summarizes that Hadoop has a native data storage format called Hadoop Record or "Jute" that has a Data Definition Language and compiler. It then discusses how the author built a Python parsing library to parse this format into generic Python data types by using Ply (lex and yacc for Python). Future work mentioned includes building a DDL translator and integrating feedback.
Hadoop at Yahoo! -- Hadoop World NY 2009
Yahoo! is the largest contributor, tester, and user of Hadoop. They have over 4,000 node Hadoop clusters and use Hadoop to run many of their services and applications. Yahoo! contributes all of their Hadoop development work back to the Apache Foundation as open source software. They are also the largest employer of Hadoop contributors.
Hadoop at Yahoo! -- University Talks
Yahoo is the largest corporate contributor, tester, and user of Hadoop. They have 4000+ node clusters and contribute all their Hadoop development work back to Apache as open source. They use Hadoop for large-scale data processing and analytics across petabytes of data to power services like search and ads optimization. Some challenges of using Hadoop at Yahoo's scale include unpredictable user behavior, distributed systems issues, and the difficulties of collaboration in open source projects.
Recommended
Twitter Protobufs And Hadoop Hug 021709
The document discusses using Protocol Buffers at Twitter to store tweets in a way that is robust, efficient, amenable to large-scale analysis, and reusable. Protocol Buffers were chosen because they allow for splitting, efficient parsing, adding new fields, ignoring unused fields, small data size, and hierarchical structure - all important requirements for storing Twitter's large volume of tweet data.
Searching At Scale
The document discusses tools for managing large datasets and performing search operations at scale, including HDFS, Solr, and Katta. It provides guidance on using these tools for indexing, search latency, reliability, and updates. Specifically, it recommends using SolrRecordWriter to build indexes in HDFS, running replicated Solr indexes on Katta for fault tolerance, and using map reduce or full rebuilds to handle index updates. It also provides configuration details for a 12 node Katta cluster with Solr front ends.
Ordered Record Collection
The document discusses different approaches to sorting in MapReduce frameworks over time. It describes Hadoop versions between 0.10-0.22, where sorting was handled by buffering records in memory, spilling to disk when thresholds were exceeded, and merging the spilled files. Later versions improved by distributing the sorting work across maps and making the memory footprint more predictable.
File Context
The document summarizes the motivation and design of the new File system API in Hadoop, including FileContext and AbstractFileSystem. It aims to separate the user API layer from the implementation layer, introduce FileContext as a focus point on multiple file systems, and clean up some issues with the old FileSystem API. The new APIs provide a first-class URI namespace, simplify file system implementation, and allow client configuration to come from the target server.
Hadoop and Voldemort @ LinkedIn
The document discusses Project Voldemort, a distributed key-value storage system developed at LinkedIn. It provides an overview of Voldemort's motivation and features, including high availability, horizontal scalability, and consistency guarantees. It also describes LinkedIn's use of Voldemort and Hadoop for applications like event logging, online lookups, and batch processing of large datasets.
Hadoop Record Reader In Python
This document discusses building a Hadoop Record Reader in Python. It summarizes that Hadoop has a native data storage format called Hadoop Record or "Jute" that has a Data Definition Language and compiler. It then discusses how the author built a Python parsing library to parse this format into generic Python data types by using Ply (lex and yacc for Python). Future work mentioned includes building a DDL translator and integrating feedback.
Hadoop at Yahoo! -- Hadoop World NY 2009
Yahoo! is the largest contributor, tester, and user of Hadoop. They have over 4,000 node Hadoop clusters and use Hadoop to run many of their services and applications. Yahoo! contributes all of their Hadoop development work back to the Apache Foundation as open source software. They are also the largest employer of Hadoop contributors.
Hadoop at Yahoo! -- University Talks
Yahoo is the largest corporate contributor, tester, and user of Hadoop. They have 4000+ node clusters and contribute all their Hadoop development work back to Apache as open source. They use Hadoop for large-scale data processing and analytics across petabytes of data to power services like search and ads optimization. Some challenges of using Hadoop at Yahoo's scale include unpredictable user behavior, distributed systems issues, and the difficulties of collaboration in open source projects.
HUG August 2010: Best practices
•Arun Murthy, from the Hadoop team at Yahoo! will introduce compendium of best practices for applications running on Apache Hadoop. In fact, we introduce the notion of a Grid Pattern which, similar to Design Pattern, represents a general reusable solution for applications running on the Grid. He will even cover the anti-patterns of applications running on the Apache Hadoop clusters. Arun will enumerate characteristics of well-behaved applications and provide guidance on appropriate uses of various features and capabilities of the Hadoop framework. It is largely prescriptive in its nature; a useful way to look at the presention is to understand that applications that follow, in spirit, the best practices prescribed here are very likely to be efficient, well-behaved in the multi-tenant environment of the Apache Hadoop clusters and unlikely to fall afoul of most policies and limits.
1 content optimization-hug-2010-07-21
The document discusses Yahoo's use of Hadoop to optimize online content by delivering the right content to the right user at the right time through personalized recommendations. It describes Yahoo's modeling framework that uses Pig and HBase to generate user and item features from millions of events and terabytes of data. Models are trained and updated continuously based on real-time user interactions to provide customized content recommendations.
Karmasphere Studio for Hadoop
Karmasphere Studio is an integrated development environment (IDE) for Hadoop that allows users to develop, debug, deploy, and monitor Hadoop jobs. It integrates with major Hadoop distributions and cloud providers like Amazon to enable easy deployment of jobs to private or public clusters. The IDE aims to provide all the typical development and deployment tools within a single interface for working with Hadoop.
2 hadoop@e bay-hug-2010-07-21
The document summarizes eBay's Hadoop analytics platform called Athena:
1. Athena is eBay's large Hadoop cluster comprising over 500 nodes that was built in 2010 to power advanced analytics on complex data.
2. The platform runs the Hadoop ecosystem including HDFS, MapReduce, HBase, Pig and Hive for data access and processing.
3. Athena is used for applications like search relevance ranking, data mining and description extraction to enhance eBay's catalog and search capabilities.
Mumak
Mumak is a discrete event simulation tool that can simulate a large-scale distributed system like Hadoop faster than running actual benchmarks on a cluster. It reproduces the behavior and performance of the real JobTracker and scheduler by plugging them in and simulating different conditions. While still a work in progress, Mumak has been used to find bugs, profile memory usage, and verify that patches address issues without side effects. Future work includes simulating more conditions and using Mumak to generate unit tests.
The Bixo Web Mining Toolkit
The document introduces Bixo, an open source web mining toolkit built on Hadoop, Cascading, and Tika. It provides an example of how Bixo was used to analyze Apache Hadoop mailing list archives to determine the most helpful contributors. The workflow involved collecting email data, parsing it, analyzing messages to score contributors, and producing a ranked list. Bixo allows building custom workflows through a pipe model to extract structured data from unstructured web sources for business intelligence, competitive analysis, and other applications.
Nov 2010 HUG: Business Intelligence for Big Data
The document discusses business intelligence for big data using Hadoop. It describes how 90% of companies are using or plan to use Hadoop to transform structured or semi-structured data for analysis and reporting. While Hadoop provides scalability through distributed processing and storage, its MapReduce programming model makes data transformation difficult for developers accustomed to graphical tools. The document traces how Google and Yahoo developed MapReduce for specific use cases of indexing the internet at massive scales, and how it has since been generalized beyond those specific needs.
Nov 2010 HUG: Fuzzy Table - B.A.H
The document discusses fuzzy matching and describes Fuzzy Table, a scalable solution for performing fuzzy matching on large multimedia databases using Hadoop. Fuzzy Table uses Hadoop for bulk processing tasks like clustering and indexing data. It then enables low-latency fuzzy searches by caching HDFS metadata and performing searches in parallel across data servers, with average query times scaling linearly as servers are added. Future work involves optimizations to reduce I/O latency and reliance on the HDFS Namenode.
Upgrading To The New Map Reduce API
The document summarizes the key changes between the old MapReduce API and the new MapReduce API in Hadoop. Some of the main changes include:
- Renaming all "mapred" packages to "mapreduce"
- Methods can now throw InterruptedException in addition to IOException
- Using Configuration instead of JobConf
- Changes to Mapper, Reducer, and RecordReader interfaces and classes
- Submitting jobs uses the Job class instead of JobConf and JobClient classes
- InputFormats and OutputFormats see changes like new methods and removed classes
HUG Nov 2010: HDFS Raid - Facebook
HDFS RAID is a contrib project in Hadoop that implements RAID techniques to reduce replication of data in HDFS and tolerate block failures. At Facebook, HDFS RAID is saving about 5PB of storage space by using XOR parity with gradual deployment across tables. The key components are the RaidNode daemon that generates parity files, the BlockFixer that reconstructs missing or corrupt blocks, and the DistributedRaidFileSystem that allows reads from corrupt files by reconstructing missing blocks on the fly.
Cloudera Desktop
The document summarizes a presentation about the public beta release of Cloudera Desktop, a web-based platform for accessing and managing Hadoop clusters. Cloudera Desktop aims to improve the user experience of interacting with Hadoop by providing a desktop metaphor and allowing users to browse files, design and run jobs, and check cluster health directly from a web browser. The presentation demonstrates features of Cloudera Desktop and discusses the motivations for its web-based, desktop-inspired interface and extensible platform design. It encourages attendees to try the free public beta and provide feedback.
3 avro hug-2010-07-21
Avro is a data serialization system that provides data interchange and interoperability between systems. It allows for efficient encoding of data and schema evolution. Avro defines data using JSON schemas and provides dynamic typing which allows data to be read without code generation. It includes a file format and supports MapReduce workflows. Avro aims to become the standard data format for Hadoop applications by providing rich data types, interoperability between languages, and compatibility between versions.
Yahoo! Hadoop User Group - May Meetup - Extraordinarily rapid and robust data...
Cascalog is a query language for Hadoop written in Clojure. It provides an alternative to Pig and Hive that leverages Clojure's features like being a functional language, Lisp-based syntax, and first-class integration with Java. Cascalog allows for full use of Clojure during queries with features like custom operations, dynamic queries, and side-by-side use with other Clojure code. BackType uses Cascalog for tasks like identifying influencers and engagement on Twitter.
Public Terabyte Dataset Project: Web crawling with Amazon Elastic MapReduce
The document discusses the Public Terabyte Dataset Project which aims to create a large crawl of top US domains for public use on Amazon's cloud. It describes how the project uses various Amazon Web Services like Elastic MapReduce and SimpleDB along with technologies like Hadoop, Cascading, and Tika for web crawling and data processing. Common issues encountered include configuration problems, slow performance from fetching all web pages or using Tika language detection, and generating log files instead of results.
January 2011 HUG: Howl Presentation
This document introduces Howl, a table management service for Hadoop that aims to provide a shared schema, data typing system, and table abstraction across tools like Pig, Hive, and MapReduce. It discusses Howl's motivations of enabling collaboration, interoperability, and evolvability. The architecture takes elements from Hive and allows different tools to share data. Howl presents data stored in HDFS as tables that can be partitioned and with columns of specific data types. It provides loaders and storages for Pig and input/output formats for MapReduce. The roadmap includes adding write support for Hive, notifications, and other features to improve interoperability between tools processing data on Hadoop.
January 2011 HUG: Pig Presentation
1) Pig 0.8 focuses on usability, integration, performance, and backwards compatibility with 0.7.
2) It allows UDFs to be written in scripting languages like Jython and provides better statistics and integration with Hadoop job history.
3) Other new features include invoking static Java functions as UDFs, improved HBase integration, and casting relations to scalars.
Bay Area HUG Feb 2011 Intro
Bay Area HUG Feb 2011 introduction and Yahoo refocusing on Apache Hadoop releases.
More information at:
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hug-feb-2011-recap/
Next Generation MapReduce
The next generation of Hadoop MapReduce
Arun C. Murthy presented the plans for the next generation of Apache Hadoop MapReduce. The MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization.
More information and video available at:
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hug-feb-2011-recap/
Hadoop Versioning
This document discusses the versioning conventions and history of Hadoop releases. It notes that features were occasionally developed on branches off the trunk codeline and that some releases included features from different branches, causing confusion. It also summarizes the status of Hadoop 1.0, which unified many previously separated features, and the versioning of the Cloudera CDH distribution in relation to Apache Hadoop releases.
Apache Hadoop 0.23
This document summarizes Apache Hadoop release 0.23, which is scheduled to be the first stable release since 0.20 in 2009. Key highlights include improvements to HDFS federation, MapReduce, and high availability. The release aims to support large clusters of thousands of machines with high concurrency. Extensive testing is being done to validate performance gains from changes like MapReduce shuffle reimplementation and optimizations for small jobs. The 0.23 branch is expected in August 2011 with an alpha release in October and production release in late Q1 2012.
Beeswax Hive editor in Hue
This document discusses Hive Editor in Hue, an open source web interface for Hadoop that includes applications for Hive, Pig, Impala, Oozie, Solr, Sqoop, and HBase. The Hive Editor in Hue provides syntax highlighting, query autocomplete, live progress and logs for Hive queries and MapReduce jobs, the ability to work with multiple databases and statements, and features for saving, exporting, and sharing queries. It connects to HiveServer2 and supports Sentry for authorization. A demo of the Hive Editor and Metastore Browser is provided.
Hw09 Clouderas Distribution For Hadoop
This document provides an overview of Cloudera's Distribution for Hadoop (CDH). It explains that CDH is a Hadoop distribution that packages Apache Hadoop and its ecosystem components in an easy to install way, similar to how Linux distributions work. The document outlines what is included in CDH, such as Apache Hadoop, Pig, Hive, HBase and ZooKeeper. It also describes how to install CDH using repositories, tarballs or on Amazon EC2. Finally, it discusses CDH versions and support options available from Cloudera.
More Related Content
Viewers also liked
HUG August 2010: Best practices
•Arun Murthy, from the Hadoop team at Yahoo! will introduce compendium of best practices for applications running on Apache Hadoop. In fact, we introduce the notion of a Grid Pattern which, similar to Design Pattern, represents a general reusable solution for applications running on the Grid. He will even cover the anti-patterns of applications running on the Apache Hadoop clusters. Arun will enumerate characteristics of well-behaved applications and provide guidance on appropriate uses of various features and capabilities of the Hadoop framework. It is largely prescriptive in its nature; a useful way to look at the presention is to understand that applications that follow, in spirit, the best practices prescribed here are very likely to be efficient, well-behaved in the multi-tenant environment of the Apache Hadoop clusters and unlikely to fall afoul of most policies and limits.
1 content optimization-hug-2010-07-21
The document discusses Yahoo's use of Hadoop to optimize online content by delivering the right content to the right user at the right time through personalized recommendations. It describes Yahoo's modeling framework that uses Pig and HBase to generate user and item features from millions of events and terabytes of data. Models are trained and updated continuously based on real-time user interactions to provide customized content recommendations.
Karmasphere Studio for Hadoop
Karmasphere Studio is an integrated development environment (IDE) for Hadoop that allows users to develop, debug, deploy, and monitor Hadoop jobs. It integrates with major Hadoop distributions and cloud providers like Amazon to enable easy deployment of jobs to private or public clusters. The IDE aims to provide all the typical development and deployment tools within a single interface for working with Hadoop.
2 hadoop@e bay-hug-2010-07-21
The document summarizes eBay's Hadoop analytics platform called Athena:
1. Athena is eBay's large Hadoop cluster comprising over 500 nodes that was built in 2010 to power advanced analytics on complex data.
2. The platform runs the Hadoop ecosystem including HDFS, MapReduce, HBase, Pig and Hive for data access and processing.
3. Athena is used for applications like search relevance ranking, data mining and description extraction to enhance eBay's catalog and search capabilities.
Mumak
Mumak is a discrete event simulation tool that can simulate a large-scale distributed system like Hadoop faster than running actual benchmarks on a cluster. It reproduces the behavior and performance of the real JobTracker and scheduler by plugging them in and simulating different conditions. While still a work in progress, Mumak has been used to find bugs, profile memory usage, and verify that patches address issues without side effects. Future work includes simulating more conditions and using Mumak to generate unit tests.
The Bixo Web Mining Toolkit
The document introduces Bixo, an open source web mining toolkit built on Hadoop, Cascading, and Tika. It provides an example of how Bixo was used to analyze Apache Hadoop mailing list archives to determine the most helpful contributors. The workflow involved collecting email data, parsing it, analyzing messages to score contributors, and producing a ranked list. Bixo allows building custom workflows through a pipe model to extract structured data from unstructured web sources for business intelligence, competitive analysis, and other applications.
Nov 2010 HUG: Business Intelligence for Big Data
The document discusses business intelligence for big data using Hadoop. It describes how 90% of companies are using or plan to use Hadoop to transform structured or semi-structured data for analysis and reporting. While Hadoop provides scalability through distributed processing and storage, its MapReduce programming model makes data transformation difficult for developers accustomed to graphical tools. The document traces how Google and Yahoo developed MapReduce for specific use cases of indexing the internet at massive scales, and how it has since been generalized beyond those specific needs.
Nov 2010 HUG: Fuzzy Table - B.A.H
The document discusses fuzzy matching and describes Fuzzy Table, a scalable solution for performing fuzzy matching on large multimedia databases using Hadoop. Fuzzy Table uses Hadoop for bulk processing tasks like clustering and indexing data. It then enables low-latency fuzzy searches by caching HDFS metadata and performing searches in parallel across data servers, with average query times scaling linearly as servers are added. Future work involves optimizations to reduce I/O latency and reliance on the HDFS Namenode.
Upgrading To The New Map Reduce API
The document summarizes the key changes between the old MapReduce API and the new MapReduce API in Hadoop. Some of the main changes include:
- Renaming all "mapred" packages to "mapreduce"
- Methods can now throw InterruptedException in addition to IOException
- Using Configuration instead of JobConf
- Changes to Mapper, Reducer, and RecordReader interfaces and classes
- Submitting jobs uses the Job class instead of JobConf and JobClient classes
- InputFormats and OutputFormats see changes like new methods and removed classes
HUG Nov 2010: HDFS Raid - Facebook
HDFS RAID is a contrib project in Hadoop that implements RAID techniques to reduce replication of data in HDFS and tolerate block failures. At Facebook, HDFS RAID is saving about 5PB of storage space by using XOR parity with gradual deployment across tables. The key components are the RaidNode daemon that generates parity files, the BlockFixer that reconstructs missing or corrupt blocks, and the DistributedRaidFileSystem that allows reads from corrupt files by reconstructing missing blocks on the fly.
Cloudera Desktop
The document summarizes a presentation about the public beta release of Cloudera Desktop, a web-based platform for accessing and managing Hadoop clusters. Cloudera Desktop aims to improve the user experience of interacting with Hadoop by providing a desktop metaphor and allowing users to browse files, design and run jobs, and check cluster health directly from a web browser. The presentation demonstrates features of Cloudera Desktop and discusses the motivations for its web-based, desktop-inspired interface and extensible platform design. It encourages attendees to try the free public beta and provide feedback.
3 avro hug-2010-07-21
Avro is a data serialization system that provides data interchange and interoperability between systems. It allows for efficient encoding of data and schema evolution. Avro defines data using JSON schemas and provides dynamic typing which allows data to be read without code generation. It includes a file format and supports MapReduce workflows. Avro aims to become the standard data format for Hadoop applications by providing rich data types, interoperability between languages, and compatibility between versions.
Yahoo! Hadoop User Group - May Meetup - Extraordinarily rapid and robust data...
Cascalog is a query language for Hadoop written in Clojure. It provides an alternative to Pig and Hive that leverages Clojure's features like being a functional language, Lisp-based syntax, and first-class integration with Java. Cascalog allows for full use of Clojure during queries with features like custom operations, dynamic queries, and side-by-side use with other Clojure code. BackType uses Cascalog for tasks like identifying influencers and engagement on Twitter.
Public Terabyte Dataset Project: Web crawling with Amazon Elastic MapReduce
The document discusses the Public Terabyte Dataset Project which aims to create a large crawl of top US domains for public use on Amazon's cloud. It describes how the project uses various Amazon Web Services like Elastic MapReduce and SimpleDB along with technologies like Hadoop, Cascading, and Tika for web crawling and data processing. Common issues encountered include configuration problems, slow performance from fetching all web pages or using Tika language detection, and generating log files instead of results.
January 2011 HUG: Howl Presentation
This document introduces Howl, a table management service for Hadoop that aims to provide a shared schema, data typing system, and table abstraction across tools like Pig, Hive, and MapReduce. It discusses Howl's motivations of enabling collaboration, interoperability, and evolvability. The architecture takes elements from Hive and allows different tools to share data. Howl presents data stored in HDFS as tables that can be partitioned and with columns of specific data types. It provides loaders and storages for Pig and input/output formats for MapReduce. The roadmap includes adding write support for Hive, notifications, and other features to improve interoperability between tools processing data on Hadoop.
January 2011 HUG: Pig Presentation
1) Pig 0.8 focuses on usability, integration, performance, and backwards compatibility with 0.7.
2) It allows UDFs to be written in scripting languages like Jython and provides better statistics and integration with Hadoop job history.
3) Other new features include invoking static Java functions as UDFs, improved HBase integration, and casting relations to scalars.
Bay Area HUG Feb 2011 Intro
Bay Area HUG Feb 2011 introduction and Yahoo refocusing on Apache Hadoop releases.
More information at:
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hug-feb-2011-recap/
Next Generation MapReduce
The next generation of Hadoop MapReduce
Arun C. Murthy presented the plans for the next generation of Apache Hadoop MapReduce. The MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization.
More information and video available at:
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hug-feb-2011-recap/
Viewers also liked (18)
Yahoo! Hadoop User Group - May Meetup - Extraordinarily rapid and robust data...
Yahoo! Hadoop User Group - May Meetup - Extraordinarily rapid and robust data...
Public Terabyte Dataset Project: Web crawling with Amazon Elastic MapReduce
Public Terabyte Dataset Project: Web crawling with Amazon Elastic MapReduce
Similar to Hadoop Release Plan Feb17
Hadoop Versioning
This document discusses the versioning conventions and history of Hadoop releases. It notes that features were occasionally developed on branches off the trunk codeline and that some releases included features from different branches, causing confusion. It also summarizes the status of Hadoop 1.0, which unified many previously separated features, and the versioning of the Cloudera CDH distribution in relation to Apache Hadoop releases.
Apache Hadoop 0.23
This document summarizes Apache Hadoop release 0.23, which is scheduled to be the first stable release since 0.20 in 2009. Key highlights include improvements to HDFS federation, MapReduce, and high availability. The release aims to support large clusters of thousands of machines with high concurrency. Extensive testing is being done to validate performance gains from changes like MapReduce shuffle reimplementation and optimizations for small jobs. The 0.23 branch is expected in August 2011 with an alpha release in October and production release in late Q1 2012.
Beeswax Hive editor in Hue
This document discusses Hive Editor in Hue, an open source web interface for Hadoop that includes applications for Hive, Pig, Impala, Oozie, Solr, Sqoop, and HBase. The Hive Editor in Hue provides syntax highlighting, query autocomplete, live progress and logs for Hive queries and MapReduce jobs, the ability to work with multiple databases and statements, and features for saving, exporting, and sharing queries. It connects to HiveServer2 and supports Sentry for authorization. A demo of the Hive Editor and Metastore Browser is provided.
Hw09 Clouderas Distribution For Hadoop
This document provides an overview of Cloudera's Distribution for Hadoop (CDH). It explains that CDH is a Hadoop distribution that packages Apache Hadoop and its ecosystem components in an easy to install way, similar to how Linux distributions work. The document outlines what is included in CDH, such as Apache Hadoop, Pig, Hive, HBase and ZooKeeper. It also describes how to install CDH using repositories, tarballs or on Amazon EC2. Finally, it discusses CDH versions and support options available from Cloudera.
Apache Hadoop at 10
This document summarizes the history and evolution of Apache Hadoop over the past 10 years. It discusses how Hadoop originated from Doug Cutting's work on Nutch in 2002. It grew to include HDFS for storage and MapReduce for processing. Yahoo was an early large-scale user. The community has expanded Hadoop to include over 25 components like Hive, HBase, Spark and more. The open source model and ability to adapt have helped Hadoop succeed and it will continue to evolve to handle new data sources and cloud deployments in the next 10 years.
Apache Hadoop 0.22 and Other Versions
The document discusses the Apache Hadoop ecosystem and versions. It provides details on Hadoop versioning from 0.1 to the current versions of 0.22, 0.23, and 1.0. It summarizes the key features and testing of Hadoop 0.22, which has been stabilized by eBay for production use. The document recommends Hadoop 0.22 as a reliable version to use until further versions are released.
Introduction to HiveQL
Interested in learning Hadoop, but you’re overwhelmed by the number of components in the Hadoop ecosystem? You’d like to get some hands on experience with Hadoop but you don’t know Linux or Java? This session will focus on giving a high level explanation of Hive and HiveQL and how you can use them to get started with Hadoop without knowing Linux or Java.
Hive2.0 big dataspain-nov-2016
Most recent version of the Hive 2.0 talk, given at Big Data Spain and Apache Big Data EU in November of 2016
Hadoop demo ppt
This is the basis for some talks I've given at Microsoft Technology Center, the Chicago Mercantile exchange, and local user groups over the past 2 years. It's a bit dated now, but it might be useful to some people. If you like it, have feedback, or would like someone to explain Hadoop or how it and other new tools can help your company, let me know.
Upgrading HDFS to 3.3.0 and deploying RBF in production #LINE_DM
LINE Developer Meetup #68 - Big Data Platformの発表資料です。HDFSのメジャーバージョンアップとRouter-based Federation(RBF)の適用について紹介しています。イベントページ: https://line.connpass.com/event/188176/
Up-Armoring The Elephant: Adding Kerberos-based Security to Hadoop
The document discusses securing Hadoop through adding authentication using Kerberos. It notes that currently Hadoop is not very secure as it is too trusting, allowing bad actors to potentially do bad things. Kerberos provides authentication that requires users and services to authenticate, addressing this issue. The document outlines how Kerberos works and what a secure Hadoop system looks like when integrated with Kerberos, including how MapReduce jobs and other components are written to support the new security features.
Capital onehadoopintro
This document provides an overview of Capital One's plans to introduce Hadoop and discusses several proof of concepts (POCs) that could be developed. It summarizes the history and practices of using Hadoop at other companies like LinkedIn, Netflix, and Yahoo. It then outlines possible POCs for Hadoop distributions, ETL/analytics frameworks, performance testing, and developing a scaling layer. The goal is to contribute open source code and help with Capital One's transition to using Hadoop in production.
Hadoop installation steps
This document provides instructions to install Hadoop version 2.6.0 on an Ubuntu 12.04 LTS machine. It involves updating the system, installing Java, creating a Hadoop user, installing SSH, downloading and extracting Hadoop, configuring files and directories, and starting the Hadoop services. The installation is verified by checking running processes and accessing the NameNode and Secondary NameNode web interfaces. The document concludes by providing commands to stop the Hadoop cluster services.
LibreOffice Development
I had a chance to talk about the history and other interesting tidbits about the LibreOffice project at the Triangle C++ Developer's Group, and here is my slides.
Epub summit 2017 - Readium, the perfect EPUB/PWP companion
Readium provides open-source software for reading digital publications like EPUB. It has four projects including Readium JS and SDK. Readium-2 is a new approach that will have a modular architecture, support additional formats like audiobooks, and be released under a BSD license. It is being developed by a community including the Readium Foundation, Feedbooks, NYPL, and EDRLab to create a sustainable solution and reference reader for EPUB 3.1.
Architecting the Future of Big Data and Search
The document discusses the potential for integrating Apache Lucene and Apache Hadoop technologies. It covers their histories and current uses, as well as opportunities and challenges around making them work better together through tighter integration or code sharing. Developers and businesses are interested in ways to improve searching large amounts of data stored using Hadoop technologies.
Hue: The Hadoop UI - Hadoop Singapore
Learn about Hue, the UI for Apache Hadoop.
Presented by Enrico Berti at the Hadoop Singapore meetup.
Find out everything you need about Hue at http://gethue.com
RHEL/Fedora + Docker (and SELinux)
Is Red Hat / Fedora / Centos ready for lightweight Docker containers? Is Docker secure enough? How about SELinux? How could we deploy Jboss or Django within Docker / RHEL?
I gave this talk at DevOPS meetup in Krakow at 2014-02-26.
Playing with Hadoop (NPW2013)
This document is a presentation on Hadoop given by Søren Lund. It begins with disclaimers that the speaker has no production experience with Hadoop. It then provides an overview of Hadoop, how it addresses the problem of scaling to large amounts of data, and its core components. The presentation demonstrates how to install and run Hadoop on a single machine, provides examples of running word count jobs locally and on Hadoop, and discusses related tools like Hive and Pig. It concludes with notes on the Hadoop user interface, joins, running Hadoop in the cloud, and other Hadoop distributions.
Similar to Hadoop Release Plan Feb17 (19)
Upgrading HDFS to 3.3.0 and deploying RBF in production #LINE_DM
Upgrading HDFS to 3.3.0 and deploying RBF in production #LINE_DM
Up-Armoring The Elephant: Adding Kerberos-based Security to Hadoop
Up-Armoring The Elephant: Adding Kerberos-based Security to Hadoop
Epub summit 2017 - Readium, the perfect EPUB/PWP companion
Epub summit 2017 - Readium, the perfect EPUB/PWP companion
More from Hadoop User Group
Common crawlpresentation
CommonCrawl is a non-profit organization that builds a comprehensive web-scale crawl using Hadoop. It crawls broadly and frequently across all top-level domains, prioritizing the crawl based on rank and freshness. The data is uploaded to Amazon S3 and made widely accessible to enable innovation. CommonCrawl uses a modest Hadoop cluster to crawl over 100 million URLs per day and processes over 800 million documents during post processing. The goal is to reduce the cost of "mapping and reducing the internet" to spur new opportunities.
Hdfs high availability
This document discusses high availability solutions for HDFS and proposes AvatarNode as an improvement. AvatarNode uses an active-standby pair of NameNodes coordinated by Zookeeper. During normal operation, the active NameNode writes transaction logs to persistent storage. The standby NameNode reads the logs to keep its metadata up-to-date. Failover occurs within seconds by switching the roles and updating Zookeeper. This allows clients to retrieve the new primary and resume operations with minimal downtime.
Cascalog internal dsl_preso
Cascalog is an internal DSL for Clojure that allows defining MapReduce workflows for Hadoop. It provides helper functions, a way to define custom functions analogous to UDFs, and functions to programmatically generate all possible data aggregations from an input based on business requirements. The workflows can be unit tested and executed on Hadoop. Cascalog abstracts away lower-level MapReduce details and allows defining the entire workflow within a single language.
Karmasphere hadoop-productivity-tools
This presentation introduces Karmasphere Studio, a productivity suite for Hadoop developers and analysts. It provides point-and-click interfaces for tasks like writing Hadoop jobs using high-level languages like Pig and Hive, debugging jobs on a cluster, and performing common analytical tasks without writing code. The tool aims to automate repetitive data processing tasks and help users understand job performance and results. The presentation demonstrates how Karmasphere Studio can simplify common Hadoop development and analysis challenges.
Building a Scalable Web Crawler with Hadoop
Building a Scalable Web Crawler with Hadoop by Ahad Rana from CommonCrawl
Ahad Rana, engineer at CommonCrawl, will go over CommonCrawl’s extensive use of Hadoop to fulfill their mission of building an open, and accessible Web-Scale crawl. He will discuss their Hadoop data processing pipeline, including their PageRank implementation, describe techniques they use to optimize Hadoop, discuss the design of their URL Metadata service, and conclude with details on how you can leverage the crawl (using Hadoop) today.
Hdfs high availability
This document discusses high availability solutions for HDFS and proposes AvatarNode as an improvement. AvatarNode uses an active-standby pair of NameNodes coordinated by Zookeeper. During normal operation, the active NameNode writes transaction logs to persistent storage. The standby NameNode reads the logs to keep its metadata up-to-date. Failover occurs within seconds by switching the roles and updating Zookeeper. This allows clients to retrieve the new primary and resume operations with minimal downtime.
Pig at Linkedin
Pig at LinkedIn by Chris Riccomini from LinkedIn
Pig is an integral part of data analytics at LinkedIn. Learn about LinkedIn’s analytic stack, and see how Pig is used to design, develop, and deliver data products at LinkedIn. We’ll explore a successful example of Pig deployment at LinkedIn, pain points, and integration with Azkaban, Voldemort, Hadoop, and the rest of LinkedIn’s ecosystem.
1 hadoop security_in_details_hadoop_summit2010
- Yahoo needed to secure its Hadoop clusters to isolate data between different users and prevent unauthorized access, as it had hundreds of users sharing over 20 clusters with sensitive financial and personal data.
- The solution introduced Kerberos authentication to Hadoop to provide distributed authentication across HDFS clients and MapReduce jobs. Authorization controls were also added through access control lists.
- The changes were made largely transparent to users through single sign-on with Kerberos and required only minimal API changes, while maintaining backwards compatibility. Security features were added on schedule and preparation was underway to deploy them broadly across Yahoo's production clusters.
Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...
The document summarizes Twitter's use of Hadoop, Pig, and HBase for their data pipeline. Key points include:
1) Twitter ingests 7 TB of data daily and runs 20,000 Hadoop jobs across a cluster to process logs, tweets, and other data.
2) They use Pig for ETL processes and to load data into HBase and MySQL for querying and reports.
3) HBase is used for mutable data like user tables where updates need to be resolved, while HDFS is used for immutable data like logs.
Yahoo! Hadoop User Group - May 2010 Meetup - Apache Hadoop Release Plans for ...
Tom White discusses release plans for Apache Hadoop 0.21, which includes fixing blockers, creating build artifacts, testing, voting, and later bug fix releases. The 0.21.0 release is aimed at improving quality and should not be used for production. It features a new MapReduce API, file context symlinks, and hundreds of bug fixes and other improvements.
Hadoop, Hbase and Hive- Bay area Hadoop User Group
This document summarizes Facebook's use cases and architecture for integrating Apache Hive and HBase. It discusses loading data from Hive into HBase tables using INSERT statements, querying HBase tables from Hive using SELECT statements, and maintaining low latency access to dimension tables stored in HBase while performing analytics on fact data stored in Hive. The architecture involves writing a storage handler and SerDe to map between the two systems and executing Hive queries by generating MapReduce jobs that read from or write to HBase.
Yahoo! Mail antispam - Bay area Hadoop user group
This document discusses shape-shifting spam and how Hadoop can help fight against it. It describes the challenges of spam filtering at scale given spammers' ability to rapidly change their techniques. It outlines the evolution of anti-spam techniques from manual and rule-based approaches to machine learning models. It then introduces Hadoop and how its distributed processing capabilities allow for more expressive models that can learn from large datasets and quickly update to match spammers' changes. Examples are given of how Hadoop enables standard reporting, ad hoc queries, campaign discovery, and other applications to provide insights into spammer behavior.
Hadoop Security Preview
The document discusses adding security features to Hadoop including authentication and authorization. Kerberos will be used for authentication and users will need to authenticate with Kerberos tickets to access HDFS and MapReduce. The APIs have been updated minimally to support the new security features and web UIs will also require authentication. Some remaining issues are lack of encryption of data in transit or at rest.
Flightcaster Presentation Hadoop
The document discusses using the Cascading framework and Clojure for data processing pipelines. It mentions using Cascading and Clojure together, lessons learned like serialization and data structures for wide data, and potential future directions like abstractions for the planner and MapReduce. The document also asks questions about discrete event simulation and fuzzy joins.
Map Reduce Online
Map Reduce Online presentation at the Hadoop Bay Area User Group, March 24 at Yahoo! Sunnyvale campus
Hadoop Security Preview
Hadoop Security preview presentation at the Hadoop Bay Area User Group, March 24 at Yahoo! Sunnyvale campus
Hadoop Security Preview
Hadoop Security preview presentation at the Hadoop Bay Area User Group, March 24 at Yahoo! Sunnyvale campus
More from Hadoop User Group (17)
Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...
Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...
Yahoo! Hadoop User Group - May 2010 Meetup - Apache Hadoop Release Plans for ...
Yahoo! Hadoop User Group - May 2010 Meetup - Apache Hadoop Release Plans for ...
Hadoop, Hbase and Hive- Bay area Hadoop User Group
Hadoop, Hbase and Hive- Bay area Hadoop User Group
Hadoop Release Plan Feb17
- 1. Hadoop Release Plans February 17th, 2010 Owen O'Malley, Yahoo! Hadoop Team
Editor's Notes
- Fleshed out Context object API include JobClient