A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system.
This presentation discusses the follow topics
What is Hadoop?
Need for Hadoop
History of Hadoop
Hadoop Overview
Advantages and Disadvantages of Hadoop
Hadoop Distributed File System
Comparing: RDBMS vs. Hadoop
Advantages and Disadvantages of HDFS
Hadoop frameworks
Modules of Hadoop frameworks
Features of 'Hadoop‘
Hadoop Analytics Tools
Presentation slides of the workshop on "Introduction to Pig" at Fifth Elephant, Bangalore, India on 26th July, 2012.
http://fifthelephant.in/2012/workshop-pig
A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system.
This presentation discusses the follow topics
What is Hadoop?
Need for Hadoop
History of Hadoop
Hadoop Overview
Advantages and Disadvantages of Hadoop
Hadoop Distributed File System
Comparing: RDBMS vs. Hadoop
Advantages and Disadvantages of HDFS
Hadoop frameworks
Modules of Hadoop frameworks
Features of 'Hadoop‘
Hadoop Analytics Tools
Presentation slides of the workshop on "Introduction to Pig" at Fifth Elephant, Bangalore, India on 26th July, 2012.
http://fifthelephant.in/2012/workshop-pig
Apache Spark - Basics of RDD | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab
Big Data with Hadoop & Spark Training: http://bit.ly/2L4rPmM
This CloudxLab Basics of RDD tutorial helps you to understand Basics of RDD in detail. Below are the topics covered in this tutorial:
1) What is RDD - Resilient Distributed Datasets
2) Creating RDD in Scala
3) RDD Operations - Transformations & Actions
4) RDD Transformations - map() & filter()
5) RDD Actions - take() & saveAsTextFile()
6) Lazy Evaluation & Instant Evaluation
7) Lineage Graph
8) flatMap and Union
9) Scala Transformations - Union
10) Scala Actions - saveAsTextFile(), collect(), take() and count()
11) More Actions - reduce()
12) Can We Use reduce() for Computing Average?
13) Solving Problems with Spark
14) Compute Average and Standard Deviation with Spark
15) Pick Random Samples From a Dataset using Spark
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...Simplilearn
This presentation about Hive will help you understand the history of Hive, what is Hive, Hive architecture, data flow in Hive, Hive data modeling, Hive data types, different modes in which Hive can run on, differences between Hive and RDBMS, features of Hive and a demo on HiveQL commands. Hive is a data warehouse system which is used for querying and analyzing large datasets stored in HDFS. Hive uses a query language called HiveQL which is similar to SQL. Hive issues SQL abstraction to integrate SQL queries (like HiveQL) into Java without the necessity to implement queries in the low-level Java API. Now, let us get started and understand Hadoop Hive in detail
Below topics are explained in this Hive presetntation:
1. History of Hive
2. What is Hive?
3. Architecture of Hive
4. Data flow in Hive
5. Hive data modeling
6. Hive data types
7. Different modes of Hive
8. Difference between Hive and RDBMS
9. Features of Hive
10. Demo on HiveQL
What is this Big Data Hadoop training course about?
The Big Data Hadoop and Spark developer course have been designed to impart in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab.
What are the course objectives?
This course will enable you to:
1. Understand the different components of the Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark
2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management
3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts
4. Get an overview of Sqoop and Flume and describe how to ingest data using them
5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning
6. Understand different types of file formats, Avro Schema, using Arvo with Hive, and Sqoop and Schema evolution
7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations
8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS
9. Gain a working knowledge of Pig and its components
10. Do functional programming in Spark
11. Understand resilient distribution datasets (RDD) in detail
12. Implement and build Spark applications
13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques
14. Understand the common use-cases of Spark and the various interactive algorithms
15. Learn Spark SQL, creating, transforming, and querying Data frames
Learn more at https://www.simplilearn.com/big-data-and-analytics/big-data-and-hadoop-training
Compression Options in Hadoop - A Tale of TradeoffsDataWorks Summit
Yahoo! is one of the most-visited web sites in the world. It runs one of the largest private cloud infrastructures, one that operates on petabytes of data every day. Being able to store and manage that data well is essential to the efficient functioning of Yahoo!`s Hadoop clusters. A key component that enables this efficient operation is data compression. With regard to compression algorithms, there is an underlying tension between compression ratio and compression performance. Consequently, Hadoop provides support for several compression algorithms, including gzip, bzip2, Snappy, LZ4 and others. This plethora of options can make it difficult for users to select appropriate codecs for their MapReduce jobs. This paper attempts to provide guidance in that regard. Performance results with Gridmix and with several corpuses of data are presented. The paper also describes enhancements we have made to the bzip2 codec that improve its performance. This will be of particular interest to the increasing number of users operating on “Big Data” who require the best possible ratios. The impact of using the Intel IPP libraries is also investigated; these have the potential to improve performance significantly. Finally, a few proposals for future enhancements to Hadoop in this area are outlined.
This presentation about Hadoop architecture will help you understand the architecture of Apache Hadoop in detail. In this video, you will learn what is Hadoop, components of Hadoop, what is HDFS, HDFS architecture, Hadoop MapReduce, Hadoop MapReduce example, Hadoop YARN and finally, a demo on MapReduce. Apache Hadoop offers a versatile, adaptable and reliable distributed computing big data framework for a group of systems with capacity limit and local computing power. After watching this video, you will also understand the Hadoop Distributed File System and its features along with the practical implementation.
Below are the topics covered in this Hadoop Architecture presentation:
1. What is Hadoop?
2. Components of Hadoop
3. What is HDFS?
4. HDFS Architecture
5. Hadoop MapReduce
6. Hadoop MapReduce Example
7. Hadoop YARN
8. Demo on MapReduce
What are the course objectives?
This course will enable you to:
1. Understand the different components of Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark
2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management
3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts
4. Get an overview of Sqoop and Flume and describe how to ingest data using them
5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning
6. Understand different types of file formats, Avro Schema, using Arvo with Hive, and Sqoop and Schema evolution
7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations
8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS
9. Gain a working knowledge of Pig and its components
10. Do functional programming in Spark
11. Understand resilient distribution datasets (RDD) in detail
12. Implement and build Spark applications
13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques
14. Understand the common use-cases of Spark and the various interactive algorithms
15. Learn Spark SQL, creating, transforming, and querying Data frames
Who should take up this Big Data and Hadoop Certification Training Course?
Big Data career opportunities are on the rise, and Hadoop is quickly becoming a must-know technology for the following professionals:
1. Software Developers and Architects
2. Analytics Professionals
3. Senior IT professionals
4. Testing and Mainframe professionals
5. Data Management Professionals
6. Business Intelligence Professionals
7. Project Managers
8. Aspiring Data Scientists
Learn more at https://www.simplilearn.com/big-data-and-analytics/big-data-and-hadoop-training
Apache Hadoop Tutorial | Hadoop Tutorial For Beginners | Big Data Hadoop | Ha...Edureka!
This Edureka "Hadoop tutorial For Beginners" ( Hadoop Blog series: https://goo.gl/LFesy8 ) will help you to understand the problem with traditional system while processing Big Data and how Hadoop solves it. This tutorial will provide you a comprehensive idea about HDFS and YARN along with their architecture that has been explained in a very simple manner using examples and practical demonstration. At the end, you will get to know how to analyze Olympic data set using Hadoop and gain useful insights.
Below are the topics covered in this tutorial:
1. Big Data Growth Drivers
2. What is Big Data?
3. Hadoop Introduction
4. Hadoop Master/Slave Architecture
5. Hadoop Core Components
6. HDFS Data Blocks
7. HDFS Read/Write Mechanism
8. What is MapReduce
9. MapReduce Program
10. MapReduce Job Workflow
11. Hadoop Ecosystem
12. Hadoop Use Case: Analyzing Olympic Dataset
HDFS is a Java-based file system that provides scalable and reliable data storage, and it was designed to span large clusters of commodity servers. HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks.
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.
Hadoop is the popular open source like Facebook, Twitter, RFID readers, sensors, and implementation of MapReduce, a powerful tool so on.Your management wants to derive designed for deep analysis and transformation of information from both the relational data and thevery large data sets. Hadoop enables you to unstructuredexplore complex data, using custom analyses data, and wants this information as soon astailored to your information and questions. possible.Hadoop is the system that allows unstructured What should you do? Hadoop may be the answer!data to be distributed across hundreds or Hadoop is an open source project of the Apachethousands of machines forming shared nothing Foundation.clusters, and the execution of Map/Reduce It is a framework written in Java originallyroutines to run on the data in that cluster. Hadoop developed by Doug Cutting who named it after hishas its own filesystem which replicates data to sons toy elephant.multiple nodes to ensure if one node holding data Hadoop uses Google’s MapReduce and Google Filegoes down, there are at least 2 other nodes from System technologies as its foundation.which to retrieve that piece of information. This It is optimized to handle massive quantities of dataprotects the data availability from node failure, which could be structured, unstructured orsomething which is critical when there are many semi-structured, using commodity hardware, thatnodes in a cluster (aka RAID at a server level). is, relatively inexpensive computers. This massive parallel processing is done with greatWhat is Hadoop? performance. However, it is a batch operation handling massive quantities of data, so theThe data are stored in a relational database in your response time is not immediate.desktop computer and this desktop computer As of Hadoop version 0.20.2, updates are nothas no problem handling this load. possible, but appends will be possible starting inThen your company starts growing very quickly, version 0.21.and that data grows to 10GB. Hadoop replicates its data across differentAnd then 100GB. computers, so that if one goes down, the data areAnd you start to reach the limits of your current processed on one of the replicated computers.desktop computer. Hadoop is not suitable for OnLine Transaction So you scale-up by investing in a larger computer, Processing workloads where data are randomly and you are then OK for a few more months. accessed on structured data like a relational When your data grows to 10TB, and then 100TB. database.Hadoop is not suitable for OnLineAnd you are fast approaching the limits of that Analytical Processing or Decision Support Systemcomputer. workloads where data are sequentially accessed onMoreover, you are now asked to feed your structured data like a relational database, to application with unstructured data coming from generate reports that provide business sources intelligence. Hadoop is used for Big Data. It complements OnLine Transaction Processing and OnLine Analytical Pro
Apache Spark - Basics of RDD | Big Data Hadoop Spark Tutorial | CloudxLabCloudxLab
Big Data with Hadoop & Spark Training: http://bit.ly/2L4rPmM
This CloudxLab Basics of RDD tutorial helps you to understand Basics of RDD in detail. Below are the topics covered in this tutorial:
1) What is RDD - Resilient Distributed Datasets
2) Creating RDD in Scala
3) RDD Operations - Transformations & Actions
4) RDD Transformations - map() & filter()
5) RDD Actions - take() & saveAsTextFile()
6) Lazy Evaluation & Instant Evaluation
7) Lineage Graph
8) flatMap and Union
9) Scala Transformations - Union
10) Scala Actions - saveAsTextFile(), collect(), take() and count()
11) More Actions - reduce()
12) Can We Use reduce() for Computing Average?
13) Solving Problems with Spark
14) Compute Average and Standard Deviation with Spark
15) Pick Random Samples From a Dataset using Spark
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...Simplilearn
This presentation about Hive will help you understand the history of Hive, what is Hive, Hive architecture, data flow in Hive, Hive data modeling, Hive data types, different modes in which Hive can run on, differences between Hive and RDBMS, features of Hive and a demo on HiveQL commands. Hive is a data warehouse system which is used for querying and analyzing large datasets stored in HDFS. Hive uses a query language called HiveQL which is similar to SQL. Hive issues SQL abstraction to integrate SQL queries (like HiveQL) into Java without the necessity to implement queries in the low-level Java API. Now, let us get started and understand Hadoop Hive in detail
Below topics are explained in this Hive presetntation:
1. History of Hive
2. What is Hive?
3. Architecture of Hive
4. Data flow in Hive
5. Hive data modeling
6. Hive data types
7. Different modes of Hive
8. Difference between Hive and RDBMS
9. Features of Hive
10. Demo on HiveQL
What is this Big Data Hadoop training course about?
The Big Data Hadoop and Spark developer course have been designed to impart in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab.
What are the course objectives?
This course will enable you to:
1. Understand the different components of the Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark
2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management
3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts
4. Get an overview of Sqoop and Flume and describe how to ingest data using them
5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning
6. Understand different types of file formats, Avro Schema, using Arvo with Hive, and Sqoop and Schema evolution
7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations
8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS
9. Gain a working knowledge of Pig and its components
10. Do functional programming in Spark
11. Understand resilient distribution datasets (RDD) in detail
12. Implement and build Spark applications
13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques
14. Understand the common use-cases of Spark and the various interactive algorithms
15. Learn Spark SQL, creating, transforming, and querying Data frames
Learn more at https://www.simplilearn.com/big-data-and-analytics/big-data-and-hadoop-training
Compression Options in Hadoop - A Tale of TradeoffsDataWorks Summit
Yahoo! is one of the most-visited web sites in the world. It runs one of the largest private cloud infrastructures, one that operates on petabytes of data every day. Being able to store and manage that data well is essential to the efficient functioning of Yahoo!`s Hadoop clusters. A key component that enables this efficient operation is data compression. With regard to compression algorithms, there is an underlying tension between compression ratio and compression performance. Consequently, Hadoop provides support for several compression algorithms, including gzip, bzip2, Snappy, LZ4 and others. This plethora of options can make it difficult for users to select appropriate codecs for their MapReduce jobs. This paper attempts to provide guidance in that regard. Performance results with Gridmix and with several corpuses of data are presented. The paper also describes enhancements we have made to the bzip2 codec that improve its performance. This will be of particular interest to the increasing number of users operating on “Big Data” who require the best possible ratios. The impact of using the Intel IPP libraries is also investigated; these have the potential to improve performance significantly. Finally, a few proposals for future enhancements to Hadoop in this area are outlined.
This presentation about Hadoop architecture will help you understand the architecture of Apache Hadoop in detail. In this video, you will learn what is Hadoop, components of Hadoop, what is HDFS, HDFS architecture, Hadoop MapReduce, Hadoop MapReduce example, Hadoop YARN and finally, a demo on MapReduce. Apache Hadoop offers a versatile, adaptable and reliable distributed computing big data framework for a group of systems with capacity limit and local computing power. After watching this video, you will also understand the Hadoop Distributed File System and its features along with the practical implementation.
Below are the topics covered in this Hadoop Architecture presentation:
1. What is Hadoop?
2. Components of Hadoop
3. What is HDFS?
4. HDFS Architecture
5. Hadoop MapReduce
6. Hadoop MapReduce Example
7. Hadoop YARN
8. Demo on MapReduce
What are the course objectives?
This course will enable you to:
1. Understand the different components of Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark
2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management
3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts
4. Get an overview of Sqoop and Flume and describe how to ingest data using them
5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning
6. Understand different types of file formats, Avro Schema, using Arvo with Hive, and Sqoop and Schema evolution
7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations
8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS
9. Gain a working knowledge of Pig and its components
10. Do functional programming in Spark
11. Understand resilient distribution datasets (RDD) in detail
12. Implement and build Spark applications
13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques
14. Understand the common use-cases of Spark and the various interactive algorithms
15. Learn Spark SQL, creating, transforming, and querying Data frames
Who should take up this Big Data and Hadoop Certification Training Course?
Big Data career opportunities are on the rise, and Hadoop is quickly becoming a must-know technology for the following professionals:
1. Software Developers and Architects
2. Analytics Professionals
3. Senior IT professionals
4. Testing and Mainframe professionals
5. Data Management Professionals
6. Business Intelligence Professionals
7. Project Managers
8. Aspiring Data Scientists
Learn more at https://www.simplilearn.com/big-data-and-analytics/big-data-and-hadoop-training
Apache Hadoop Tutorial | Hadoop Tutorial For Beginners | Big Data Hadoop | Ha...Edureka!
This Edureka "Hadoop tutorial For Beginners" ( Hadoop Blog series: https://goo.gl/LFesy8 ) will help you to understand the problem with traditional system while processing Big Data and how Hadoop solves it. This tutorial will provide you a comprehensive idea about HDFS and YARN along with their architecture that has been explained in a very simple manner using examples and practical demonstration. At the end, you will get to know how to analyze Olympic data set using Hadoop and gain useful insights.
Below are the topics covered in this tutorial:
1. Big Data Growth Drivers
2. What is Big Data?
3. Hadoop Introduction
4. Hadoop Master/Slave Architecture
5. Hadoop Core Components
6. HDFS Data Blocks
7. HDFS Read/Write Mechanism
8. What is MapReduce
9. MapReduce Program
10. MapReduce Job Workflow
11. Hadoop Ecosystem
12. Hadoop Use Case: Analyzing Olympic Dataset
HDFS is a Java-based file system that provides scalable and reliable data storage, and it was designed to span large clusters of commodity servers. HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks.
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.
Hadoop is the popular open source like Facebook, Twitter, RFID readers, sensors, and implementation of MapReduce, a powerful tool so on.Your management wants to derive designed for deep analysis and transformation of information from both the relational data and thevery large data sets. Hadoop enables you to unstructuredexplore complex data, using custom analyses data, and wants this information as soon astailored to your information and questions. possible.Hadoop is the system that allows unstructured What should you do? Hadoop may be the answer!data to be distributed across hundreds or Hadoop is an open source project of the Apachethousands of machines forming shared nothing Foundation.clusters, and the execution of Map/Reduce It is a framework written in Java originallyroutines to run on the data in that cluster. Hadoop developed by Doug Cutting who named it after hishas its own filesystem which replicates data to sons toy elephant.multiple nodes to ensure if one node holding data Hadoop uses Google’s MapReduce and Google Filegoes down, there are at least 2 other nodes from System technologies as its foundation.which to retrieve that piece of information. This It is optimized to handle massive quantities of dataprotects the data availability from node failure, which could be structured, unstructured orsomething which is critical when there are many semi-structured, using commodity hardware, thatnodes in a cluster (aka RAID at a server level). is, relatively inexpensive computers. This massive parallel processing is done with greatWhat is Hadoop? performance. However, it is a batch operation handling massive quantities of data, so theThe data are stored in a relational database in your response time is not immediate.desktop computer and this desktop computer As of Hadoop version 0.20.2, updates are nothas no problem handling this load. possible, but appends will be possible starting inThen your company starts growing very quickly, version 0.21.and that data grows to 10GB. Hadoop replicates its data across differentAnd then 100GB. computers, so that if one goes down, the data areAnd you start to reach the limits of your current processed on one of the replicated computers.desktop computer. Hadoop is not suitable for OnLine Transaction So you scale-up by investing in a larger computer, Processing workloads where data are randomly and you are then OK for a few more months. accessed on structured data like a relational When your data grows to 10TB, and then 100TB. database.Hadoop is not suitable for OnLineAnd you are fast approaching the limits of that Analytical Processing or Decision Support Systemcomputer. workloads where data are sequentially accessed onMoreover, you are now asked to feed your structured data like a relational database, to application with unstructured data coming from generate reports that provide business sources intelligence. Hadoop is used for Big Data. It complements OnLine Transaction Processing and OnLine Analytical Pro
introduction to data processing using Hadoop and PigRicardo Varela
In this talk we make an introduction to data processing with big data and review the basic concepts in MapReduce programming with Hadoop. We also comment about the use of Pig to simplify the development of data processing applications
YDN Tuesdays are geek meetups organized the first Tuesday of each month by YDN in London
Hadoop, Pig, and Twitter (NoSQL East 2009)Kevin Weil
A talk on the use of Hadoop and Pig inside Twitter, focusing on the flexibility and simplicity of Pig, and the benefits of that for solving real-world big data problems.
Apache Hive provides SQL-like access to your stored data in Apache Hadoop. Apache HBase stores tabular data in Hadoop and supports update operations. The combination of these two capabilities is often desired, however, the current integration show limitations such as performance issues. In this talk, Enis Soztutar will present an overview of Hive and HBase and discuss new updates/improvements from the community on the integration of these two projects. Various techniques used to reduce data exchange and improve efficiency will also be provided.
Big Data and Hadoop training course is designed to provide knowledge and skills to become a successful Hadoop Developer. In-depth knowledge of concepts such as Hadoop Distributed File System, Setting up the Hadoop Cluster, Map-Reduce,PIG, HIVE, HBase, Zookeeper, SQOOP etc. will be covered in the course.
This is the basis for some talks I've given at Microsoft Technology Center, the Chicago Mercantile exchange, and local user groups over the past 2 years. It's a bit dated now, but it might be useful to some people. If you like it, have feedback, or would like someone to explain Hadoop or how it and other new tools can help your company, let me know.
In this document, we will present a very brief introduction to BigData (what is BigData?), Hadoop (how does Hadoop fits the picture?) and Cloudera Hadoop (what is the difference between Cloudera Hadoop and regular Hadoop?).
Please note that this document is for Hadoop beginners looking for a place to start.
In KDD2011, Vijay Narayanan (Yahoo!) and Milind Bhandarkar (Greenplum Labs, EMC) conducted a tutorial on "Modeling with Hadoop". This is the first half of the tutorial.
Apache Hadoop has emerged as the storage and processing platform of choice for Big Data. In this tutorial, I will give an overview of Apache Hadoop and its ecosystem, with specific use cases. I will explain the MapReduce programming framework in detail, and outline how it interacts with Hadoop Distributed File System (HDFS). While Hadoop is written in Java, MapReduce applications can be written using a variety of languages using a framework called Hadoop Streaming. I will give several examples of MapReduce applications using Hadoop Streaming.
Hadoop is a Java software framework that supports data-intensive distributed applications and is developed under open source license. It enables applications to work with thousands of nodes and petabytes of data.
Disclaimer :
The images, company, product and service names that are used in this presentation, are for illustration purposes only. All trademarks and registered trademarks are the property of their respective owners.
Data/Image collected from various sources from Internet.
Intention was to present the big picture of Big Data & Hadoop
We Provide Hadoop training institute in Hyderabad and Bangalore with corporate training by 12+ Experience faculty.
Real-time industry experts from MNCs
Resume Preparation by expert Professionals
Lab exercises
Interview Preparation
Experts advice
"Big Data" is a much-hyped term nowadays in Business Computing. However, the core concept of collaborative environments conducting experiments over large shared data repositories has existed for decades. In this talk, I will outline how recent advances in Cloud Computing, Big Data processing frameworks, and agile application development platforms enable Data Intensive Cloud Applications. I will provide a brief history of efforts in building scalable & adaptive run-time environments, and the role these runtime systems will play in new Cloud Applications. I will present a vision for cloud platforms for science, where data-intensive frameworks such as Apache Hadoop will play a key role.
Apache Hadoop, since its humble beginning as an execution engine for web crawler and building search indexes, has matured into a general purpose distributed application platform and data store. Large Scale Machine Learning (LSML) techniques and algorithms proved to be quite tricky for Hadoop to handle, ever since we started offering Hadoop as a service at Yahoo in 2006. In this talk, I will discuss early experiments of implementing LSML algorithms on Hadoop at Yahoo. I will describe how it changed Hadoop, and led to generalization of the Hadoop platform to accommodate programming paradigms other than MapReduce. I will unveil some of our recent efforts to incorporate diverse LSML runtimes into Hadoop, evolving it to become *THE* LSML platform. I will also make a case for an industry-standard LSML benchmark, based on common deep analytics pipelines that utilize LSML workload.
The Zoo Expands: Labrador *Loves* Elephant, Thanks to HamsterMilind Bhandarkar
The refactoring of Hadoop MapReduce framework, by separating resource management (YARN) from job execution (MapReduce) has allowed multiple programming paradigms to take advantage of the massive scale Hadoop Distributed File System (HDFS) clusters. Hamster (Hadoop And Mpi on the same cluSTER) is a port of OpenMPI to use YARN as a resource manager. Hamster allows applications written using MPI (Message Passing Interface) to run alongside other YARN applications and frameworks, such as MapReduce, on the same Hadoop cluster. In this talk, I will describe the architecture of Hamster, and present a few MPI applications that have been demonstrated to run in Hadoop. GraphLab uses MPI as one of the supported communication libraries, and can read/write data from/to HDFS. I will describe how GraphLab runs on top of Hadoop using Hamster, and present a few benchmarks in graph analytics, comparing GraphLab with other machine frameworks.
Apache Hadoop project, and the Hadoop ecosystem has been designed be extremely flexible, and extensible. HDFS, Yarn, and MapReduce combined have more that 1000 configuration parameters that allow users to tune performance of Hadoop applications, and more importantly, extend Hadoop with application-specific functionality, without having to modify any of the core Hadoop code.
In this talk, I will start with simple extensions, such as writing a new InputFormat to efficiently process video files. I will provide with some extensions that boost application performance, such as optimized compression codecs, and pluggable shuffle implementations. With refactoring of MapReduce framework, and emergence of YARN, as a generic resource manager for Hadoop, one can extend Hadoop further by implementing new computation paradigms.
I will discuss one such computation framework, that allows Message Passing applications to run in the Hadoop cluster alongside MapReduce. I will conclude by outlining some of our ongoing work, that extends HDFS, by removing namespace limitations of the current Namenode implementation.
In KDD2011, Vijay Narayanan (Yahoo!) and Milind Bhandarkar (Greenplum Labs, EMC) conducted a tutorial on "Modeling with Hadoop". This is the second half of the tutorial.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
5. About Me
• Lead Yahoo! Grid Solutions Team since June
2005
• Contributor to Hadoop since January 2006
• Trained 1000+ Hadoop users at Yahoo! &
elsewhere
• 20+ years of experience in Parallel
Programming
Middleware 2009 5
7. Hadoop At Yahoo!
(Some Statistics)
• 25,000 + machines in 10+ clusters
• Largest cluster is 3,000 machines
• 3 Petabytes of data (compressed,
unreplicated)
• 700+ users
• 10,000+ jobs/week
Middleware 2009 7
8. Sample Applications
• Data analysis is the inner loop of Web 2.0
• Data ⇒ Information ⇒ Value
• Log processing: reporting, buzz
• Search index
• Machine learning: Spam filters
• Competitive intelligence
Middleware 2009 8
9. Prominent Hadoop
Users
• Yahoo! • Quantcast
• A9.com • Joost
• EHarmony • Last.fm
• Facebook • Powerset
• Fox Interactive Media • New York Times
• IBM • Rackspace
9
11. Search Assist
• Insight: Related concepts appear close
together in text corpus
• Input: Web pages
• 1 Billion Pages, 10K bytes each
• 10 TB of input data
• Output: List(word, List(related words))
Middleware 2009 11
12. Search Assist
// Input: List(URL, Text)
foreach URL in Input :
Words = Tokenize(Text(URL));
foreach word in Tokens :
Insert (word, Next(word, Tokens)) in Pairs;
Insert (word, Previous(word, Tokens)) in Pairs;
// Result: Pairs = List (word, RelatedWord)
Group Pairs by word;
// Result: List (word, List(RelatedWords)
foreach word in Pairs :
Count RelatedWords in GroupedPairs;
// Result: List (word, List(RelatedWords, count))
foreach word in CountedPairs :
Sort Pairs(word, *) descending by count;
choose Top 5 Pairs;
// Result: List (word, Top5(RelatedWords))
12
14. You Might Also Know
• Insight:You might also know Joe Smith if a
lot of folks you know, know Joe Smith
• if you don’t know Joe Smith already
• Numbers:
• 300 MM users
• Average connections per user is 100
Middleware 2009 14
15. You Might Also Know
// Input: List(UserName, List(Connections))
foreach u in UserList : // 300 MM
foreach x in Connections(u) : // 100
foreach y in Connections(x) : // 100
if (y not in Connections(u)) :
Count(u, y)++; // 3 Trillion Iterations
Sort (u,y) in descending order of Count(u,y);
Choose Top 3 y;
Store (u, {y0, y1, y2}) for serving;
15
16. Performance
• 101 Random accesses for each user
• Assume 1 ms per random access
• 100 ms per user
• 300 MM users
• 300 days on a single machine
Middleware 2009 16
21. Parallelism
• Map is inherently parallel
• Each list element processed
independently
• Reduce is inherently sequential
• Unless processing multiple lists
• Grouping to produce multiple lists
Middleware 2009 21
24. Issues with Large Data
• Map Parallelism: Splitting input data
• Shipping input data
• Reduce Parallelism:
• Grouping related data
• Dealing with failures
• Load imbalance
Middleware 2009 24
25.
26. Apache Hadoop
• January 2006: Subproject of Lucene
• January 2008: Top-level Apache project
• Latest Version: 0.21
• Stable Version: 0.20.x
• Major contributors:Yahoo!, Facebook,
Powerset
Middleware 2009 26
28. Problem: Bandwidth to
Data
• Scan 100TB Datasets on 1000 node cluster
• Remote storage @ 10MB/s = 165 mins
• Local storage @ 50-200MB/s = 33-8 mins
• Moving computation is more efficient than
moving data
• Need visibility into data placement
Middleware 2009 28
29. Problem: Scaling
Reliably
• Failure is not an option, it’s a rule !
• 1000 nodes, MTBF < 1 day
• 4000 disks, 8000 cores, 25 switches, 1000
NICs, 2000 DIMMS (16TB RAM)
• Need fault tolerant store with reasonable
availability guarantees
• Handle hardware faults transparently
Middleware 2009 29
30. Hadoop Goals
• Scalable: Petabytes (1015 Bytes) of data on
thousands on nodes
• Economical: Commodity components only
• Reliable
• Engineering reliability into every
application is expensive
Middleware 2009 30
32. HDFS
• Data is organized into files and directories
• Files are divided into uniform sized blocks
(default 64MB) and distributed across
cluster nodes
• HDFS exposes block placement so that
computation can be migrated to data
Middleware 2009 32
33. HDFS
• Blocks are replicated (default 3) to handle
hardware failure
• Replication for performance and fault
tolerance (Rack-Aware placement)
• HDFS keeps checksums of data for
corruption detection and recovery
Middleware 2009 33
34. HDFS
• Master-Worker Architecture
• Single NameNode
• Many (Thousands) DataNodes
Middleware 2009 34
36. Namenode
• Mapping of datanode to list of blocks
• Monitor datanode health
• Replicate missing blocks
• Keeps ALL namespace in memory
• 60M objects (File/Block) in 16GB
Middleware 2009 36
37. Datanodes
• Handle block storage on multiple volumes
& block integrity
• Clients access the blocks directly from data
nodes
• Periodically send heartbeats and block
reports to Namenode
• Blocks are stored as underlying OS’s files
Middleware 2009 37
41. HDFS Java API
// Get default file system instance
fs = Filesystem.get(new Configuration());
// Or Get file system instance from URI
fs = Filesystem.get(URI.create(uri),
new Configuration());
// Create, open, list, …
OutputStream out = fs.create(path, …);
InputStream in = fs.open(path, …);
boolean isDone = fs.delete(path, recursive);
FileStatus[] fstat = fs.listStatus(path);
41
60. Example: Unigrams
• Input: Huge text corpus
• Wikipedia Articles (40GB uncompressed)
• Output: List of words sorted in descending
order of frequency
Middleware 2009 60
62. MR for Unigrams
mapper (filename, file-contents):
for each word in file-contents:
emit (word, 1)
reducer (word, values):
sum = 0
for each value in values:
sum = sum + value
emit (word, sum)
62
63. MR for Unigrams
mapper (word, frequency):
emit (frequency, word)
reducer (frequency, words):
for each word in words:
emit (word, frequency)
63
66. Unigrams: Java Mapper
public static class MapClass extends MapReduceBase
implements Mapper
<LongWritable, Text, Text, IntWritable> {
public void map(LongWritable key, Text value,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer itr = new StringTokenizer(line);
while (itr.hasMoreTokens()) {
Text word = new Text(itr.nextToken());
output.collect(word, new IntWritable(1));
}
}
}
66
67. Unigrams: Java Reducer
public static class Reduce extends MapReduceBase
implements Reducer
<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key,
Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
67
68. Unigrams: Driver
public void run(String inputPath, String outputPath)
throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");
conf.setMapperClass(MapClass.class);
conf.setReducerClass(Reduce.class);
FileInputFormat.addInputPath(conf,
new Path(inputPath));
FileOutputFormat.setOutputPath(conf,
new Path(outputPath));
JobClient.runJob(conf);
}
68
73. InputFormats
Format Key Type Value Type
TextInputFormat
File Offset Text Line
(Default)
KeyValueInputFormat Text (upto t) Remaining Text
SequenceFileInputFormat User-Defined User-Defined
74. OutputFormats
Format Description
TextOutputFormat
Key t Value n
(default)
Binary Serialized keys and
SequenceFileOutputFormat
values
NullOutputFormat Discards Output
75. Hadoop Streaming
• Hadoop is written in Java
• Java MapReduce code is “native”
• What about Non-Java Programmers ?
• Perl, Python, Shell, R
• grep, sed, awk, uniq as Mappers/Reducers
• Text Input and Output
Middleware 2009 75
76. Hadoop Streaming
• Thin Java wrappers for Map & Reduce Tasks
• Forks actual Mapper & Reducer
• IPC via stdin, stdout, stderr
• Key.toString() t Value.toString() n
• Slower than Java programs
• Allows for quick prototyping / debugging
Middleware 2009 76
106. Debugging
• Run job with the Local Runner
• Set mapred.job.tracker to “local”
• Runs application in a single thread
• Run job on a small data set on a 1 node
cluster
Middleware 2009 106
107. Debugging
• Set keep.failed.task.files to keep files from
failed tasks
• Use the IsolationRunner to run just the
failed task
• Java Debugging hints
• Send a kill -QUIT to the Java process to
get the call stack, locks held, deadlocks
Middleware 2009 107
109. Example
• “Bob” wants to count records in AdServer
logs (several hundred GB)
• Used Identity Mapper & Single counting
reducer
• What is he doing wrong ?
• This happened, really !
Middleware 2009 109
110. MapReduce
Performance
• Reduce intermediate data size
• map outputs + reduce inputs
• Maximize map input transfer rate
• Pipelined writes from reduce
• Opportunity to load balance
Middleware 2009 110
111. Shuffle
• Often the most expensive component
• M * R Transfers over the network
• Sort map outputs (intermediate data)
• Merge reduce inputs
Middleware 2009 111
112. Improving Shuffle
• Avoid shuffling/sorting if possible
• Minimize redundant transfers
• Compress intermediate data
Middleware 2009 112
113. Avoid Shuffle
• Set mapred.reduce.tasks to zero
• Known as map-only computations
• Filters, Projections, Transformations
• Number of output files = number of input
splits = number of input blocks
• May overwhelm namenode
Middleware 2009 113
115. Combiners
• When Maps produce many repeated keys
• Combiner: Local aggregation after Map &
before Reduce
• Side-effect free
• Same interface as Reducers, and often the
same class
Middleware 2009 115
116. Compression
• Often yields huge performance gains
• Set mapred.output.compress to true to
compress job output
• Set mapred.compress.map.output to true to
compress map outputs
• Codecs: Java zlib (default), LZO, bzip2,
native gzip
Middleware 2009 116
117. Load Imbalance
• Inherent in application
• Imbalance in input splits
• Imbalance in computations
• Imbalance in partitions
• Heterogenous hardware
• Degradation over time
Middleware 2009 117
118. Optimal Number of
Nodes
• T = Map slots per TaskTracker
m
• N = optimal number of nodes
• S = N * T = Total Map slots in cluster
m m
• M = Map tasks in application
• Rule of thumb: 5*S < M < 10*S
m m
Middleware 2009 118
119. Configuring Task Slots
• mapred.tasktracker.map.tasks.maximum
• mapred.tasktracker.reduce.tasks.maximum
• Tradeoffs: Number of cores, RAM, number
and size of disks
• Also consider resources consumed by
TaskTracker & DataNode
Middleware 2009 119
120. Speculative Execution
• Runs multiple instances of slow tasks
• Instance that finishes first, succeeds
• mapred.map.speculative.execution=true
• mapred.reduce.speculative.execution=true
• Can dramatically bring in long tails on jobs
Middleware 2009 120
129. Bigrams
• Input: A large text corpus
• Output: List(word , Top (word ))
1 K 2
• Two Stages:
• Generate all possible bigrams
• Find most frequent K bigrams for each
word
Middleware 2009 129
130. Bigrams: Stage 1
Map
• Generate all possible Bigrams
• Map Input: Large text corpus
• Map computation
• In each sentence, or each “word word ”
1 2
• Output (word , word ), (word , word )
1 2 2 1
• Partition & Sort by (word , word )
1 2
Middleware 2009 130
135. Bigrams: Stage 2
Reduce
• Input: List(word , [freq,word ])
1 2
• partitioned by word 1
• sorted descending by (word , freq) 1
• Output: Top (List(word , [freq, word ]))
K 1 2
• For each word, throw away after K records
Middleware 2009 135
137. Partitioner
• By default, evenly distributes keys
• hashcode(key) % NumReducers
• Overriding partitioner
• Skew in map-outputs
• Restrictions on reduce outputs
• All URLs in a domain together
Middleware 2009 137
139. Fully Sorted Output
• By contract, reducer gets input sorted on
key
• Typically reducer output order is the same
as input order
• Each output file (part file) is sorted
• How to make sure that Keys in part i are all
less than keys in part i+1 ?
Middleware 2009 139
140. Fully Sorted Output
• Use single reducer for small output
• Insight: Reducer input must be fully sorted
• Partitioner should provide fully sorted
reduce input
• Sampling + Histogram equalization
Middleware 2009 140
141. Number of Maps
• Number of Input Splits
• Number of HDFS blocks
• mapred.map.tasks
• Minimum Split Size (mapred.min.split.size)
• split_size = max(min(hdfs_block_size,
data_size/#maps), min_split_size)
Middleware 2009 141
142. Parameter Sweeps
• External program processes data based on
command-line parameters
• ./prog –params=“0.1,0.3” < in.dat > out.dat
• Objective: Run an instance of ./prog for each
parameter combination
• Number of Mappers = Number of different
parameter combinations
Middleware 2009 142
143. Parameter Sweeps
• Input File: params.txt
• Each line contains one combination of
parameters
• Input format is NLineInputFormat (N=1)
• Number of maps = Number of splits =
Number of lines in params.txt
Middleware 2009 143
144. Auxiliary Files
• -file auxFile.dat
• Job submitter adds file to job.jar
• Unjarred on the task tracker
• Available to task as $cwd/auxFile.dat
• Not suitable for large / frequently used files
Middleware 2009 144
145. Auxiliary Files
• Tasks need to access “side” files
• Read-only Dictionaries (such as for porn
filtering)
• Dynamically linked libraries
• Tasks themselves can fetch files from HDFS
• Not Always ! (Hint: Unresolved symbols)
Middleware 2009 145
146. Distributed Cache
• Specify “side” files via –cacheFile
• If lot of such files needed
• Create a tar.gz archive
• Upload to HDFS
• Specify via –cacheArchive
Middleware 2009 146
147. Distributed Cache
• TaskTracker downloads these files “once”
• Untars archives
• Accessible in task’s $cwd before task starts
• Cached across multiple tasks
• Cleaned up upon exit
Middleware 2009 147
148. Joining Multiple
Datasets
• Datasets are streams of key-value pairs
• Could be split across multiple files in a
single directory
• Join could be on Key, or any field in Value
• Join could be inner, outer, left outer, cross
product etc
• Join is a natural Reduce operation
Middleware 2009 148
149. Example
• A = (id, name), B = (name, address)
• A is in /path/to/A/part-*
• B is in /path/to/B/part-*
• Select A.name, B.address where A.name ==
B.name
Middleware 2009 149
150. Map in Join
• Input: (Key ,Value ) from A or B
1 1
• map.input.file indicates A or B
• MAP_INPUT_FILE in Streaming
• Output: (Key , [Value , A|B])
2 2
• Key is the Join Key
2
Middleware 2009 150
151. Reduce in Join
• Input: Groups of [Value , A|B] for each Key
2 2
• Operation depends on which kind of join
• Inner join checks if key has values from
both A & B
• Output: (Key , JoinFunction(Value ,…))
2 2
Middleware 2009 151
152. MR Join Performance
• Map Input = Total of A & B
• Map output = Total of A & B
• Shuffle & Sort
• Reduce input = Total of A & B
• Reduce output = Size of Joined dataset
• Filter and Project in Map
Middleware 2009 152
153. Join Special Cases
• Fragment-Replicate
• 100GB dataset with 100 MB dataset
• Equipartitioned Datasets
• Identically Keyed
• Equal Number of partitions
• Each partition locally sorted
Middleware 2009 153
154. Fragment-Replicate
• Fragment larger dataset
• Specify as Map input
• Replicate smaller dataset
• Use Distributed Cache
• Map-Only computation
• No shuffle / sort
Middleware 2009 154
155. Equipartitioned Join
• Available since Hadoop 0.16
• Datasets joined “before” input to mappers
• Input format: CompositeInputFormat
• mapred.join.expr
• Simpler to use in Java, but can be used in
Streaming
Middleware 2009 155
159. What is Pig?
• System for processing large semi-
structured data sets using Hadoop
MapReduce platform
• Pig Latin: High-level procedural language
• Pig Engine: Parser, Optimizer and
distributed query execution
Middleware 2009 159
160. Pig vs SQL
• Pig is procedural • SQL is declarative
• Nested relational data • Flat relational data
model model
• Schema is optional • Schema is required
• Scan-centric analytic • OLTP + OLAP
workloads workloads
• Limited query • Significant opportunity
optimization for query optimization
160
161. Pig vs Hadoop
• Increases programmer productivity
• Decreases duplication of effort
• Insulates against Hadoop complexity
• Version Upgrades
• JobConf configuration tuning
• Job Chains
Middleware 2009 161
162. Example
• Input: User profiles, Page
visits
• Find the top 5 most
visited pages by users
aged 18-25
164. In Pig
Users = load ‘users’ as (name, age);
Filtered = filter Users by age >= 18 and age <= 25;
Pages = load ‘pages’ as (user, url);
Joined = join Filtered by name, Pages by user;
Grouped = group Joined by url;
Summed = foreach Grouped generate group,
COUNT(Joined) as clicks;
Sorted = order Summed by clicks desc;
Top5 = limit Sorted 5;
store Top5 into ‘top5sites’;
164
167. Flexibility & Control
• Easy to plug-in user code
• Metadata is not mandatory
• Does not impose a data model
• Fine grained control
• Complex data types
Middleware 2009 167
168. Pig Data Types
• Tuple: Ordered set of fields
• Field can be simple or complex type
• Nested relational model
• Bag: Collection of tuples
• Can contain duplicates
• Map: Set of (key, value) pairs
Middleware 2009 168
170. Expressions
A = LOAD ‘data.txt’ AS
(f1:int , f2:{t:(n1:int, n2:int)}, f3: map[] )
A =
{
( 1, -- A.f1 or A.$0
{ (2, 3), (4, 6) }, -- A.f2 or A.$1
[ ‘yahoo’#’mail’ ] -- A.f3 or A.$2
)
}
170
171. Pig Unigrams
• Input: Large text document
• Process:
• Load the file
• For each line, generate word tokens
• Group by word
• Count words in each group
Middleware 2009 171
176. Store
store counts into ‘/user/milindb/output’
using PigStorage();
pig 5
latin 3
hadoop 1
program 3
176
177. Example: Log
Processing
-- use a custom loader
Logs = load ‘/var/log/access_log’ using
CommonLogLoader() as (addr, logname,
user, time, method, uri, p, bytes);
-- apply your own function
Cleaned = foreach Logs generate addr,
canonicalize(url) as url;
Grouped = group Cleaned by url;
-- run the result through a binary
Analyzed = stream Grouped through
‘urlanalyzer.py’;
store Analyzed into ‘analyzedurls’;
177
178. Schema on the fly
-- declare your types
Grades = load ‘studentgrades’ as
(name: chararray, age: int,
gpa: double);
Good = filter Grades by age > 18
and gpa > 3.0;
-- ordering will be by type
Sorted = order Good by gpa;
store Sorted into ‘smartgrownups’;
178
179. Nested Data
Logs = load ‘weblogs’ as (url, userid);
Grouped = group Logs by url;
-- Code inside {} will be applied to each
-- value in turn.
DisinctCount = foreach Grouped {
Userid = Logs.userid;
DistinctUsers = distinct Userid;
generate group, COUNT(DistinctUsers);
}
store DistinctCount into ‘distinctcount’;
179
182. Logical Plan
• Directed Acyclic Graph
• Logical Operator as Node
• Data flow as edges
• Logical Operators
• One per Pig statement
• Type checking with Schema
Middleware 2009 182
183. Pig Statements
Read data from the file
Load
system
Write data to the file
Store
system
Dump Write data to stdout
184. Pig Statements
Apply expression to
each record and
Foreach..Generate
generate one or more
records
Apply predicate to each
Filter record and remove
records where false
Stream records through
Stream..through
user-provided binary
185. Pig Statements
Collect records with
Group/CoGroup the same key from one
or more inputs
Join two or more inputs
Join
based on a key
Sort records based on a
Order..by
key
186. Physical Plan
• Pig supports two back-ends
• Local
• Hadoop MapReduce
• 1:1 correspondence with most logical
operators
• Except Distinct, Group, Cogroup, Join etc
Middleware 2009 186
187. MapReduce Plan
• Detect Map-Reduce boundaries
• Group, Cogroup, Order, Distinct
• Coalesce operators into Map and Reduce
stages
• Job.jar is created and submitted to Hadoop
JobControl
Middleware 2009 187
195. SQL to Pig
SQL Pig
A = LOAD ‘MyTable’ USING PigStorage(‘t’) AS
...FROM MyTable...
(col1:int, col2:int, col3:int);
SELECT col1 +
B = FOREACH A GENERATE col1 + col2, col3;
col2, col3 ...
...WHERE col2 > 2 C = FILTER B by col2 > 2;
196. SQL to Pig
SQL Pig
D = GROUP A BY (col1, col2)
SELECT col1, col2, sum(col3)
E = FOREACH D GENERATE
FROM X GROUP BY col1, col2
FLATTEN(group), SUM(A.col3);
...HAVING sum(col3) > 5 F = FILTER E BY $2 > 5;
...ORDER BY col1 G = ORDER F BY $0;
197. SQL to Pig
SQL Pig
SELECT DISTINCT col1 I = FOREACH A GENERATE col1;
from X J = DISTINCT I;
K = GROUP A BY col1;
SELECT col1, L = FOREACH K {
count(DISTINCT col2) M = DISTINCT A.col2;
FROM X GROUP BY col1 GENERATE FLATTEN(group), count(M);
}
198. SQL to Pig
SQL Pig
N = JOIN A by col1 INNER, B by col1 INNER;
O = FOREACH N GENERATE A.col1, B.col3;
SELECT A.col1, B. -- Or
col3 FROM A JOIN B
USING (col1) N = COGROUP A by col1 INNER, B by col1 INNER;
O = FOREACH N GENERATE flatten(A), flatten(B);
P = FOREACH O GENERATE A.col1, B.col3