Download to read offline
Uploaded byMansi Mehra
Hadoop - A big data initiative
This document discusses big data solutions and introduces Hadoop. It defines common big data problems related to volume, velocity, and variety of data. Traditional storage does not work well for this type of unstructured data. Hadoop provides solutions through HDFS for storage, MapReduce for processing, and additional tools like HBase, Pig, Hive, Zookeeper, and Spark to handle different data and analytic needs. Each tool is described briefly in terms of its purpose and how it works with Hadoop.
More Related Content
Similar to Hadoop - A big data initiative
![제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계](https://cdn.slidesharecdn.com/ss_thumbnails/3-1boaz23rdconferencemboax-260203102709-9d519923-thumbnail.jpg?width=640&height=640&fit=bounds)
Hadoop - A big data initiative
- 1. HADOOP (A BIGDATA INITIATIVE) -Mansi Mehra
- 2. AGENDA What isthe problem? What is the solution? HDFS MapReduce Hbase PIG HIVE Zookeeper Spark
- 3. DEFINING THE PROBLEM– 3V Volume - Lots and lots of data Datasets are so large and complex Cannot use relational database Challenges: capture, curation, storage, search, sharing, transfer, analysis and visualization.
- 4. DEFINING THE PROBLEM– 3V (CONTD.) Velocity - Huge amounts of data generated at incredible speed NYSE generates about 1 TB of new trade data per day AT&T anonymized Call Detail Records (CDRs) top at around 1 GB per hour. Variety - Differently formatted data sets from different sources Twitter keeps tracks of tweets, Facebook produces posts and likes data, Youtube streams videos)
- 5. WHY TRADITIONAL STORAGESDON’T WORK Unstructured data is exploding, not much of data produced has relational nature. No redundancy High computational cost Capacity limit for structured data (costly hardware) Expensive License Data type Nature XML Semi-structured Word docs, PDF files etc. Unstructured Email body Unstructured Data from Enterprise Systems (ERP, CRM etc.) Structured
- 6. HOW DOES HADOOPWORK?
- 7. HDFS (HADOOP 1.0VS 2.0)
- 8.
- 9. YARN (2.0)- YETANOTHER RESOURCE NEGOTIATOR Computing framework for Hadoop. YARN has Resource Manager- Manages and allocates cluster resources Improves performance and Quality of Service
- 10. MAP REDUCE Programmingmodel in Java Work on large amounts of data Provides redundancy & fault tolerance Runs the code on each data node
- 11. MAP REDUCE (CONTD.) Steps for Map Reduce: Read in lots of data Map: extract something you care about from each record/line. Shuffle and sort Reduce: aggregate, summarize, filter or transform Write results.
- 12.
- 13. HIVE (OVERVIEW) Adata warehouse infrastructure built on top of Hadoop Compile SQL queries as MapReduce jobs and run the job in the cluster. Brings structure to unstructured data Key Building Principles: Structured data with rich data types (structs, lists and maps) Directly query data from different formats (text/binary) and file formats (Flat/sequence). SQL as a familiar programming tool and for standard analytics Types of applications: Summarization: Daily/weekly aggregations Ad hoc analysis Data Mining Spam detection Many more ….
- 14. PIG (OVERVIEW) Highlevel dataflow language Has its own syntax (Preferable for people with programming background) Compiler that produces sequences of MapReduce programs. Structure is agreeable to substantial parallelization. Key properties of PIG: Ease of programming: Trivial to achieve parallel execution of simple and parallel data analysis tasks Optimization opportunities: allows user to focus on semantics rather than efficiency. Extensibility: Users can create their own functions to do special purpose processing.
- 15. HBASE (OVERVIEW) HBaseis a distributed column-oriented data store built on top of HDFS. Data is logically organized into tables, rows and columns. HDFS is good for batch processing (scan over big files). Not good for record lookup. Not good for incremental addition of small batches. Not good for updates. HBase is designed to efficiently address the above points Fast record lookup Support for record level insertion Support for updates (not in place). Updates are done by creating new versions of values.
- 16. ZOOKEEPER (OVERVIEW) Zookeeperis a distributed, open source coordination service for distributed applications. Exposes simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. Coordination services are notoriously hard to get right. They are prone to errors like race conditions and deadlock. The motivation behind zookeeper is to relieve distributed applications the responsibility of implementing coordination services from scratch.
- 17. SPARK (OVERVIEW) Motivation: MapReduce programming model transform data flowing from stable storage to stable storage (disk to disk). Acyclic data flow is a powerful abstraction, but not efficient for applications that repeatedly reuse a working set of data. Iterative algorithms Interactive data mining Spark makes working sets a first-class concept to efficiently support these applications. Goal: To provide distributed memory abstractions for clusters to support apps with working sets. Retain the attractive properties of map reduce. Fault tolerance Data locality Scalability Augment data flow model with “resilient distributed datasets” (RDDs)
- 18. SPARK (OVERVIEW CONTD.) Resilient distributed datasets (RDDs) Immutable collections partitioned across cluster that can be rebuilt if a partition is lost. Created by transforming data in stable storage using data flow operators (map, filter, group-by, ..) Can be cached across parallel operations. Parallel operations on RDDs. Reduce, collect, count, save, ….. Restricted shared variables Accumulators, broadcast variables.
- 19. SPARK (OVERVIEW CONTD) Fast map reduce like engine Uses in memory cluster computing Compatible with Hadoop storage API. Has API’s written in Scala, Java, Python. Useful for large datasets and iterative algorithms. Up to 40x faster than MapReduce. Support for: Spark SQL : Hive on Spark Mlib : Machine learning library Graphx : Graph processing.
Editor's Notes
- #7 Founder: Doug Cutting. He named it after his son’s toy elephant
- #8 Active Namenode: In order to provide HDFS high availability, we have an active and standby NameNode in the architecture now. Namenode keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data itself, just reference meta data. Client applications talk to the namenode whenever they wish to locate a file, or add/copy/move/delete a file. The NameNode responds the successful requests by returning a list of relevant Datanode servers where the data lives. It is essential to look after the NameNode. Here are some recommendations from production use Use a good server with lots of RAM. The more RAM you have, the bigger the file system, or the smaller the block size. Use ECC RAM. On Java6u15 or later, run the server VM with compressed pointers -XX:+UseCompressedOops to cut the JVM heap size down. List more than one name node directory in the configuration, so that multiple copies of the file system meta-data will be stored. As long as the directories are on separate disks, a single disk failure will not corrupt the meta-data. Configure the NameNode to store one set of transaction logs on a separate disk from the image. Configure the NameNode to store another set of transaction logs to a network mounted disk. Monitor the disk space available to the NameNode. If free space is getting low, add more storage. Do not host DataNode, JobTracker or TaskTracker services on the same system. DataNodes: A datanode stores data in the HDFS. A functional file system has more than one Datanode, with data replicated across them. On startup, the datanode connects to the Namenode, spinning until that service comes up. It then responds to requests from the NameNode for filesystem operations. Client applications can talk directly to a DataNode, once the NameNode has provided the location of the data. Similarly, MapReduce operations farmed out to TaskTracker instances near a DataNode, talk directly to the DataNode to access the files. TaskTracker instances can, indeed should, be deployed on the same servers that host Datanode instances, so that MapReduce operations are performed close to the data. DataNode instances can talk to each other, which is what they do when they are replicating data. There is usually no need to use RAID storage for DataNode data, because data is designed to be replicated across multiple servers, rather than multiple disks on the same server. An ideal configuration is for a server to have a DataNode, a TaskTracker, and then physical disks one TaskTracker slot per CPU. This will allow every TaskTracker 100% of a CPU, and separate disks to read and write data. Avoid using NFS for data storage in production system. Node Manager: The NM is YARN’s per-node agent, and takes care of the individual compute nodes in a Hadoop cluster. This includes keeping up-to-date with Resource Manager(RM), overseeing containers’ life-cycle management, monitoring resource usage (memory, CPU) of individual containers, tracking node-health, log’s management and auxiliary services which may be exploited by different YARN applications. Resource Manager: RM is the master that arbitrates all the available cluster resources and thus helps manage the distributed applications running on the YARN system. It works together with the per-node NMs and the per-application ApplicationMasters (Ams). Application Masters: are responsible for negotiating resources with the ResourceManager and for working with the Node Managers to start the containers.
- #9 Active Namenode: In order to provide HDFS high availability, we have an active and standby NameNode in the architecture now. Namenode keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data itself, just reference meta data. Client applications talk to the namenode whenever they wish to locate a file, or add/copy/move/delete a file. The NameNode responds the successful requests by returning a list of relevant Datanode servers where the data lives. It is essential to look after the NameNode. Here are some recommendations from production use Use a good server with lots of RAM. The more RAM you have, the bigger the file system, or the smaller the block size. Use ECC RAM. On Java6u15 or later, run the server VM with compressed pointers -XX:+UseCompressedOops to cut the JVM heap size down. List more than one name node directory in the configuration, so that multiple copies of the file system meta-data will be stored. As long as the directories are on separate disks, a single disk failure will not corrupt the meta-data. Configure the NameNode to store one set of transaction logs on a separate disk from the image. Configure the NameNode to store another set of transaction logs to a network mounted disk. Monitor the disk space available to the NameNode. If free space is getting low, add more storage. Do not host DataNode, JobTracker or TaskTracker services on the same system. DataNodes: A datanode stores data in the HDFS. A functional file system has more than one Datanode, with data replicated across them. On startup, the datanode connects to the Namenode, spinning until that service comes up. It then responds to requests from the NameNode for filesystem operations. Client applications can talk directly to a DataNode, once the NameNode has provided the location of the data. Similarly, MapReduce operations farmed out to TaskTracker instances near a DataNode, talk directly to the DataNode to access the files. TaskTracker instances can, indeed should, be deployed on the same servers that host Datanode instances, so that MapReduce operations are performed close to the data. DataNode instances can talk to each other, which is what they do when they are replicating data. There is usually no need to use RAID storage for DataNode data, because data is designed to be replicated across multiple servers, rather than multiple disks on the same server. An ideal configuration is for a server to have a DataNode, a TaskTracker, and then physical disks one TaskTracker slot per CPU. This will allow every TaskTracker 100% of a CPU, and separate disks to read and write data. Avoid using NFS for data storage in production system. Node Manager: The NM is YARN’s per-node agent, and takes care of the individual compute nodes in a Hadoop cluster. This includes keeping up-to-date with Resource Manager(RM), overseeing containers’ life-cycle management, monitoring resource usage (memory, CPU) of individual containers, tracking node-health, log’s management and auxiliary services which may be exploited by different YARN applications. Resource Manager: RM is the master that arbitrates all the available cluster resources and thus helps manage the distributed applications running on the YARN system. It works together with the per-node NMs and the per-application ApplicationMasters (Ams). Application Masters: are responsible for negotiating resources with the ResourceManager and for working with the Node Managers to start the containers.
- #13 Active Namenode: In order to provide HDFS high availability, we have an active and standby NameNode in the architecture now. Namenode keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data itself, just reference meta data. Client applications talk to the namenode whenever they wish to locate a file, or add/copy/move/delete a file. The NameNode responds the successful requests by returning a list of relevant Datanode servers where the data lives. It is essential to look after the NameNode. Here are some recommendations from production use Use a good server with lots of RAM. The more RAM you have, the bigger the file system, or the smaller the block size. Use ECC RAM. On Java6u15 or later, run the server VM with compressed pointers -XX:+UseCompressedOops to cut the JVM heap size down. List more than one name node directory in the configuration, so that multiple copies of the file system meta-data will be stored. As long as the directories are on separate disks, a single disk failure will not corrupt the meta-data. Configure the NameNode to store one set of transaction logs on a separate disk from the image. Configure the NameNode to store another set of transaction logs to a network mounted disk. Monitor the disk space available to the NameNode. If free space is getting low, add more storage. Do not host DataNode, JobTracker or TaskTracker services on the same system. DataNodes: A datanode stores data in the HDFS. A functional file system has more than one Datanode, with data replicated across them. On startup, the datanode connects to the Namenode, spinning until that service comes up. It then responds to requests from the NameNode for filesystem operations. Client applications can talk directly to a DataNode, once the NameNode has provided the location of the data. Similarly, MapReduce operations farmed out to TaskTracker instances near a DataNode, talk directly to the DataNode to access the files. TaskTracker instances can, indeed should, be deployed on the same servers that host Datanode instances, so that MapReduce operations are performed close to the data. DataNode instances can talk to each other, which is what they do when they are replicating data. There is usually no need to use RAID storage for DataNode data, because data is designed to be replicated across multiple servers, rather than multiple disks on the same server. An ideal configuration is for a server to have a DataNode, a TaskTracker, and then physical disks one TaskTracker slot per CPU. This will allow every TaskTracker 100% of a CPU, and separate disks to read and write data. Avoid using NFS for data storage in production system. Node Manager: The NM is YARN’s per-node agent, and takes care of the individual compute nodes in a Hadoop cluster. This includes keeping up-to-date with Resource Manager(RM), overseeing containers’ life-cycle management, monitoring resource usage (memory, CPU) of individual containers, tracking node-health, log’s management and auxiliary services which may be exploited by different YARN applications. Resource Manager: RM is the master that arbitrates all the available cluster resources and thus helps manage the distributed applications running on the YARN system. It works together with the per-node NMs and the per-application ApplicationMasters (Ams). Application Masters: are responsible for negotiating resources with the ResourceManager and for working with the Node Managers to start the containers.