Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hadoop admin presentation demo | Introduction | Basics


Published on

Kerneltraining Provides Hadoop Administration Tutorial for Beginners and we also provide online training

Visit Us

Email Us:
Phone: 91 8099 77 6681

How we Teach?

1. This is an Online Course with Instructor led LIVE and Interactive Sessions.
2. The course contains Practical Work that involves Practical Hands-on, Lab Assignments, and real-world Case Studies. Candidate can conduct practical work at their own pace.
3. You will have access to 24×7 Technical Support. You can request for assistance for any problem you might face or for any clarifications you may require during the course.
4. At the end of the training, you will have to work on a real time project.
5. Course participants will get verifiable certificate after successful completion of the project work.
About the Course:
The course also covers Configuring, Deploying, and Maintaining a Hadoop Cluster. The Hadoop Admin training is focused on practical hands-on exercises and encourages open discussions of how people are using Hadoop in enterprises dealing with large data sets.

Published in: Education
  • Be the first to comment

Hadoop admin presentation demo | Introduction | Basics

  1. 1. Introduction to By Laxmi Edi M.Tech (Ph.D)
  2. 2. Topics What is Big Data? Limitations of the existing solutions Solving the problem with Hadoop Introduction to Hadoop HadoopEco-System Hadoop Core Components HDFS Architecture MapRedcueJob execution Anatomy of a File Write and Read
  3. 3. What Is Big Data? •Lots of Data (Terabytes or Petabytes) •Big data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. •Big data is the term for a collection of data sets so large and complexthat it becomes difficultto process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. •Systems / Enterprises generate huge amount of data from Terabytes to and even Petabytesof information.
  4. 4. NYSE generates about one terabyte of new trade data per day to Perform stock trading analytics to determine trends for optimal trades.
  5. 5. Where does Big Data come from? Now the next question would be from where this Big Data originates, what makes the Big Data? Basically the data coming from everywhere like • sensors used to gather climate information • posts to social media sites • digital pictures and videos • software logs, cameras • microphones • scans of government documents • GPS trails • purchase transaction records • cell phone GPS signals • traffic • and many more. All these together constitute Big Data.
  6. 6. Exploding Un-Structured Data
  7. 7. Big DataCharacteristics 1. Volume: BIG DATA depends upon how gigantic it is. It could amount to hundreds of terabytes or even petabytes of information. For instance, 15 terabytes of facebook posts or 400 billion annual medical records could mean Big Data! 2. Velocity:Velocity means the rate at which data is flowing in the companies. Big data requires fast processing. Time factor plays a very crucial role in several organizations. For instance, processing 2 million records at share market or evaluating results of lakhs of students applied for competitive exams could mean Big Data! 3. Variety: Big Data may not belong to a specific format. It could be in any form such as structured, unstructured, text, images, audio, video, log files, emails, simulations, 3D models, etc. New research shows that a substantial amount of an organization’s data is not numeric; however, such data is equally important for decision-making process. So, organizations need to think beyond stock records, documents, personnel files, finances, etc.
  8. 8. Question Map the following to corresponding data type: -XML Files -Word Docs, PDF files, Text files -E-Mail body -Data from Enterprise systems (ERP, CRM etc.)
  9. 9. Answer XML Files -> Semi-structured data Word Docs, PDF files, Text files -> Unstructured Data E-Mail body -> Unstructured Data Data from Enterprise systems (ERP, CRM etc.) -> Structured Data
  10. 10. Big Data Customer Scenarios Web and e-tailing Recommendation Engines Ad Targeting Search Quality Abuse and Click Fraud Detection Telecommunications Customer Churn Prevention Network Performance Optimization Calling Data Record (CDR) Analysis Analyzing Network to Predict Failure
  11. 11. Big Data Customer Scenarios Fraud Detection And Cyber Security Welfare schemes Justice Healthcare & Life Sciences Health information exchange Gene sequencing Serialization Healthcare service quality improvements Drug Safety
  12. 12. Big Data Customer Scenarios Banks and Financial services ModelingTrue Risk Threat Analysis Fraud Detection Trade Surveillance Credit Scoring And Analysis Retail Point of sales Transaction Analysis Customer Churn Analysis Sentiment Analysis
  13. 13. Why DFS
  14. 14. What Is Hadoop Apache Hadoopis a frameworkthat allows for the distributed processing of large data sets across clusters of commodity computers using a simple programming model. It is an Open-source Data Management with scale-out storage & distributed processing.
  15. 15. Why Hadoop? Key features – Why Hadoop? 1. Flexible 2. Scalable 3. Building more efficient data economy: 4. Robust Ecosystem 5. Hadoop is getting more “Real-Time”! 6. Cost Effective: 7. Upcoming Technologies using Hadoop: 8. Hadoop is getting Cloudy!
  16. 16. Question Hadoop is a framework that allows for the distributed processing of: -Small Data Sets -Large Data Sets
  17. 17. Answer Large Data Sets. It is also capable to process small data-sets however to experience the true power of Hadoop one needs to have data in TB’s because this where RDBMS takes hours and fails whereas Hadoop does the same in couple of minutes.
  18. 18. Hadoop Eco-System
  19. 19. Machine Learning with Mahout •Mahout is a data mining library. •It takes the most popular data mining algorithms for performing clustering, regression testing and statistical modeling and implements them using the Map Reduce model.
  20. 20. Hadoop Core Components Hadoopis a system for large scale data processing. It has two main components: HDFS –Hadoop Distributed File System(Storage) Distributed across “nodes” Natively redundant NameNodetracks locations. MapReduce(Processing) Splits a task across processors “near” the data & assembles results Self-Healing, High Bandwidth Clustered storage JobTrackermanages the TaskTrackers
  21. 21. Hadoop Core Components (Contd.)
  22. 22. HDFS Architecture HDFS has master/slave architecture. An HDFS cluster consists of a single Master Node, a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of Slave Nodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on.
  23. 23. Main Components Of HDFS NameNode: master of the system maintains and manages the blocks which are present on the DataNodes DataNodes: slaves which are deployed on each machine and provide the actual storage responsible for serving read and write requests for the clients
  24. 24. HDFS - Read Anatomy requests the block from first datanode on the list. It tries two times and if no response then it adds the datanode to "deadnodes" list. And requests block from next datanode on the list. 7-8. After usccessful read of all the blocks, "DFSClient" send the deadnodes list back to NN for it to take action. 1. Client request the document 2. NN, checks the permissions and sends back the list of blocks and datanodes list (including port number to talk) for each block. 3-6. "DFSClient" class on client- side picks up first block and
  25. 25. HDFS - Write Anatomy Client has to write directly to datanode. However each datanodes has to notify receipt of each block back to client and namenode. Also each datanode passes on the block to next datanode to write, that means client has to transmit block to only first datanode and rest of the block movement is handled inside the cluster. Here is the flow of data file create and write on HDFS. Create and Write of HDFS file •Creation and writing of a file is more complicated than the read of a HDFS file. •Here also NameNode(NN) never writes any data directly to DataNodes(DN). It, as per it's role, only manages the namespace and inodes.
  26. 26. THANK YOU for attending Demo of