Prashant Gupta, profile picture
Uploaded byPrashant Gupta
PPTX, PDF11,389 views

Hadoop File system (HDFS)

The document provides an overview of big data and its characteristics, explaining its challenges and the limitations of traditional data processing tools. It introduces Hadoop as a solution, detailing its components including HDFS (Hadoop Distributed File System) and MapReduce for large-scale data processing, while also discussing various use cases and customer scenarios across different industries. The architecture of Hadoop, including the role of names nodes and data nodes in managing and storing large datasets, is elaborated upon, showcasing its distributed nature and efficiency.

Embed presentation

Downloaded 485 times
Hadoop HDFS
 What is Big Data?  Limitations of the existing solutions  Solving the problem with Hadoop  Introduction to Hadoop  Hadoop Eco-System  Hadoop Core Components  HDFS Architecture  Anatomy of a File Write and Read Topics of the Day Slide 2
What is Big Data? Slide 3  Lots of Data (Terabytes or Petabytes)  Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.  The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. cloud tools statistics No SQL compression storage support database analyze information terabytes mobile processing Big Data
What is Big Data? Slide 4  Systems / Enterprises generate huge amount of data from Terabytes to and even Petabytes of information. NYSE generates about one terabyte of new trade data per day to perform stock trading analytics to determine trends for optimal trades.
Un – Structured Data is Exploding Slide 5
 IBM’s Definition – Big Data Characteristics http://www-01.ibm.com/software/data/bigdata/ Web logs Videos Images Audios Sensor Data Slide 6 Volume Velocity Variety IBM’s Definition
 Web and e-tailing  Recommendation Engines  Ad Targeting  Search Quality  Abuse and Click Fraud Detection  Telecommunications  Customer Churn Prevention  Network Performance Optimization  Calling Data Record (CDR) Analysis  Analyzing Network to Predict Failure Common Big Data Customer Scenarios Slide 7
 Government  Fraud Detection and Cyber Security  Welfare Schemes  Justice  Healthcare & Life Sciences  Health Information Exchange  Gene Sequencing  Serialization  Healthcare Service Quality Improvements  Drug Safety Common Big Data Customer Scenarios(Contd.) Slide 8
Common Big Data Customer Scenarios Contd.) Slide 9  Banks and Financial services  Modeling True Risk  Threat Analysis  Fraud Detection  Trade Surveillance  Credit Scoring and Analysis  Retail  Point of Sales Transaction Analysis  Customer Churn Analysis  Sentiment Analysis
Hidden Treasure Case Study: Sears Holding Corporation X *Sears was using traditional systems such as Oracle Exadata, Teradata and SAS etc. to store and process the customer activity and sales data. Slide 17  Insight into data can provide Business Advantage.  Some key early indicators can mean Fortunes to Business.  More Precise Analysis with more data.
Mostly Append Collection Instrumentation http://www.informationweek.com/it-leadership/why-sears-is-going-all-in-on-hadoop/d/d-id/1107038? 90% of the ~2PB Archived Storage Processing BI Reports + Interactive Apps RDBMS (Aggregated Data) ETL Compute Grid 3. Premature data death 1. Can’t explore original high fidelity raw data 2. Moving data to compute doesn’t scale A meagre 10% of the ~2PB Data is available for BI Storage only Grid (original Raw Data) Limitations of Existing Data Analytics Architecture Slide 11
*Sears moved to a 300-Node Hadoop cluster to keep 100% of its data available for processing rather than a meagre 10% as was the case with existing Non-Hadoop solutions. RDBMS (Aggregated Data) No Data Archiving 1. Data Exploration & Advanced analytics 2. Scalable throughput for ETL & aggregation 3. Keep data alive forever Mostly Append Collection Instrumentation BI Reports + Interactive Apps Hadoop : Storage + Compute Grid Entire ~2PB Data is available for processing Both Storage And Processing Solution: A Combined Storage Computer Layer Slide 12
Accessible RobustSimple Scalable Differentiating Factors Hadoop Differentiating Factors Slide 13
RDBMS EDW Hadoop – It’s about Scale and Structure Slide 14 MPP NoSQL HADOOP Structured Data Types Multi and Unstructured Limited, No Data Processing Processing Processing coupled with Data Standards & Structured Governance Loosely Structured Required On write Schema Required On Read Software License Cost Support Only Known Entity Resources Growing, Complexities, Wide Interactive OLAP Analytics Operational Data Store Best Fit Use Massive Storage/Processing
Read 1 TB Data 1 Machine 4 I/O Channels Each Channel – 100 MB/s 10 Machines 4 I/O Channels Each Channel – 100 MB/s Why DFS? Slide 15
Read 1 TB Data 10 Machines 4 I/O Channels Each Channel – 100 MB/s Why DFS? Slide 16 45 Minutes 1 Machine 4 I/O Channels Each Channel – 100 MB/s
Read 1 TB Data 10 Machines 4 I/O Channels Each Channel – 100 MB/s Why DFS? Slide 17 45 Minutes 4.5 Minutes 1 Machine 4 I/O Channels Each Channel – 100 MB/s
What is Hadoop? Slide 18  Apache Hadoop is a framework that allows for the distributed processing of large data sets across clusters of commodity computers using a simple programming model.  It is an Open-source Data Management with scale-out storage & distributed processing.
Apache Oozie (Workflow) MapReduce Framework HBase HDFS (Hadoop Distributed File System) Flume Sqoop Import Or Export Pig Latin Data Analysis Mahout Machine Learning Hive DW System Unstructured or Semi-Structured data Structured Data Hadoop Eco-System Slide 19
Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker MapReduce Engine HDFS Cluster Job Tracker Admin Node Name node Hadoop Core Components Slide 20
Hadoop Core Components Slide 21 Hadoop is a system for large scale data processing. It has two main components:  HDFS – Hadoop Distributed File System (Storage)  Distributed across “nodes”  Natively redundant  NameNode tracks locations.  MapReduce (Processing)  Splits a task across processors  “near” the data & assembles results  Self-Healing, High Bandwidth  Clustered storage  JobTracker manages the TaskTrackers
HDFS Definition Slide 22 The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. HDFS is a distributed, scalable, and portable filesystem written in Java for the Hadoop framework. It has many similarities with existing distributed file systems. Hadoop Distributed File System (HDFS™) is the primary storage system used by Hadoop applications. HDFS creates multiple replicas of data blocks and distributes them on compute nodes throughout a cluster to enable reliable, extremely rapid computations. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets • HDFS consists of following components (daemons) – Name Node – Data Node – Secondary Name Node
HDFS Components Slide 23 Namenode: NameNode, a master server, manages the file system namespace and regulates access to files by clients. Maintains and Manages the blocks which are present on the datanode.  Meta-data in Memory – The entire metadata is in main memory  Types of Metadata – List of files – List of Blocks for each file – List of DataNodes for each block – File attributes, e.g creation time, replication factor  A Transaction Log – Records file creations, file deletions. Etc
HDFS Components Slide 24 Data Node: Slaves which are deployed on each mechine and provide the actual storage. DataNodes, one per node in the cluster, manages storage attached to the nodes that they run on  A Block Server – Stores data in the local file system (e.g. ext3) – Stores meta-data of a block (e.g. CRC) – Serves data and meta-data to Clients – Block Report – Periodically sends a report of all existing blocks to the NameNode  Facilitates Pipelining of Data – Forwards data to other specified DataNodes
Understanding the File system Block placement • Current Strategy  One replica on local node  Second replica on a remote rack  Third replica on same remote rack  Additional replicas are randomly placed • Clients read from nearest replica Data Correctness • Use Checksums to validate data  Use CRC32 • File Creation  Client computes checksum per 512 byte  DataNode stores the checksum • File access  Client retrieves the data and checksum from DataNode
Understanding the File system Data pipelining  Client retrieves a list of DataNodes on which to place replicas of a block  Client writes block to the first DataNode  The first DataNode forwards the data to the next DataNode in the Pipeline  When all replicas are written, the Client moves on to write the next block in file Rebalancer • Goal: % of disk occupied on Datanodes should be similar  Usually run when new Datanodes are added  Cluster is online when Rebalancer is active  Rebalancer is throttled to avoid network congestion  Command line tool
Secondary NameNode Slide 27  Secondary NameNode: Not a hot standby for the NameNode Connects to NameNode every hour* Housekeeping, backup of NemeNode metadata Saved metadata can build a failed NameNode
Metadata (Name, replicas,…) Rack 1 Blocks DataNodes Block ops Replication Write DataNodes Metadata ops Client Read NameNode Rack 2 Client HDFS Architecture Slide 28
NameNode DataNode DataNode DataNode DataNode 2. Create 7. Complete 5. ack Packet4. Write Packet Pipeline of Data nodes DataNode DataNode Distributed File System HDFS Client 1. Create 3. Write 6. Close Anatomy of A File Write Slide 29
NameNode NameNode DataNode DataNode DataNode DataNode 2. Get Block locations 4. Read DataNode DataNode Distributed File System HDFS Client 1. Open 3. Read 5. Read FS Data Input Stream 6. Close Client JVM Client Node Anatomy of A File Read Slide 30
Replication and Rack Awareness Slide 31
Task TrackerTask Tracker Task TrackerTask TrackerData Node Task Tracker Client HDFS Name Node Task Tracker Task Tracker Data Node Data Node Map Reduce Job Tracker Hadoop Cluster Architecture (Contd.) Data Node Data Node Task Tracker Data Node Data Node Data Node
Data Node RAM: 16GB Hard disk: 6 X 2TB Processor: Xenon with 2 cores. Ethernet: 3 X 10 GB/s OS: 64-bit CentOS Name Node RAM: 64 GB, Hard disk: 1 TB Processor: Xenon with 8 Cores Ethernet: 3 X 10 GB/s OS: 64-bit CentOS Power: Redundant Power Supply Secondary Name Node RAM: 32 GB, Hard disk: 1 TB Processor: Xenon with 4 Cores Ethernet: 3 X 10 GB/s OS: 64-bit CentOS Power: Redundant Power Supply Data Node RAM: 16GB Hard disk: 6 X 2TB Processor: Xenon with 2 cores. Ethernet: 3 X 10 GB/s OS: 64-bit CentOS Hadoop Cluster: A Typical Use Case
http://wiki.apache.org/hadoop/PoweredBy Hadoop Cluster: Facebook
Hadoop can run in any of the following three modes: Standalone (or Local) Mode  No daemons, everything runs in a single JVM.  Suitable for running MapReduce programs during development.  Has no DFS. Pseudo-Distributed Mode  Hadoop daemons run on the local machine. Fully-Distributed Mode  Hadoop daemons run on a cluster of machines. Hadoop Cluster Modes
Core HDFS core-site.xml hdfs-site.xml mapred-site.xml Map Reduce Hadoop 1.x: Core Configuration Files
Hadoop Configuration Files Configuration Filenames Description of Log Files hadoop-env.sh Environment variables that are used in the scripts to run Hadoop. core-site.xml Configuration settings for Hadoop Core such as I/O settings that are common to HDFS and MapReduce. hdfs-site.xml Configuration settings for HDFS daemons, the namenode, the secondary namenode and the data nodes. mapred-site.xml Configuration settings for MapReduce daemons : the job-tracker and the task-trackers. masters A list of machines (one per line) that each run a secondary namenode. slaves A list of machines (one per line) that each run a datanode and a task-tracker.
core-site.xml and hdfs-site.xml hdfs-site.xml core-site.xml <?xml version - "1.0"?> <?xml version ="1.0"?> <!--hdfs-site.xml--> <!--core-site.xml--> <configuration> <configuration> <property> <property> <name>dfs.replication</name> <name>fs.default.name</name> <value>1</value> <value>hdfs://localhost:8020/</value> </property> </property> </configuration> </configuration>
Defining HDFS Details In hdfs-site.xml Property Value Description dfs.data.dir <value> /disk1/hdfs/data, /disk2/hdfs/data </value> A list of directories where the datanode stores blocks. Each block is stored in only one of these directories. ${hadoop.tmp.dir}/dfs/data fs.checkpoint.dir <value> /disk1/hdfs/namesecondary, /disk2/hdfs/namesecondary </value> A list of directories where the secondary namenode stores checkpoints. It stores a copy of the checkpoint in each directory in the list ${hadoop.tmp.dir}/dfs/namesecondary
mapred-site.xml mapred-site.xml <?xml version=“1.0”?> <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:8021</value> <property> </configuration>
Defining mapred-site.xml Property Value Description mapred.job.tracker <value> localhost:8021 </value> The hostname and the port that the jobtracker RPC server runs on. If set to the default value of local, then the jobtracker runs in-process on demand when you run a MapReduce job. mapred.local.dir ${hadoop.tmp.dir}/mapred/local A list of directories where MapReduce stores intermediate data for jobs. The data is cleared out when the job ends. mapred.system.dir ${hadoop.tmp.dir}/mapred/syste m The directory relative to fs.default.name where shared files are stored, during a job run. mapred.tasktracker.map.tasks. maximum 2 The number of map tasks that may be run on a tasktracker at any one time mapred.tasktracker.reduce.tasks .maximum 2 The number of reduce tasks tat may be run on a tasktracker at any one time.
1. http://hadoop.apache.org/docs/r1.1.2/core-default.html 2. http://hadoop.apache.org/docs/r1.1.2/mapred-default.html 3. http://hadoop.apache.org/docs/r1.1.2/hdfs-default.html All Properties
Masters  Contains a list of hosts, one per line, that are to host Secondary NameNode servers. Slaves and Masters Two files are used by the startup and shutdown commands: Slaves  Contains a list of hosts, one per line, that are to host DataNode and TaskTracker servers.
 This file also offers a way to provide custom parameters for each of the servers.  Hadoop-env.sh is sourced by all of the Hadoop Core scripts provided in the conf/ directory of the installation.  Examples of environment variables that you can specify: export HADOOP_DATANODE_HEAPSIZE="128" export HADOOP_TASKTRACKER_HEAPSIZE="512" Set parameter JAVA_HOME JVMhadoop-env.sh Per-Process Runtime Environment
 NameNode status: http://localhost:50070/dfshealth.jsp  JobTracker status: http://localhost:50030/jobtracker.jsp  TaskTracker status: http://localhost:50060/tasktracker.jsp  DataBlock Scanner Report: http://localhost:50075/blockScannerReport Web UI URLs
Practice HDFS Commands
Open a terminal window to the current working directory. # /home/notroot ----------------------------------------------------------------- # 1. Print the Hadoop version hadoop version ----------------------------------------------------------------- # 2. List the contents of the root directory in HDFS hadoop fs -ls / ----------------------------------------------------------------- # 3. Report the amount of space used and available on currently mounted filesystem hadoop fs -df hdfs:/ ----------------------------------------------------------------- # 4. Count the number of directories,files and bytes under the paths that match the specified file pattern hadoop fs -count hdfs:/
5. Run a DFS filesystem checking utility hadoop fsck – / ----------------------------------------------------------------- # 6. Run a cluster balancing utility hadoop balancer ----------------------------------------------------------------- # 7. Create a new directory named “hadoop” below the /user/training directory in HDFS. hadoop fs -mkdir /user/training/hadoop ----------------------------------------------------------------- # 8. Add a sample text file from the local directory named “data” to the new directory you created in HDFS hadoop fs -put data/sample.txt /user/training/hadoop
9. List the contents of this new directory in HDFS. hadoop fs -ls /user/training/hadoop ----------------------------------------------------------------- # 10. Add the entire local directory called “retail” to the /user/training directory in HDFS. hadoop fs -put data/retail /user/training/hadoop ----------------------------------------------------------------- # 11. Since /user/training is your home directory in HDFS, any command that does not have an absolute path is interpreted as relative to that directory. # The next command will therefore list your home directory, and should show the items you’ve just added there. hadoop fs -ls ----------------------------------------------------------------- # 12. See how much space this directory occupies in HDFS. hadoop fs -du -s -h hadoop/retail
13. Delete a file ‘customers’ from the “retail” directory. hadoop fs -rm hadoop/retail/customers ----------------------------------------------------------------- # 14. Ensure this file is no longer in HDFS. hadoop fs -ls hadoop/retail/customers ----------------------------------------------------------------- # 15. Delete all files from the “retail” directory using a wildcard. hadoop fs -rm hadoop/retail/* ----------------------------------------------------------------- # 16. To empty the trash hadoop fs –expunge ----------------------------------------------------------------- # 17. Finally, remove the entire retail directory and all of its contents in HDFS. hadoop fs -rm -r hadoop/retail
# 18. Add the purchases.txt file from the local directory named “/home/training/” to the hadoop directory you created in HDFS hadoop fs -copyFromLocal /home/training/purchases.txt hadoop/ ----------------------------------------------------------------- # 19. To view the contents of your text file purchases.txt which is present in your hadoop directory. hadoop fs -cat hadoop/purchases.txt ----------------------------------------------------------------- # 20. Move a directory from one location to other hadoop fs -mv hadoop apache_hadoop ----------------------------------------------------------------- # 21. Add the purchases.txt file from “hadoop” directory which is present in HDFS directory to the directory “data” which is present in your local directory hadoop fs -copyToLocal hadoop/purchases.txt /home/training/data
# 26. Default names of owner and group are training,training # Use ‘-chown’ to change owner name and group name simultaneously hadoop fs -ls hadoop/purchases.txt sudo -u hdfs hadoop fs -chown root:root hadoop/purchases.txt ----------------------------------------------------------------- # 27. Default name of group is training # Use ‘-chgrp’ command to change group name hadoop fs -ls hadoop/purchases.txt sudo -u hdfs hadoop fs -chgrp training hadoop/purchases.txt ----------------------------------------------------------------- # 28. Default replication factor to a file is 3. # Use ‘-setrep’ command to change replication factor of a file hadoop fs -setrep -w 2 apache_hadoop/sample.txt
29. Copy a directory from one node in the cluster to another # Use ‘-distcp’ command to copy, # Use '-overwrite' option to overwrite in an existing files # Use '-update' command to synchronize both directories hadoop fs -distcp hdfs://namenodeA/apache_hadoop hdfs://namenodeB/hadoop ----------------------------------------------------------------- # 30. Command to make the name node leave safe mode hadoop fs -expunge sudo -u hdfs hdfs dfsadmin -safemode leave ----------------------------------------------------------------- # 31. List all the hadoop file system shell commands hadoop fs ----------------------------------------------------------------- # 33. Last but not least, always ask for help! hadoop fs -help
Thank You

More Related Content

PPTX
Introduction to HDFS
byBhavesh Padharia
 
PPTX
Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...
bySimplilearn
 
PPTX
Introduction to Hadoop and Hadoop component
byrebeccatho
 
PPTX
PPT on Hadoop
byShubham Parmar
 
PPSX
Hadoop
byNishant Gandhi
 
PDF
Hadoop Overview & Architecture
byEMC
 
PPTX
Hadoop And Their Ecosystem ppt
bysunera pathan
 
PDF
Hadoop YARN
byVigen Sahakyan
 
Introduction to HDFS
byBhavesh Padharia
 
Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...
bySimplilearn
 
Introduction to Hadoop and Hadoop component
byrebeccatho
 
PPT on Hadoop
byShubham Parmar
 
Hadoop
byNishant Gandhi
 
Hadoop Overview & Architecture
byEMC
 
Hadoop And Their Ecosystem ppt
bysunera pathan
 
Hadoop YARN
byVigen Sahakyan
 

What's hot

PPTX
Design of Hadoop Distributed File System
byDr. C.V. Suresh Babu
 
PPTX
Big Data Analytics with Hadoop
byPhilippe Julio
 
PDF
HDFS Architecture
byJeff Hammerbacher
 
PDF
Introduction to Hadoop
byApache Apex
 
PPTX
Apache HBase™
byPrashant Gupta
 
PPTX
Introduction to Hadoop
byDr. C.V. Suresh Babu
 
PDF
Big Data Ecosystem
byLucian Neghina
 
PPTX
Hadoop Tutorial For Beginners
byDataflair Web Services Pvt Ltd
 
PPT
Seminar Presentation Hadoop
byVarun Narang
 
PPTX
Hadoop Distributed File System
byRutvik Bapat
 
PPTX
Map Reduce
byPrashant Gupta
 
PPTX
introduction to NOSQL Database
bynehabsairam
 
PDF
Data warehouse architecture
bypcherukumalla
 
PPT
Map reduce in BIG DATA
byGauravBiswas9
 
PPTX
Apache PIG
byPrashant Gupta
 
ODP
Machine Learning with Decision trees
byKnoldus Inc.
 
PPTX
Introduction to Map Reduce
byApache Apex
 
PPT
Hadoop Map Reduce
byVNIT-ACM Student Chapter
 
PPTX
Big data and Hadoop
byRahul Agarwal
 
PPTX
Hadoop Architecture
byDr. C.V. Suresh Babu
 
Design of Hadoop Distributed File System
byDr. C.V. Suresh Babu
 
Big Data Analytics with Hadoop
byPhilippe Julio
 
HDFS Architecture
byJeff Hammerbacher
 
Introduction to Hadoop
byApache Apex
 
Apache HBase™
byPrashant Gupta
 
Introduction to Hadoop
byDr. C.V. Suresh Babu
 
Big Data Ecosystem
byLucian Neghina
 
Hadoop Tutorial For Beginners
byDataflair Web Services Pvt Ltd
 
Seminar Presentation Hadoop
byVarun Narang
 
Hadoop Distributed File System
byRutvik Bapat
 
Map Reduce
byPrashant Gupta
 
introduction to NOSQL Database
bynehabsairam
 
Data warehouse architecture
bypcherukumalla
 
Map reduce in BIG DATA
byGauravBiswas9
 
Apache PIG
byPrashant Gupta
 
Machine Learning with Decision trees
byKnoldus Inc.
 
Introduction to Map Reduce
byApache Apex
 
Hadoop Map Reduce
byVNIT-ACM Student Chapter
 
Big data and Hadoop
byRahul Agarwal
 
Hadoop Architecture
byDr. C.V. Suresh Babu
 

Similar to Hadoop File system (HDFS)

PPTX
Hadoop Training | Hadoop Training For Beginners | Hadoop Architecture | Hadoo...
bySimplilearn
 
PDF
Hadoop data management
bySubhas Kumar Ghosh
 
PPTX
Introduction to hadoop and hdfs
byshrey mehrotra
 
PDF
Big Data Architecture Workshop - Vahid Amiri
bydatastack
 
PPTX
Managing Big data with Hadoop
byNalini Mehta
 
PPT
Big data & hadoop framework
byTu Pham
 
PPTX
Module 1- Introduction to Big Data and Hadoop
bySiddheshMhatre27
 
PPTX
Big Data Analytics -Introduction education
bymohammedansaralima
 
PDF
Hadoop introduction
bySubhas Kumar Ghosh
 
PPTX
BDA_UNIT_II_BDA_UNIT_II_BDA_UNIT_II_BDA_UNIT_II_
bySrikanthYadav578790
 
PPTX
Hadoop.pptx
bysonukumar379092
 
PPT
haoop_architecture.pptHadoop Architecture is a framework designed for storing...
byRohiniRajaramPandian
 
PPT
HDFS_architecture.ppt
byvijayapraba1
 
PDF
Introduction to Big Data Hadoop Training Online by www.itjobzone.biz
byITJobZone.biz
 
PPTX
Unit-1 Introduction to Big Data.pptx
byAnkitChauhan817826
 
PPT
Hadoop training in bangalore
byKelly Technologies
 
PPTX
Hadoop
byRittikaBaksi
 
PPTX
Introduction to HDFS
bySiddharth Mathur
 
PPTX
Bigdata and Hadoop
byGirish L
 
PPTX
Bigdata workshop february 2015
byclairvoyantllc
 
Hadoop Training | Hadoop Training For Beginners | Hadoop Architecture | Hadoo...
bySimplilearn
 
Hadoop data management
bySubhas Kumar Ghosh
 
Introduction to hadoop and hdfs
byshrey mehrotra
 
Big Data Architecture Workshop - Vahid Amiri
bydatastack
 
Managing Big data with Hadoop
byNalini Mehta
 
Big data & hadoop framework
byTu Pham
 
Module 1- Introduction to Big Data and Hadoop
bySiddheshMhatre27
 
Big Data Analytics -Introduction education
bymohammedansaralima
 
Hadoop introduction
bySubhas Kumar Ghosh
 
BDA_UNIT_II_BDA_UNIT_II_BDA_UNIT_II_BDA_UNIT_II_
bySrikanthYadav578790
 
Hadoop.pptx
bysonukumar379092
 
haoop_architecture.pptHadoop Architecture is a framework designed for storing...
byRohiniRajaramPandian
 
HDFS_architecture.ppt
byvijayapraba1
 
Introduction to Big Data Hadoop Training Online by www.itjobzone.biz
byITJobZone.biz
 
Unit-1 Introduction to Big Data.pptx
byAnkitChauhan817826
 
Hadoop training in bangalore
byKelly Technologies
 
Hadoop
byRittikaBaksi
 
Introduction to HDFS
bySiddharth Mathur
 
Bigdata and Hadoop
byGirish L
 
Bigdata workshop february 2015
byclairvoyantllc
 

More from Prashant Gupta

PDF
Sqoop
byPrashant Gupta
 
PPTX
6.hive
byPrashant Gupta
 
PPTX
Mongodb - NoSql Database
byPrashant Gupta
 
PPTX
Spark core
byPrashant Gupta
 
PPTX
Map reduce prashant
byPrashant Gupta
 
PPTX
Spark Sql and DataFrame
byPrashant Gupta
 
PPTX
Sonar Tool - JAVA code analysis
byPrashant Gupta
 
Sqoop
byPrashant Gupta
 
6.hive
byPrashant Gupta
 
Mongodb - NoSql Database
byPrashant Gupta
 
Spark core
byPrashant Gupta
 
Map reduce prashant
byPrashant Gupta
 
Spark Sql and DataFrame
byPrashant Gupta
 
Sonar Tool - JAVA code analysis
byPrashant Gupta
 

Recently uploaded

PDF
Time Travel on ACID - A Gentle Introduction To Apache Iceberg
byMichal Gancarski
 
PPTX
Technical_Workflow_of_Indian_Startup_Funding_Trends_Project_1 (4).pptx
byAryanthakur98
 
PDF
Apache Iceberg - Transactional Storage for Large Datasets
byMichal Gancarski
 
DOCX
Cost Effective Design for Streaming Data Systems.docx
bytotabrez1
 
PDF
Security and confidentiality in big data.pdf
byrayessafa21
 
PPT
Teaching Material 1 - HOW SPATIAL DATABASE-GIS DEFINED.ppt
bySafizaSuhanaFtmk1
 
PPTX
Artificial neural network Available for instant download in exchange for a bi...
byvinstonr
 
PDF
Bratislava Tableau User group (BA_TUG) - 15_01_2026 - in-person meeting.pdf
byPavol Hromadka
 
PPTX
Decision science introduction in decision making
bymatthewquayson2
 
PDF
Applied_Data_Science_Unit_One_Notes_for_reference
bysanguleumeshit
 
PDF
Power BI Sales Analysis: Performance, Trends, and Strategic Insights my final...
bymaduruth09
 
PDF
Torre Avianca- Modernity, Identity, and Technology in Colombian Architecture.pdf
bysanchezpipe1234
 
PDF
mythbusting-cdc-what-employers-need-to-know.pdf
byHenry Tapper
 
PPTX
[DSC Europe 25] Slobodan Dolinic - Smart and Intelligent Green Region.pptx
byDataScienceConferenc1
 
PDF
(UPDATE) JADWAL GEREJA OPS LILIN 2025.pdf
byMinlantasSumber
 
PPT
linear_algebra mldataaimachiee leARNINGML
byssuser70a8841
 
PDF
The Architecture of Tomorrow: Elon Musk’s Vision for a Post-Work Society
bySubtract 8
 
PDF
Nepal's Health Policies: From 1991 to 2024 - A Complete Analysis of Nepal's H...
byPublic Health Concern Nepal
 
PPTX
Child Safeguarding training ppt for one day.pptx
bymeasho Birhanu
 
PDF
Comprehensive Power BI Dashboard for Data-Driven Business Insights and Perfor...
byNusrat Gulbarga
 
Time Travel on ACID - A Gentle Introduction To Apache Iceberg
byMichal Gancarski
 
Technical_Workflow_of_Indian_Startup_Funding_Trends_Project_1 (4).pptx
byAryanthakur98
 
Apache Iceberg - Transactional Storage for Large Datasets
byMichal Gancarski
 
Cost Effective Design for Streaming Data Systems.docx
bytotabrez1
 
Security and confidentiality in big data.pdf
byrayessafa21
 
Teaching Material 1 - HOW SPATIAL DATABASE-GIS DEFINED.ppt
bySafizaSuhanaFtmk1
 
Artificial neural network Available for instant download in exchange for a bi...
byvinstonr
 
Bratislava Tableau User group (BA_TUG) - 15_01_2026 - in-person meeting.pdf
byPavol Hromadka
 
Decision science introduction in decision making
bymatthewquayson2
 
Applied_Data_Science_Unit_One_Notes_for_reference
bysanguleumeshit
 
Power BI Sales Analysis: Performance, Trends, and Strategic Insights my final...
bymaduruth09
 
Torre Avianca- Modernity, Identity, and Technology in Colombian Architecture.pdf
bysanchezpipe1234
 
mythbusting-cdc-what-employers-need-to-know.pdf
byHenry Tapper
 
[DSC Europe 25] Slobodan Dolinic - Smart and Intelligent Green Region.pptx
byDataScienceConferenc1
 
(UPDATE) JADWAL GEREJA OPS LILIN 2025.pdf
byMinlantasSumber
 
linear_algebra mldataaimachiee leARNINGML
byssuser70a8841
 
The Architecture of Tomorrow: Elon Musk’s Vision for a Post-Work Society
bySubtract 8
 
Nepal's Health Policies: From 1991 to 2024 - A Complete Analysis of Nepal's H...
byPublic Health Concern Nepal
 
Child Safeguarding training ppt for one day.pptx
bymeasho Birhanu
 
Comprehensive Power BI Dashboard for Data-Driven Business Insights and Perfor...
byNusrat Gulbarga
 

Hadoop File system (HDFS)

  • 1.
  • 2.
     What isBig Data?  Limitations of the existing solutions  Solving the problem with Hadoop  Introduction to Hadoop  Hadoop Eco-System  Hadoop Core Components  HDFS Architecture  Anatomy of a File Write and Read Topics of the Day Slide 2
  • 3.
    What is BigData? Slide 3  Lots of Data (Terabytes or Petabytes)  Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.  The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. cloud tools statistics No SQL compression storage support database analyze information terabytes mobile processing Big Data
  • 4.
    What is BigData? Slide 4  Systems / Enterprises generate huge amount of data from Terabytes to and even Petabytes of information. NYSE generates about one terabyte of new trade data per day to perform stock trading analytics to determine trends for optimal trades.
  • 5.
    Un – StructuredData is Exploding Slide 5
  • 6.
     IBM’s Definition– Big Data Characteristics http://www-01.ibm.com/software/data/bigdata/ Web logs Videos Images Audios Sensor Data Slide 6 Volume Velocity Variety IBM’s Definition
  • 7.
     Web ande-tailing  Recommendation Engines  Ad Targeting  Search Quality  Abuse and Click Fraud Detection  Telecommunications  Customer Churn Prevention  Network Performance Optimization  Calling Data Record (CDR) Analysis  Analyzing Network to Predict Failure Common Big Data Customer Scenarios Slide 7
  • 8.
     Government  FraudDetection and Cyber Security  Welfare Schemes  Justice  Healthcare & Life Sciences  Health Information Exchange  Gene Sequencing  Serialization  Healthcare Service Quality Improvements  Drug Safety Common Big Data Customer Scenarios(Contd.) Slide 8
  • 9.
    Common Big DataCustomer Scenarios Contd.) Slide 9  Banks and Financial services  Modeling True Risk  Threat Analysis  Fraud Detection  Trade Surveillance  Credit Scoring and Analysis  Retail  Point of Sales Transaction Analysis  Customer Churn Analysis  Sentiment Analysis
  • 10.
    Hidden Treasure Case Study:Sears Holding Corporation X *Sears was using traditional systems such as Oracle Exadata, Teradata and SAS etc. to store and process the customer activity and sales data. Slide 17  Insight into data can provide Business Advantage.  Some key early indicators can mean Fortunes to Business.  More Precise Analysis with more data.
  • 11.
    Mostly Append Collection Instrumentation http://www.informationweek.com/it-leadership/why-sears-is-going-all-in-on-hadoop/d/d-id/1107038? 90% of the~2PB Archived Storage Processing BI Reports + Interactive Apps RDBMS (Aggregated Data) ETL Compute Grid 3. Premature data death 1. Can’t explore original high fidelity raw data 2. Moving data to compute doesn’t scale A meagre 10% of the ~2PB Data is available for BI Storage only Grid (original Raw Data) Limitations of Existing Data Analytics Architecture Slide 11
  • 12.
    *Sears moved toa 300-Node Hadoop cluster to keep 100% of its data available for processing rather than a meagre 10% as was the case with existing Non-Hadoop solutions. RDBMS (Aggregated Data) No Data Archiving 1. Data Exploration & Advanced analytics 2. Scalable throughput for ETL & aggregation 3. Keep data alive forever Mostly Append Collection Instrumentation BI Reports + Interactive Apps Hadoop : Storage + Compute Grid Entire ~2PB Data is available for processing Both Storage And Processing Solution: A Combined Storage Computer Layer Slide 12
  • 13.
  • 14.
    RDBMS EDW Hadoop –It’s about Scale and Structure Slide 14 MPP NoSQL HADOOP Structured Data Types Multi and Unstructured Limited, No Data Processing Processing Processing coupled with Data Standards & Structured Governance Loosely Structured Required On write Schema Required On Read Software License Cost Support Only Known Entity Resources Growing, Complexities, Wide Interactive OLAP Analytics Operational Data Store Best Fit Use Massive Storage/Processing
  • 15.
    Read 1 TBData 1 Machine 4 I/O Channels Each Channel – 100 MB/s 10 Machines 4 I/O Channels Each Channel – 100 MB/s Why DFS? Slide 15
  • 16.
    Read 1 TBData 10 Machines 4 I/O Channels Each Channel – 100 MB/s Why DFS? Slide 16 45 Minutes 1 Machine 4 I/O Channels Each Channel – 100 MB/s
  • 17.
    Read 1 TBData 10 Machines 4 I/O Channels Each Channel – 100 MB/s Why DFS? Slide 17 45 Minutes 4.5 Minutes 1 Machine 4 I/O Channels Each Channel – 100 MB/s
  • 18.
    What is Hadoop? Slide18  Apache Hadoop is a framework that allows for the distributed processing of large data sets across clusters of commodity computers using a simple programming model.  It is an Open-source Data Management with scale-out storage & distributed processing.
  • 19.
    Apache Oozie (Workflow) MapReduceFramework HBase HDFS (Hadoop Distributed File System) Flume Sqoop Import Or Export Pig Latin Data Analysis Mahout Machine Learning Hive DW System Unstructured or Semi-Structured data Structured Data Hadoop Eco-System Slide 19
  • 20.
    Data Node Task Tracker Data Node Task Tracker DataNode Task Tracker Data Node Task Tracker MapReduce Engine HDFS Cluster Job Tracker Admin Node Name node Hadoop Core Components Slide 20
  • 21.
    Hadoop Core Components Slide21 Hadoop is a system for large scale data processing. It has two main components:  HDFS – Hadoop Distributed File System (Storage)  Distributed across “nodes”  Natively redundant  NameNode tracks locations.  MapReduce (Processing)  Splits a task across processors  “near” the data & assembles results  Self-Healing, High Bandwidth  Clustered storage  JobTracker manages the TaskTrackers
  • 22.
    HDFS Definition Slide 22 TheHadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. HDFS is a distributed, scalable, and portable filesystem written in Java for the Hadoop framework. It has many similarities with existing distributed file systems. Hadoop Distributed File System (HDFS™) is the primary storage system used by Hadoop applications. HDFS creates multiple replicas of data blocks and distributes them on compute nodes throughout a cluster to enable reliable, extremely rapid computations. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets • HDFS consists of following components (daemons) – Name Node – Data Node – Secondary Name Node
  • 23.
    HDFS Components Slide 23 Namenode: NameNode,a master server, manages the file system namespace and regulates access to files by clients. Maintains and Manages the blocks which are present on the datanode.  Meta-data in Memory – The entire metadata is in main memory  Types of Metadata – List of files – List of Blocks for each file – List of DataNodes for each block – File attributes, e.g creation time, replication factor  A Transaction Log – Records file creations, file deletions. Etc
  • 24.
    HDFS Components Slide 24 DataNode: Slaves which are deployed on each mechine and provide the actual storage. DataNodes, one per node in the cluster, manages storage attached to the nodes that they run on  A Block Server – Stores data in the local file system (e.g. ext3) – Stores meta-data of a block (e.g. CRC) – Serves data and meta-data to Clients – Block Report – Periodically sends a report of all existing blocks to the NameNode  Facilitates Pipelining of Data – Forwards data to other specified DataNodes
  • 25.
    Understanding the Filesystem Block placement • Current Strategy  One replica on local node  Second replica on a remote rack  Third replica on same remote rack  Additional replicas are randomly placed • Clients read from nearest replica Data Correctness • Use Checksums to validate data  Use CRC32 • File Creation  Client computes checksum per 512 byte  DataNode stores the checksum • File access  Client retrieves the data and checksum from DataNode
  • 26.
    Understanding the Filesystem Data pipelining  Client retrieves a list of DataNodes on which to place replicas of a block  Client writes block to the first DataNode  The first DataNode forwards the data to the next DataNode in the Pipeline  When all replicas are written, the Client moves on to write the next block in file Rebalancer • Goal: % of disk occupied on Datanodes should be similar  Usually run when new Datanodes are added  Cluster is online when Rebalancer is active  Rebalancer is throttled to avoid network congestion  Command line tool
  • 27.
    Secondary NameNode Slide 27 Secondary NameNode: Not a hot standby for the NameNode Connects to NameNode every hour* Housekeeping, backup of NemeNode metadata Saved metadata can build a failed NameNode
  • 28.
    Metadata (Name, replicas,…) Rack1 Blocks DataNodes Block ops Replication Write DataNodes Metadata ops Client Read NameNode Rack 2 Client HDFS Architecture Slide 28
  • 29.
    NameNode DataNode DataNode DataNode DataNode 2. Create 7. Complete 5.ack Packet4. Write Packet Pipeline of Data nodes DataNode DataNode Distributed File System HDFS Client 1. Create 3. Write 6. Close Anatomy of A File Write Slide 29
  • 30.
    NameNode NameNode DataNode DataNode DataNode DataNode 2. Get Blocklocations 4. Read DataNode DataNode Distributed File System HDFS Client 1. Open 3. Read 5. Read FS Data Input Stream 6. Close Client JVM Client Node Anatomy of A File Read Slide 30
  • 31.
    Replication and RackAwareness Slide 31
  • 32.
    Task TrackerTask Tracker TaskTrackerTask TrackerData Node Task Tracker Client HDFS Name Node Task Tracker Task Tracker Data Node Data Node Map Reduce Job Tracker Hadoop Cluster Architecture (Contd.) Data Node Data Node Task Tracker Data Node Data Node Data Node
  • 33.
    Data Node RAM: 16GB Harddisk: 6 X 2TB Processor: Xenon with 2 cores. Ethernet: 3 X 10 GB/s OS: 64-bit CentOS Name Node RAM: 64 GB, Hard disk: 1 TB Processor: Xenon with 8 Cores Ethernet: 3 X 10 GB/s OS: 64-bit CentOS Power: Redundant Power Supply Secondary Name Node RAM: 32 GB, Hard disk: 1 TB Processor: Xenon with 4 Cores Ethernet: 3 X 10 GB/s OS: 64-bit CentOS Power: Redundant Power Supply Data Node RAM: 16GB Hard disk: 6 X 2TB Processor: Xenon with 2 cores. Ethernet: 3 X 10 GB/s OS: 64-bit CentOS Hadoop Cluster: A Typical Use Case
  • 34.
  • 35.
    Hadoop can runin any of the following three modes: Standalone (or Local) Mode  No daemons, everything runs in a single JVM.  Suitable for running MapReduce programs during development.  Has no DFS. Pseudo-Distributed Mode  Hadoop daemons run on the local machine. Fully-Distributed Mode  Hadoop daemons run on a cluster of machines. Hadoop Cluster Modes
  • 36.
  • 37.
    Hadoop Configuration Files Configuration Filenames Descriptionof Log Files hadoop-env.sh Environment variables that are used in the scripts to run Hadoop. core-site.xml Configuration settings for Hadoop Core such as I/O settings that are common to HDFS and MapReduce. hdfs-site.xml Configuration settings for HDFS daemons, the namenode, the secondary namenode and the data nodes. mapred-site.xml Configuration settings for MapReduce daemons : the job-tracker and the task-trackers. masters A list of machines (one per line) that each run a secondary namenode. slaves A list of machines (one per line) that each run a datanode and a task-tracker.
  • 38.
    core-site.xml and hdfs-site.xml hdfs-site.xmlcore-site.xml <?xml version - "1.0"?> <?xml version ="1.0"?> <!--hdfs-site.xml--> <!--core-site.xml--> <configuration> <configuration> <property> <property> <name>dfs.replication</name> <name>fs.default.name</name> <value>1</value> <value>hdfs://localhost:8020/</value> </property> </property> </configuration> </configuration>
  • 39.
    Defining HDFS DetailsIn hdfs-site.xml Property Value Description dfs.data.dir <value> /disk1/hdfs/data, /disk2/hdfs/data </value> A list of directories where the datanode stores blocks. Each block is stored in only one of these directories. ${hadoop.tmp.dir}/dfs/data fs.checkpoint.dir <value> /disk1/hdfs/namesecondary, /disk2/hdfs/namesecondary </value> A list of directories where the secondary namenode stores checkpoints. It stores a copy of the checkpoint in each directory in the list ${hadoop.tmp.dir}/dfs/namesecondary
  • 40.
  • 41.
    Defining mapred-site.xml Property ValueDescription mapred.job.tracker <value> localhost:8021 </value> The hostname and the port that the jobtracker RPC server runs on. If set to the default value of local, then the jobtracker runs in-process on demand when you run a MapReduce job. mapred.local.dir ${hadoop.tmp.dir}/mapred/local A list of directories where MapReduce stores intermediate data for jobs. The data is cleared out when the job ends. mapred.system.dir ${hadoop.tmp.dir}/mapred/syste m The directory relative to fs.default.name where shared files are stored, during a job run. mapred.tasktracker.map.tasks. maximum 2 The number of map tasks that may be run on a tasktracker at any one time mapred.tasktracker.reduce.tasks .maximum 2 The number of reduce tasks tat may be run on a tasktracker at any one time.
  • 42.
  • 43.
    Masters  Contains alist of hosts, one per line, that are to host Secondary NameNode servers. Slaves and Masters Two files are used by the startup and shutdown commands: Slaves  Contains a list of hosts, one per line, that are to host DataNode and TaskTracker servers.
  • 44.
     This filealso offers a way to provide custom parameters for each of the servers.  Hadoop-env.sh is sourced by all of the Hadoop Core scripts provided in the conf/ directory of the installation.  Examples of environment variables that you can specify: export HADOOP_DATANODE_HEAPSIZE="128" export HADOOP_TASKTRACKER_HEAPSIZE="512" Set parameter JAVA_HOME JVMhadoop-env.sh Per-Process Runtime Environment
  • 45.
     NameNode status:http://localhost:50070/dfshealth.jsp  JobTracker status: http://localhost:50030/jobtracker.jsp  TaskTracker status: http://localhost:50060/tasktracker.jsp  DataBlock Scanner Report: http://localhost:50075/blockScannerReport Web UI URLs
  • 46.
  • 47.
    Open a terminalwindow to the current working directory. # /home/notroot ----------------------------------------------------------------- # 1. Print the Hadoop version hadoop version ----------------------------------------------------------------- # 2. List the contents of the root directory in HDFS hadoop fs -ls / ----------------------------------------------------------------- # 3. Report the amount of space used and available on currently mounted filesystem hadoop fs -df hdfs:/ ----------------------------------------------------------------- # 4. Count the number of directories,files and bytes under the paths that match the specified file pattern hadoop fs -count hdfs:/
  • 48.
    5. Run aDFS filesystem checking utility hadoop fsck – / ----------------------------------------------------------------- # 6. Run a cluster balancing utility hadoop balancer ----------------------------------------------------------------- # 7. Create a new directory named “hadoop” below the /user/training directory in HDFS. hadoop fs -mkdir /user/training/hadoop ----------------------------------------------------------------- # 8. Add a sample text file from the local directory named “data” to the new directory you created in HDFS hadoop fs -put data/sample.txt /user/training/hadoop
  • 49.
    9. List thecontents of this new directory in HDFS. hadoop fs -ls /user/training/hadoop ----------------------------------------------------------------- # 10. Add the entire local directory called “retail” to the /user/training directory in HDFS. hadoop fs -put data/retail /user/training/hadoop ----------------------------------------------------------------- # 11. Since /user/training is your home directory in HDFS, any command that does not have an absolute path is interpreted as relative to that directory. # The next command will therefore list your home directory, and should show the items you’ve just added there. hadoop fs -ls ----------------------------------------------------------------- # 12. See how much space this directory occupies in HDFS. hadoop fs -du -s -h hadoop/retail
  • 50.
    13. Delete afile ‘customers’ from the “retail” directory. hadoop fs -rm hadoop/retail/customers ----------------------------------------------------------------- # 14. Ensure this file is no longer in HDFS. hadoop fs -ls hadoop/retail/customers ----------------------------------------------------------------- # 15. Delete all files from the “retail” directory using a wildcard. hadoop fs -rm hadoop/retail/* ----------------------------------------------------------------- # 16. To empty the trash hadoop fs –expunge ----------------------------------------------------------------- # 17. Finally, remove the entire retail directory and all of its contents in HDFS. hadoop fs -rm -r hadoop/retail
  • 51.
    # 18. Addthe purchases.txt file from the local directory named “/home/training/” to the hadoop directory you created in HDFS hadoop fs -copyFromLocal /home/training/purchases.txt hadoop/ ----------------------------------------------------------------- # 19. To view the contents of your text file purchases.txt which is present in your hadoop directory. hadoop fs -cat hadoop/purchases.txt ----------------------------------------------------------------- # 20. Move a directory from one location to other hadoop fs -mv hadoop apache_hadoop ----------------------------------------------------------------- # 21. Add the purchases.txt file from “hadoop” directory which is present in HDFS directory to the directory “data” which is present in your local directory hadoop fs -copyToLocal hadoop/purchases.txt /home/training/data
  • 52.
    # 26. Defaultnames of owner and group are training,training # Use ‘-chown’ to change owner name and group name simultaneously hadoop fs -ls hadoop/purchases.txt sudo -u hdfs hadoop fs -chown root:root hadoop/purchases.txt ----------------------------------------------------------------- # 27. Default name of group is training # Use ‘-chgrp’ command to change group name hadoop fs -ls hadoop/purchases.txt sudo -u hdfs hadoop fs -chgrp training hadoop/purchases.txt ----------------------------------------------------------------- # 28. Default replication factor to a file is 3. # Use ‘-setrep’ command to change replication factor of a file hadoop fs -setrep -w 2 apache_hadoop/sample.txt
  • 53.
    29. Copy adirectory from one node in the cluster to another # Use ‘-distcp’ command to copy, # Use '-overwrite' option to overwrite in an existing files # Use '-update' command to synchronize both directories hadoop fs -distcp hdfs://namenodeA/apache_hadoop hdfs://namenodeB/hadoop ----------------------------------------------------------------- # 30. Command to make the name node leave safe mode hadoop fs -expunge sudo -u hdfs hdfs dfsadmin -safemode leave ----------------------------------------------------------------- # 31. List all the hadoop file system shell commands hadoop fs ----------------------------------------------------------------- # 33. Last but not least, always ask for help! hadoop fs -help
  • 54.