Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of commodity hardware. It was designed to scale up from single servers to thousands of machines, with very high fault tolerance. The document provides an overview of Hadoop, including its evolution from centralized mainframe systems, the motivation for its development, its core architecture components like HDFS and YARN, common technologies that operate within the Hadoop ecosystem like Hive and Spark, and how businesses can use it for tasks like data analytics and business intelligence.

1. HADOOP OVERVIEW
By Sunitha Flowerhill
(Masters in Computer Applications-MCA)
Data, Business Intelligence and Hadoop Architect

2. AGENDA
EVOLUTION AND EXPANSION OF BUSINESS DATA PROCESSING
MOTIVATION BEHIND HADOOP
HADOOP ARCHITECTURE
HADOOP TECHNOLOGIES AND USAGES
DATA WRANGLING ON HADOOP
BUSINESS INTELLIGENCE AND ANALYTICS ON HADOOP

3. EVOLUTION – STAGE 1
 70S – PUNCH CARDS AND PUNCH TAPES WITH HOLES IN IT
 COBOL AND JOB CONTROL LANGUAGE
 ISAM AND C-ISAM FILES – FLAT FILES WITH INDEXES
 WINCHESTER HARD DISKS WHICH LOOKED LIKE DRUMS
 EXAMPLE SYSTEM – PDP 11 BY DIGITAL CORPORATION
 DRAWBACK – VERY SLOW, LOW CAPACITY

4. EVOLUTION – STAGE 2
 80’S - CAME MINI COMPUTERS
 UNIX OPERATING SYSTEM (WHICH WAS DEVELOPED IN THE
60S IN UC-BERKLEY) WHICH IS STILL RUNNING IN MANY
FORMS LIKE HPUX, AIX AND ALSO IS THE MAJOR OPERATING
SYSTEM WHERE HADOOP RESIDES - LINUX
 RELATIONAL DATABASE SYSTEMS LIKE UNIFY, INFORMIX,
SYBASE AND DB2
 LAN BASED NETWORKED PCS – NOVELL NETWARE, DBASE,
FOXPRO – PC/MD-DOS/LAN BASED RDBMS
 SQL – STRUCTURED QUERY LANGUAGE, WHICH IS STILL
HEAVILY USED IN HADOOP AS HIVEQL, SPARK SQL ETC.
 STURDY AND FAULT TOLERANT
 DRAWBACK: LIMITED PROCESSING POWER AND GREEN
SCREEN! NOT MUCH OF A GRAPHICAL EXPERIENCE

5. EVOLUTION – STAGE 3
 CLIENT SERVER ARCHITECTURE – 2 TIER – PC BASED THICK CLIENT
FRONT END FOR PROCESSING DATA AT THE USER END AND A LAN
OR UNIX BASED SERVER FOR THE DATABASE SERVERSIDE
PROCESSING
 GRAPHICAL USER INTERFACE (GUI) FOR THE USER
 MORE PROCESSING POWER AT THE SERVER SIDE
 CONNECTION BETWEEN CLIENT AND SERVER USING OBJECT DATA
BASE CONNECTIVITY (ODBC) OR CALL LEVEL INTERFACE (CLI) –
USING DYNAMIC LINK LIBRARIES (DLLS)
 CLASSIFIED AS DISTRIBUTED SYSTEMS
 DATA STORAGE AND RECOVERY MECHANISMS SUCH AS
MIRRORING, REPLICATION, BLADING ETC WERE POSSIBLE AT THE
SERVER LEVEL
 DRAWBACK : LOW AVAILABILITY, FAILURES, LOTS OF
TROUBLESHOOTING
 “You know you have a distributed system when
the crash of a computer you’ve never
heard of stops you from getting any work
done.” -Leslie Lamport – distributed system computer scientist

6. EVOLUTION – STAGE 4
 3 TIER ARCHITECTURE – THIN CLIENT, APPLICATION-
MIDDLEWARE AND SERVERS FOR DATABASE STORAGE
 THIN APPLICATION CLIENT OR WEB BASED CLIENT, WHICH ONLY
SERVES AS DATA DELIVERY, WITH MINIMAL PROCESSING AT
CLIENT END
 INTRODUCTION OF MIDDLEWARE SUCH AS TUXEDO, WEB
SERVICES, JAVA BEANS – MOST OF BUSINESS LOGIC RESIDES
HERE
 USES PACKET TECHNOLOGY FOR EFFICIENT TRANSPORTATION
AND RECOVERY
 USES DIFFERENT INTERNET PROTOCOLS FOR SECURITY AND
EFFICIENT TRANSPORTATION OF DATA BETWEEN THIN CLIENT
AND SERVER
 MORE GEOGRAPHICALLY DISTRIBUTED SERVERS, MIDDLEWARE
SERVERS, CLUSTER COMPUTING, CHEAP HARDWARE
 LOT OF DATA CAPTURING ACROSS THE INTERNET, FROM SELF
SERVICE APPLICATIONS, USERS, MOBILE APPLICATIONS

7. THAT BRINGS US TO THE MOTIVATION BEHIND
HADOOP
 CHEAP CLUSTERED HARDWARE AVAILABLE NOW
 WE CAN RUN A HADOOP CLUSTER WITH ALL THE LAPTOPS
IN THIS CLASS CONNECTED TOGETHER AS NODES OF THE
CLUSTER
 HARDWARE FAILURE IS COMMON SO HEAVILY REPLICATED
DATA
 MULTIPLE PARALLEL PROCESSING – USAGE OF MULTIPLE
CPUS FOR A SINGLE TASK –SPARK ENGINE IS A GOOD
EXAMPLE OF MPP.
 VARIOUS ANALYSIS CAN BE DONE IN LARGE DATASETS,
FORECASTING, PREDICTIONS, DIRECTIONS FOR BUSINESS
 ANALYTICS BASED INTELLIGENCE RATHER THAN PURE
PRODUCTION BASED MIS REPORTS
 SELLING OF THE DATASETS – HUGE BUSINESS
 AND MANY MORE…….

8. HADOOP
 WE ARE DEALING WITH TERABYTES OF DATA HERE IN CLUSTERED
COMPUTING
 APACHE TOP LEVEL PROJECT, OPEN SOURCE IMPLEMENTATION,
FOR RELIABLE, SCALABLE, DISTRIBUTED COMPUTING AND STORAGE.
 DISTRIBUTED BY HORTONWORKS AND CLOUDERA
 FLEXIBLE AND HIGHLY-AVAILABLE ARCHITECTURE FOR LARGE
SCALE COMPUTATION AND DATA PROCESSING ON A NETWORK
OF COMMODITY HARDWARE.
 STORAGE AND PROCESSING OF LARGE AND RAPIDLY GROWING
DATA.
 STRUCTURED AND UNSTRUCTURED DATA
 HIGH SCALABILITY AND AVAILABILITY
 FAULT TOLERANCE
 NOW INFRASTRUCTURE MAINTENANCE IS AVAILABLE AT LOW COST
BY CLOUD COMPANIES LIKE AWS, GOOGLE, GAIA, MS AZURE ETC

9. BASIC ARCHITECTURE
 MAIN NODES OF CLUSTER ARE WHERE MOST
OF THE COMPUTATIONAL POWER AND
STORAGE OF THE SYSTEM LIES
 MAIN NODES RUN TASKTRACKER TO ACCEPT
AND REPLY TO MAPREDUCE TASKS, AND
ALSO TO DATA NODE TO STORE NEEDED
BLOCKS AS AVAILABLE AS POSSIBLE
 CENTRAL CONTROL NODE RUNS NAMENODE
TO KEEP TRACK OF HDFS DIRECTORIES &
FILES, AND JOBTRACKER TO DISPATCH
COMPUTE TASKS TO TASKTRACKER
 HADOOP IS WRITTEN IN JAVA, ALSO
SUPPORTS PYTHON, RUBY OTHER ENGINES
LIKE SPARK, MORE EFFICIENT LANGUAGES LIKE
SCALA

11. HADOOP DISTRIBUTED FILESYSTEM
(HDFS) ARCHITECTURE
 TAILORED TO THE NEEDS OF MAPREDUCE
 TARGETED TOWARDS MANY READS OF
FILESTREAMS
 WRITES ARE MORE COSTLY – TIME, EFFORT –
SO WRITE ONCE – READ MANY PREFERRED
 HIGH DEGREE OF DATA REPLICATION (3X BY
DEFAULT)
 LARGE BLOCKSIZE (128 MB)
 LOCATION AWARENESS OF DATA NODES IN
NETWORK (GEOGRAPHIC SENSIBLE STORAGE)
Cluster of machines running
Hadoop at Yahoo! (Source: Yahoo!)

12. ARCHITECTURE - NAMENODE
 STORES METADATA FOR THE FILES, LIKE THE
DIRECTORY STRUCTURE OF A TYPICAL FS
 THE SERVER HOLDING THE NAMENODE
INSTANCE IS QUITE CRUCIAL, AS THERE IS
ONLY ONE. AND THERE IS A SECONDARY OR
BACKUP NAMENODE
 TRANSACTION LOG FOR FILE DELETES/ADDS,
ETC. DOES NOT USE TRANSACTIONS FOR
WHOLE BLOCKS OR FILE-STREAMS, ONLY
METADATA
 HANDLES CREATION OF MORE REPLICA
BLOCKS WHEN NECESSARY AFTER A DATA
NODE FAILURE

13. ARCHITECTURE - NAMENODE:
 STORES THE ACTUAL DATA IN HDFS
 CAN RUN ON ANY UNDERLYING
FILESYSTEM (EXT 3/4, NTFS, ETC.)
 NOTIFIES NAMENODE OF WHAT BLOCKS
IT HAS
 NAMENODE REPLICATES BLOCKS 2X IN
LOCAL RACK, 1X ELSEWHERE

14. ARCHITECTURE – JOBTRACKER AND TASKTRACKER
 JOB TRACKER MAKES SURE THAT
EACH OPERATION IS COMPLETED
AND IF THERE IS A PROCESS
FAILURE AT ANY NODE, IT NEEDS
TO ASSIGN A DUPLICATE TASK TO
SOME TASK TRACKER. JOB
TRACKER ALSO DISTRIBUTES THE
ENTIRE TASK TO ALL THE
MACHINES.
 THE TASK TRACKERS (PROJECT
MANAGER IN OUR ANALOGY) IN
DIFFERENT MACHINES
ARE COORDINATED BY A JOB
TRACKER

15. ARCHITECTURE – YARN (YET ANOTHER
RESOURCE NEGOTIATOR)
 YARN ARCHITECTURE CAN BE A
LITTLE CONFUSING..
 HADOOP 2.0 INTRODUCED YARN
(YET ANOTHER RESOURCE
NEGOTIATOR) AS HADOOP MOVED
FROM MAP REDUCE TO MORE
GENERIC MODEL, WITH ABILITY TO
SUPPORT APACHE SPARK AND
OTHER REAL TIME ENGINES.
 ITS BASICALLY MULTI THREADING –
MORE INSTANCES OF AN
APPLICATION MANAGED BY A
MASTER-MANAGER
 EXPAND THIS IDEA TO A CLUSTER. A
NUMBER OF APPLICATIONS MAY BE
SPAWNED BY A
CORRESPONDING APPLICATION
MASTER TASKS OR WORKERS ARE
RUN AND MANAGED BY
APPLICATION MASTER. APPLICATION
MASTER REQUESTS RESOURCE
MANAGER, WHO ALLOCATE
RESOURCES

16. TECHNOLOGIES ON HADOOP
 ECOSYSTEM – WHERE ALL TOOLS RESIDES IN UNION,
LIKE A POND ECOSYSTEM
 DATA PONDS, DATA LAKES AND DATA RESERVOIRS -
WHICH ARE REPLACING TRADITIONAL DATA
WAREHOUSES
 EFFICIENT BUSINESS INTELLIGENCES BY PREDICTION
AND FORECASTING
 ALGORITHMS FOR MACHINE LEARNING AND DEEP
LEARNING
 WEB NOTEBOOKS E.G.. ZEPPELIN
 DATABASES AND SQL – NOSQL DATABASES – NON-
RELATIONAL DATABASES – CASSANDRA, HBASE,
HIVEQL, SPARKQL

17. TECHNOLOGIES ON HADOOP
 OPEN APIS FOR OPERATING ON DOCUMENTS – OPEN
JSON
 STREAM PROCESSING – DATA STREAMING – SPARK
STREAMING, APACHE STORM, REAL-TIME, EVENT
BASED – EX: FACEBOOK LIVE, REAL TIME DATA
STREAMING FOR DATA LAKES
 MESSAGING PLATFORMS – APACHE KAFKA – USED BY
LINKEDIN FOR MESSAGING, ANALYTICS, WITHOUT
HAVING TO PERFORM ANY KIND OF DATA MOVEMENT
EX: GROUPME, FACEBOOK MESSENGER
 GLOBAL RESOURCE MANAGEMENT - THE ABILITY TO
PRESSURIZE THE RESOURCES (CPU, MEMORY,
BANDWIDTH) OF AN APPLICATION. - BUSINESSES CAN
GREATLY INCREASE THEIR MOMENTUM WHEN THEY
ARE ABLE TO USE THEIR ASSETS FOR CRITICAL
PROJECTS

18. DATA PREPARATION,
WRANGLING,ANALYSIS ON HADOOP
 VARIOUS ALGORITHMS FOR
 METADATA EXTRACTION
 FORMAT CONVERSION
 MDM IDENTIFICATION
 CROSS LINKING AMONG VARIOUS DATA
 CENTRALIZED INDEXING, TAGS, BUSINESS
METADATA, TECHNICAL METADATA
 TEXTUAL PATTERN RECOGNITION
 MOST OF THESE TOOLS ARE
SELF SERVICE ONES
 DATA INTEGRATION

19. BUSINESS INTELLIGENCE ON HADOOP
 SEARCH ENGINE TOOLS FOR OFFICE DATA
DIGGING OR MINING, WITH RANKED RESULTS
AND SUGGESTIONS. EXAMPLE – ELASTIC
SEARCH
 CUBING TOOLS – PREPARE DATA, COMPUTE
COMPLEX CALCULATIONS AND KEEP FOR
CONSUMPTION/REPORTING. EX: ATSCALE,
TRIFACTA
 STATISTICAL TOOLS – JMP AND SAS
 GEOSPATIAL TOOLS AND ACCESSORIES – EX:
ESRI SPECIAL FRAMEWORK
 TARGET MARKETING – EX: ELECTION
SOLICITING TO TARGET AUDIENCE OVER
SOCIAL MEDIA
 DECENTRALIZED ANALYTICS – ANALYSIS
DIVIDED ONTO MULTIPLE LOCATIONS,
MULTIPLE TALENTS AND THEN CONVERGE
INTO GOOD RESULTS