This document provides an overview of Hadoop and several related big data technologies. It begins by defining the challenges of big data as the 3Vs - volume, velocity and variety. It then explains that traditional databases cannot handle this type and scale of unstructured data. The document goes on to describe how Hadoop works using HDFS for storage and MapReduce as the programming model. It also summarizes several Hadoop ecosystem projects including YARN, Hive, Pig, HBase, Zookeeper and Spark that help to process and analyze large datasets.

1. HADOOP (A BIG DATA INITIATIVE)
-Mansi Mehra

2. AGENDA
 Defining the problem – 3Vs
 Why traditional storages don’t work
 How does Hadoop work?
 HDFS (Hadoop 1.0 Vs 2.0)
 YARN (2.0)- Yet Another Resource Negotiator
 Map Reduce
 When we don’t know how to code
 Hive (Overview)
 PIG (Overview)
 Hbase (Overview)
 Zookeeper (Overview)
 Spark (Overview)

3. DEFINING THE PROBLEM – 3VS
 Volume - Lots and lots of data
 Datasets are so large and complex
 Cannot use relational database
 Challenges: capture, curation, storage, search, sharing,
transfer, analysis and visualization.

4. DEFINING THE PROBLEM – 3V (CONTD.)
 Velocity - Huge amounts of data generated at
incredible speed
 NYSE generates about 1 TB of new trade data per day
 AT&T anonymized Call Detail Records (CDRs) top at
around 1 GB per hour.
 Variety - Differently formatted data sets from
different sources
 Twitter keeps tracks of tweets, Facebook produces
posts and likes data, Youtube streams videos)

5. WHY TRADITIONAL STORAGES DON’T WORK
 Unstructured data is exploding, not much of data
produced has relational nature.
 No redundancy
 High computational cost
 Capacity limit for structured data (costly hardware)
 Expensive License
Data type Nature
XML Semi-structured
Word docs, PDF files etc. Unstructured
Email body Unstructured
Data from Enterprise Systems
(ERP, CRM etc.)
Structured

6. HOW DOES HADOOP WORK?

7. HDFS (HADOOP 1.0 VS 2.0)

8. HDFS (HADOOP 2.0)

9. YARN (2.0)- YET ANOTHER RESOURCE
NEGOTIATOR
 Computing framework for Hadoop.
 YARN has Resource Manager-
 Manages and allocates cluster resources
 Improves performance and Quality of Service

10. MAP REDUCE
 Programming model in Java
 Work on large amounts of data
 Provides redundancy & fault tolerance
 Runs the code on each data node

11. MAP REDUCE (CONTD.)
 Steps for Map Reduce:
 Read in lots of data
 Map: extract something you care about from each
record/line.
 Shuffle and sort
 Reduce: aggregate, summarize, filter or transform
 Write results.

12. WHEN WE DON’T KNOW HOW TO CODE

13. HIVE (OVERVIEW)
 Data warehouse infrastructure built on top of Hadoop
 Compile SQL queries as MapReduce jobs and run the job in
the cluster.
 Brings structure to unstructured data
 Key Building Principles:
 Structured data with rich data types (structs, lists and maps)
 Directly query data from different formats (text/binary) and file
formats (Flat/sequence).
 SQL as a familiar programming tool and for standard analytics
 Types of applications:
 Summarization: Daily/weekly aggregations
 Ad hoc analysis
 Data Mining
 Spam detection
 Many more ….

14. PIG (OVERVIEW)
 High level dataflow language
 Has its own syntax (Preferable for people with
programming background)
 Compiler that produces sequences of MapReduce
programs.
 Structure is agreeable to substantial parallelization.
 Key properties of PIG:
 Ease of programming: Trivial to achieve parallel execution of
simple and parallel data analysis tasks
 Optimization opportunities: allows user to focus on semantics
rather than efficiency.
 Extensibility: Users can create their own functions to do
special purpose processing.

15. HBASE (OVERVIEW)
 HBase is a distributed column-oriented data store built
on top of HDFS.
 Data is logically organized into tables, rows and
columns.
 HDFS is good for batch processing (scan over big files).
 Not good for record lookup.
 Not good for incremental addition of small batches.
 Not good for updates.
 HBase is designed to efficiently address the above
points
 Fast record lookup
 Support for record level insertion
 Support for updates (not in place).
 Updates are done by creating new versions of values.

16. ZOOKEEPER (OVERVIEW)
 Zookeeper is a distributed, open source
coordination service for distributed applications.
 Exposes simple set of primitives that distributed
applications can build upon to implement higher
level services for synchronization, configuration
maintenance, and groups and naming.
 Coordination services are notoriously hard to get
right. They are prone to errors like race conditions
and deadlock.
 The motivation behind zookeeper is to relieve
distributed applications the responsibility of
implementing coordination services from scratch.

17. SPARK (OVERVIEW)
 Motivation : MapReduce programming model transform data
flowing from stable storage to stable storage (disk to disk).
 Acyclic data flow is a powerful abstraction, but not efficient for
applications that repeatedly reuse a working set of data.
 Iterative algorithms
 Interactive data mining
 Spark makes working sets a first-class concept to efficiently
support these applications.
 Goal:
 To provide distributed memory abstractions for clusters to support
apps with working sets.
 Retain the attractive properties of map reduce.
 Fault tolerance
 Data locality
 Scalability
 Augment data flow model with “resilient distributed datasets”
(RDDs)

18. SPARK (OVERVIEW CONTD.)
 Resilient distributed datasets (RDDs)
 Immutable collections partitioned across cluster that can
be rebuilt if a partition is lost.
 Created by transforming data in stable storage using
data flow operators (map, filter, group-by, ..)
 Can be cached across parallel operations.
 Parallel operations on RDDs.
 Reduce, collect, count, save, …..
 Restricted shared variables
 Accumulators, broadcast variables.

19. SPARK (OVERVIEW CONTD)
 Fast map reduce like engine
 Uses in memory cluster computing
 Compatible with Hadoop storage API.
 Has API’s written in Scala, Java, Python.
 Useful for large datasets and iterative algorithms.
 Up to 40x faster than MapReduce.
 Support for:
 Spark SQL : Hive on Spark
 Mlib : Machine learning library
 Graphx : Graph processing.

20. THANK YOU!