Apache Hadoop In Theory And Practice

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Apache Hadoop
In Theory And Practice
Adam Kawa

Data Engineer @ Spotify

Why Data?
Get insights to offer a better product
“More data usually beats better algorithms”
Get of insights to make bette...

What Is Challenging?
Store data reliably
Analyze data quickly
Cost-effective way
Use expressible and high-level language

Fundamental Ideas
A big system of machines, not a big machine
Failures will happen
Move computation to data, not data to c...

Apache Hadoop
An open-source Java software
Storing and processing of very large data sets
A clusters of commodity machines...

Apache Hadoop
Two main components:
HDFS - a distributed file system
MapReduce – a distributed processing layer

Many other...

Component

HDFS

The Purpose Of HDFS
Store large datasets in a distributed, scalable and fault-tolerant way
High throughput
It is like a bi...

HDFS Mis-Usage
Do NOT use, if you have
Low-latency requests
Random reads and writes
Lots of small files
Then better to con...

Splitting Files And Replicating Blocks
Split a very large file into smaller (but still large) blocks
Store them redundantl...

Spiting Files Into Blocks

Today, 128MB or 256MB is recommended

The default block size is 64MB
Minimize the overhead of a...

Replicating Blocks
The default replication factor is 3
It can be changed per a file or a directory
It can be increased for...

Master And Slaves
The Master node keeps and manages all metadata information
The Slave nodes store blocks of data and serv...

Classical* HDFS Cluster
*no NameNode HA, no HDFS Replication
Manages metadata

Does some
“house-keeping”
operations for
Na...

HDFS NameNode
Performs all the metadata-related operations
Keeps information in RAM (for fast lookup)
The filesystem tree
...

HDFS DataNode
Stores and retrieves blocks of data
Data is stored as regular files on a local filesystem (e.g. ext4)
e.g. b...

HDFS Secondary NameNode
NOT a failover NameNode
Periodically merges a prior snapshot (fsimage) and editlog(s) (edits)
Fetc...

Exemplary HDFS Commands
hadoop fs -ls -R /user/kawaa
hadoop fs -cat /toplist/2013-05-15/poland.txt | less
hadoop fs -put l...

Reading A File From HDFS
Block data is never sent through the NameNode
The NameNode redirects a client to an appropriate D...

HDFS Network Topology
Network topology defined by an administrator in a supplied script
Convert IP address into a path to ...

HDFS Block Placement
Pluggable (default in BlockPlacementPolicyDefault.java)
1st replica on the same node where a writer i...

HDFS Balancer
Moves block from over-utilized DNs to under-utilized DNs
Stops when HDFS is balanced
the utilization of
diff...

Questions

HDFS

HDFS Block
Question
Why a block itself does NOT know
which file it belongs to?

HDFS Block
Question
Why a block itself does NOT know
which file it belongs to?
Answer
Design decision → simplicity, perfor...

HDFS Metadata
Question
Why NN does NOT store information
about block locations on disks?

HDFS Metadata
Question
Why NN does NOT store information
about block locations on disks?
Answer
Design decision → simplici...

HDFS Replication
Question
How many files represent a block replica in HDFS?

HDFS Replication
Question
How many files represent a block replica in HDFS?
Answer
Actually, two files:
The first file for...

HDFS Block Placement
Question
Why does NOT the default block placement strategy take the disk
space utilization (%) into a...

HDFS Block Placement
Question
Why does NOT the default block placement strategy take the disk
space utilization (%) into a...

Facts

HDFS

HDFS And Local File System
Runs on the top of a native file system (e.g. ext3, ext4, xfs)
HDFS is simply a Java applicatio...

HDFS Data Integrity
HDFS detects corrupted blocks
When writing
Client computes the checksums for each block
Client sends c...

HDFS NameNode Scalability
Stats based on Yahoo! Clusters
An average file ≈ 1.5 blocks (block size = 128 MB)
An average fil...

HDFS NameNode Performance
Read/write operations throughput limited by one machine
~120K read ops/sec
~6K write ops/sec
Map...

HDFS Main Limitations
Single NameNode
Keeps all metadata in RAM
Performs all metadata operations
Becomes a single point of...

HDFS Main Improvements
Introduce multiple NameNodes in form of:
HDFS Federation
HDFS High Availability (HA)
Find

More:

h...

In practice

HDFS

Problem
DataNode can not start on a server for some reason

Usually it means some kind of disk failure
$ ls /disk/hd12/
ls: reading directory /disk/hd12/: Input/output error
org.apac...

It was exciting this see stuff breaking!

In practice

HDFS

Problem

A user can not run resource-intensive Hive queries
It happened immediately after expanding the cluster

Description
The queries are valid
The queries are resource-intensive
The queries run successfully on a small dataset
But t...

The NameNode is throwing thousands of warnings and exceptions
14592 times only during 8 min (4768/min in a peak)

Normally
Hadoop is a very trusty elephant
The username comes from the client machine (and is not verified)
The groupname i...

Possible Fixes
Create an user account on the NameNode server (dirty, insecure)
Use AD/LDAP for a user-group resolution
had...

Our Fix
We decided to use LDAP for a user-group resolution
However, LDAP settings in Hadoop did not work for us
Because po...

Lesson Learned
Know who is going to use your cluster
Know who is abusing the cluster (HDFS access and MR jobs)
Parse the N...

Component

MapReduce

MapReduce Model
Programming model inspired by functional programming
map() and reduce() functions processing <key, value> ...

Map And Reduce Functions

Map And Reduce Functions Counting Word

MapReduce Job
Input data is divided into
splits and converted into
<key, value> pairs

Invokes map() function
multiple tim...

MapReduce Example: ArtistCount
Artist, Song, Timestamp, User

Key is the offset of the line
from the beginning
of the line...

MapReduce Example: ArtistCount
map(Integer key, EndSong value, Context context):
context.write(value.artist, 1)
reduce(Str...

MapReduce Combiner
Make sure that the Combiner
combines fast and enough
(otherwise it adds overhead only)

Data Locality in HDFS and MapReduce
By default, three replicas
should be available somewhere
on the cluster

Ideally, Mapp...

MapReduce Implementation
Batch processing system
Automatic parallelization and distribution of computation
Fault-tolerance...

“Classical” MapReduce Daemons
Keeps track of TTs,
schedules jobs and tasks executions

Runs map and reduce tasks,
Reports ...

JobTracker Reponsibilities
Manages the computational resources
Available TaskTrackers, map and reduce slots
Schedules all ...

TaskTracker Reponsibilities
Runs map and reduce tasks
Reports to JobTracker
Heartbeats saying that it is still alive
Numbe...

Apache Hadoop Cluster
It can consists of 1, 5, 100 and 4000 nodes

MapReduce Job Submission
They are copied with
a higher replication
factor
(by default, 10)

Image source: “Hadoop: The Def...

MapReduce: Sort And Shuffle Phase
Map phase

Reduce phase

Other maps tasks
Image source: “Hadoop: The Definitive Guide” b...

MapReduce: Partitioner
Specifies which Reducer should get a given <key, value> pair
Aim for an even distribution of the in...

Speculative Execution
Scheduling a redundant copy of the remaining, long-running task
The output from the one that finishe...

Speculative Execution
Enable, if tasks often experience “external” problems e.g. hardware
degradation (disk, network card)...

Facts

MapReduce

Java API
Very customizable
Input/Output Format, Record Reader/Writer,
Partitioner, Writable, Comparator …

Unit testing wi...

MapReduce Job Configuration
Tons of configuration parameters
Input split size (~implies the number of map tasks)
Number of...

Questions

MapReduce

MapReduce Input <Key, Value> Pairs
Question
Why each line in text file is, by default, converted to
<offset, line> instead...

MapReduce Input <Key, Value> Pairs
Question
Why each line in text file is, by default, converted to
<offset, line> instead...

In practice

MapReduce

“I noticed that a bottleneck seems to be coming from the map tasks.
Is there any reason that we can't open any of the allo...

This may change again soon ...

We initially started with 60/40
But today we are closer to something like 70/30
Time Spend...

We are currently introducing a new feature to Luigi
Automatic settings of
Maximum input split size (~implies the number of...

The current PoC ;)
type

# map

# reduce

avg map time

avg reduce time

job execution time

old_1

4826

25

46sec

1hrs,...

In practice

MapReduce

Problem
Surprisingly, Hive queries are running extremely long
Thousands task are constantly being killed

Only 1 task failed,
2x more task
were killed
than
were completed

Logs show that the JobTracker gets a request to kill the tasks
Who actually can send a kill request?
User (using e.g. mapr...

Key Observations
Killed tasks came from ad-hoc and resource-intensive Hive queries
Tasks are usually killed quickly after ...

Eureka!
FairScheduler has a license to kill!
Preempt the newest tasks
in an over-share pool
to forcibly make some room
for...

Hive pool was running over its minimum and fair shares
Other pools were running under their minimum and fair shares
So tha...

Possible Fixes
Disable the preemption
Tune minimum shares based on your workload
Tune preemption timeouts based on your wo...

Lessons Learned

A scheduler should NOT be considered as the ''black-box''
It is so easy to implement a long-running Hive ...

More About

Hadoop Adventures
http://slidesha.re/1ctbTHT

In the reality

Hadoop is fun!

Questions?

Stockholm and Sweden

Want to join the band?
Check out spotify.com/jobs or @Spotifyjobs
for more information
kawaa@spotify.com
HakunaMapData.com

Thank you!

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Hadoop Adventures At Spotify (Strat... by Adam Kawa 41580 views
Hadoop Operations Powered By ... Ha... by Adam Kawa 26900 views

View on Slideshare

1 of 96 Ad

View on Slideshare

1 of 96 Ad