Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apache hadoop
1. Apache Hadoop
- Large Scale Data Processing
Sharath Bandaru & Sai Dinesh Koppuravuri
Advanced Topics Presentation
ISYE 582 :Engineering Information Systems
2. Overview
Understanding Big Data
Structured/Unstructured Data
Limitations Of Existing Data Analytics Structure
Apache Hadoop
Hadoop Architecture
HDFS
Map Reduce
Conclusions
References
3. Understanding Big Data
Big Data
Is creating
Large And
Growing Files
Measured in:
Petabytes (10^12)
Terabytes (10^15)
Which is largely
unstructured
6. Challenges posed by Big Data
Velocity
Volume
Variety
400 million tweets in a day on Twitter
1 million transactions by Wal-Mart every hour
2.5 peta bytes created by Wal-Mart
transactions in an hour
Videos, Photos, Text messages, Images,
Audios, Documents, Emails, etc.,
7. Limitations Of Existing Data Analytics Architecture
BI Reports + Interactive Apps
RDBMS (aggregated data)
ETL Compute Grid
Storage Only Grid ( original raw data )
Collection
Instrumentation
Moving Data To
Compute Doesn’t Scale
Can’t Explore Original
High Fidelity Raw Data
Archiving=
Premature Data
Death
8. So What is Apache ?
• A set of tools that supports running of applications on big data.
• Core Hadoop has two main systems:
- HDFS : self-healing high-bandwidth clustered storage.
- Map Reduce : distributed fault-tolerant resource management
and scheduling coupled with a scalable data programming
abstraction.
10. The Key Benefit: Agility/Flexibility
Schema-on-Write (RDBMS): Schema-on-Read (Hadoop):
• Schema must be created before any data
can be loaded.
• An explicit load operation has to take
place which transforms data to DB
internal structure.
• New columns must be added explicitly
before new data for such columns can be
loaded into the database
• Data is simply copied to the file store,
no transformation is needed.
• A SerDe (Serializer/Deserlizer) is applied
during read time to extract the required
columns (late binding).
• New data can start flowing anytime and
will appear retroactively once the SerDe is
updated to parse it.
• Read is Fast
• Standards/Governance
• Load is Fast
• Flexibility/Agility
Pros
11. Use The Right Tool For The Right Job
Relational Databases: Hadoop:
Use when:
• Interactive OLAP Analytics (< 1 sec)
• Multistep ACID transactions
• 100 % SQL compliance
Use when:
• Structured or Not (Flexibility)
• Scalability of Storage/Compute
• Complex Data Processing
17. HDFS: Hadoop Distributed File System
• A given file is broken into blocks (default=64MB), then blocks are replicated across
cluster(default=3).
1
2
3
4
5
HDFS
3
4
5
1
2
5
1
3
4
2
4
5
1
2
3
Optimized for :
• Throughput
• Put/Get/Delete
• Appends
Block Replication for :
• Durability
• Availability
• Throughput
Block Replicas are distributed across servers
and racks
18. Fault Tolerance for Data
Task
Tracker
Job
Tracker
Name
Node
Data
Node
Master
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Slaves
HDFS
19. Fault Tolerance for Processing
Task
Tracker
Job
Tracker
Name
Node
Data
Node
Master
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Slaves
Map Reduce
20. Fault Tolerance for Processing
Task
Tracker
Job
Tracker
Name
Node
Data
Node
Master
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Slaves
Tables are backed up
22. Understanding the concept of Map Reduce
Mother
Sam
An Apple
• Believed “an apple a day keeps a doctor away”
The Story Of Sam
23. Understanding the concept of Map Reduce
• Sam thought of “drinking” the apple
He used a to cut the
and a to make juice.
24. Understanding the concept of Map Reduce
Next day
• Sam applied his invention to all the fruits he could find in
the fruit basket
(map ‘( )’)
(reduce ‘( )’) Classical Notion of Map Reduce
in Functional Programming
A list of values mapped into
another list of values, which
gets reduced into a single value
25. Understanding the concept of Map Reduce
18 Years Later
• Sam got his first job in “Tropicana” for his expertise in
making juices.
Now, it’s not just one basket
but a whole container of fruits
Also, they produce a list of
juice types separately
NOT ENOUGH !!
But, Sam had just ONE
and ONE
Large data and list of
values for output
Wait!
26. Understanding the concept of Map Reduce
Brave Sam
(<a, > , <o, > , <p, > , …)
Each input to a map is a list of <key, value> pairs
Each output of a map is a list of <key, value> pairs
(<a’, > , <o’, > , <p’, > , …)
Grouped by key
Each input to a reduce is a <key, value-
list> (possibly a list of these, depending
on the grouping/hashing mechanism)
e.g. <a’, ( …)>
Reduced into a list of values
Implemented parallel version of his innovation
27. Understanding the concept of Map Reduce
• Sam realized,
– To create his favorite mix fruit juice he can use a combiner after the reducers
– If several <key, value-list> fall into the same group (based on the
grouping/hashing algorithm) then use the blender (reducer) separately on
each of them
– The knife (mapper) and blender (reducer) should not contain residue after use
– Side Effect Free
Source: (Map Reduce, 2010).
28. Conclusions
• The key benefits of Apache Hadoop:
1) Agility/ Flexibility (Quickest Time to Insight)
2) Complex Data Processing (Any Language, Any Problem)
3) Scalability of Storage/Compute (Freedom to Grow)
4) Economical Storage (Keep All Your Data Alive Forever)
• The key systems for Apache Hadoop are:
1) Hadoop Distributed File System : self-healing high-bandwidth
clustered storage.
2) Map Reduce : distributed fault-tolerant resource management
coupled with scalable data processing.
29. References
• Ekanayake, S. (2010, March). Map Reduce : The Story Of Sam. Retrieved April 13, 2013,
from http://esaliya.blogspot.com/2010/03/mapreduce-explained-simply-as-story- of.html.
• Jeffrey Dean and Sanjay Ghemawat. (2004, December). Map Reduce : Simplified Data
Processing on Large Clusters.
• The Apache Software Foundation. (2013, April). Hadoop. Retrieved April 19, 2013, from
http://hadoop.apache.org/.
• Isabel Drost. (2010, February). Apache Hadoop : Large Scale Data Analysis made Easy.
retrieved April 13, 2013, from http://www.youtube.com/watch?v=VFHqquABHB8.
• Dr. Amr Awadallah. (2011, November). Introducing Apache Hadoop : The Modern Data
Operating System. Retrieved April 15, 2013, from
http://www.youtube.com/watch?v=d2xeNpfzsYI