The document introduces big data, defining it as large and complex data sets that challenge traditional processing tools, highlighting its volume, velocity, and variety. It discusses Hadoop as a solution for managing big data through its architecture, including components like HDFS and MapReduce. The document concludes with an invitation for questions and feedback, emphasizing engagement in data management topics.

1. Pravin Singh
introducing
BIG DATA

2. WHAT THE HECK IS BIG DATA?
Any collection of data
sets so large and complex
that it becomes difficult to
process using current data
management tools or
traditional data processing
applications.
Volume
• Exceeds
physical limits
of vertical
scalability
Velocity
• Decision
window small
due to data
change rate
Variety
• Many different
formats make
integration
expensive

3. WHY SO LOW-COST?
Source: EMC

4. WHY SO LOW-COST?
Source: EMC

5. WHY SO FAST?
 Massive Parallel Processing
 Data Locality
 Optimized for write once – read many
 Sequential reads, not random access

6. Hello Hadoop!
You have an interesting name.

7. Hadoop Architecture
Source: Hortonworks

8. The Hadoop Zoo
HDFS
MapReduce
Pig Hive HCat Giraph Mahout
Zookeeper

9. The Real Simple Hadoop Architecture
MapReduce Engine
JobTracker
TaskTracker
1
TaskTracker
2
…
TaskTracker
N
HDFS Cluster
NameNode
DataNode
1
DataNode
2
…
DataNode
N

10. Hello HDFS!
Have we met before?

11. HDFS
My Data.txt
150 MB
64 MB
64 MB
22 MB
Name
Node
64 MB64 MB
64 MB64 MB
22 MB22 MB

12. Hello MapReduce!
Have you lost some weight?

13. MapReduce
Input
File
<Key, Value>
<Key, Value>
<Key, Value>
.
.
Shuffle
& Sort
<Key, Value>
<Key, Value>
<Key, Value>
.
.
Result

14. MapReduce
Big Data for
Dummies.txt
How many times the
words “Big data” and
“Hadoop” show up?

15. MapReduce
<Big data, 7>
<Hadoop, 4>
<Big data, 9>
<Hadoop, 6>
<Big data, 3>
<Hadoop, 8>
<Big data, 7>
<Big data, 9>
<Big data, 3>
<Hadoop, 4>
<Hadoop, 6>
<Hadoop, 8>
<Big data, 7, 9, 3>
<Hadoop, 4, 6, 8>
<Big data, 19>
<Hadoop, 18>

16. Let’s Play MapReduce!
’coz All Talk and No Play Makes Session a Dull Affair.

17. ?
Questions. Comments. Feedback.

18. See you at the (Data) Lake Next Time.
THANK YOU!