The document serves as a quick guide to data engineering, detailing various technologies such as Hadoop, HDFS, MapReduce, Hive, Pig, Spark, HBase, and Kafka. It explains how these tools facilitate data storage, processing, and real-time analytics, while addressing the challenges of handling large datasets. Additionally, it discusses the Lambda architecture, which integrates both real-time and batch processing to leverage historical and current data.

1. DATA ENGINEERING QUICK GUIDE
ASIM JALIS
GALVANIZE

2. BIG DATA

3. WHY HADOOP?
How can we create a supercomputer
Using cheap Linux boxes?

4. WHAT IS HADOOP?
Operating system for cluster of machines
Combines small weak computers
To create Big Data system
Unified disk and processing power

5. HADOOP

7. WHY HDFS?
How can we store a petabyte-sized file
Using cheap Linux boxes?

8. WHAT IS HDFS?
Split petabye file into 128 MB blocks
Distribute blocks across Hadoop cluster
Make 3 copies of each block for insurance

9. HDFS

10. WHY MAPREDUCE?
How can we process the data in HDFS
Without pulling it out and pushing the result back?

11. WHAT IS MAPREDUCE?
Send program to where the data is on HDFS
Process petabyte file by processing each block
Then combining the result

12. MAPREDUCE

13. WHY HIVE?
How can people who don’t know Java
Write MapReduce jobs?

14. WHAT IS HIVE?
Hive translates SQL to MapReduce jobs

15. HIVE
SELECT *
FROM sales
WHERE amount > 400;

16. HIVE

17. WHY PIG?
How can people who don’t know Java or SQL
Write MapReduce jobs?

18. WHAT IS PIG?
Pig translates PigLatin to MapReduce jobs
PigLatin is a scripting language comparable to SQL

19. PIG
high_sales =
FILTER sales_data
BY amount > 400;

20. PIG

21. WHY SPARK?
How can we make MapReduce faster
And the API less clunky?

22. WHAT IS SPARK?
Spark is like MapReduce
Spark has a cleaner API and is faster
Speed up because it saves intermediate results in memory

23. SPARK
sc.textFile("shakespeare.txt").
flatMap(line => line.split("W+")).
map(word => (word,1)).
reduceByKey((count1,count2) => (count1 + count2)).
saveAsTextFile("output")

24. SPARK

25. WHY SPARK SQL?
How can people who don’t know Scala, Python, or Java
Write Spark code?

26. WHAT IS SPARK SQL?
Spark SQL is like Hive for Spark
Hive translates SQL to MapReduce
Spark SQL translates SQL to Spark

27. SPARK SQL
SELECT *
FROM sales
WHERE amount > 400;

28. SPARK SQL

29. WHAT IS SPARK MLLIB?
Machine Learning algorithms on Spark
Analyze data to extract insights

30. WHAT IS MACHINE LEARNING?
Technique Question
Regression Predict revenue next month
Classification Is tumor cancerous or benign
Clustering Which customers are similar to each
other
Recommendation Which movie will you like

31. REAL-TIME TECHNOLOGIES

32. WHAT IS THE DIFFERENCE
BETWEEN REAL-TIME AND
BATCH?
Term Means Example
Real-
Time
Process data when
it arrives
Reject credit card
transaction
Batch Process data
periodically
Flag suspicious
transaction at night

33. BATCH
Processing Layer SQL Layer
MapReduce Hive, Pig
Spark Spark SQL

34. REAL-TIME
HBase
Kafka
Spark Streaming
Lambda Architecture

35. WHY HBASE?
How can we store petabytes of data on HDFS
And do fast read and writes like a database?

36. WHAT IS HBASE?
HBase is a NoSQL database on top of HDFS
Can store petabytes of data
Reads/writes much faster than traditional database and
HDFS

37. HBASE

38. WHY KAFKA?
How can we hold onto incoming data and not lose it
When we are getting a million messages per second?

39. WHAT IS KAFKA?
Kafka is TiVo for the cluster
It stores real-time data as it comes in
Can store a week of data
Queuing system for Hadoop cluster

40. KAFKA

41. WHY SPARK STREAMING?
How can we process data as it comes in
Instead of every night (using Spark or MapReduce)

42. WHAT IS SPARK STREAMING?
Spark Streaming is a library on top of Spark
It allows processing data as soon as it comes in
Sits in front of Kafka

43. SPARK STREAMING

44. WHY LAMBDA ARCHITECTURE?
How can we watch historical trends and what is
happening right now?
How can we show bestsellers from this year and from last
hour?

45. WHAT IS LAMBDA
ARCHITECTURE?
Big Data system which can handle both batch and real-
time
Uses historical data as well as real-time data
Best of both worlds

46. LAMBDA ARCHITECTURE

47. REVIEW

48. BATCH REVIEW
Technology Description
Hadoop Cluster operating system
HDFS Stores petabytes of data on 100s or 1000s of
machines
MapReduce Processes data in HDFS
Hive SQL MapReduce
Pig PigLatin MapReduce
Spark Faster MapReduce
Spark SQL SQL Spark

49. REAL-TIME REVIEW
Technology Description
HBase Fast NoSQL database on top of
HDFS
Kafka Queues incoming data into cluster
Spark Streaming Process in real-time
Lambda
Architecture
Combines real-time and batch