The document discusses Extract, Transform, Load (ETL) processes and how they are used with Hadoop. ETL involves extracting data from source systems, cleaning and transforming the data, and loading it into a destination database. Traditional ETL tools have limitations for big data analytics. Hadoop provides an alternative ETL architecture using tools like Apache Sqoop and Apache Flume that can handle large volumes of data in a distributed, scalable manner by extracting data from relational databases and loading it into Hadoop for processing.

1. Hadoop ETL
Aniket Bhosale
aniketbhosale2808@gmail.com

2. What is ETL?

3. ETL…
• Extract:
Extract data from source system and make it accessible for further processing.
• Clean
Ensures quality of data in warehouse.
• Transform:
Applies a set of rules on data flowing from source to data.
• Load
Load in destination database.
• Staging

4. Benefits & Limitations
of Traditional ETL
• Benefit
• Helps to address issues like Change Management, Slowly changing dimensions, inserts,
updates, etc.
• Limitations
• Early in process someone has to decide what data is important, who can access it, what
should be updated.
• Original raw data is not stored and can’t be retrieved.

5. ETL Bottleneck in Big Data Analytics
• Business benefits of analyzing of Big Data can be significant.

6. Hadoop Architecture

7. Traditional ETL

8. ETL Tools
• For Traditional ETL tools like Teradata Warehouse Builder, DataStage are
used.
• For ETL with Hadoop,
Apache Sqoop, Apache Flume are used .

9. Apache SQOOP
• Sqoop is designed to transfer data between Hadoop and Relational
Databases.
• You can use Sqoop to import data from a relational database management
system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed
File System (HDFS), transform the data in Hadoop MapReduce, and then
export the data back into an RDBMS.

10. Traditional ETL
Application DataData
T

11. What is Sqoop ?
• A different paradigm
Data
Application
Data

12. • A very scalable different paradigm
Data
Data
Application
Data
Application
Data

13. • Where did the transform go ?
Application
Data
TTT TTT TTT TTT

14. Any Questions ?

15. References
• https://www.ibm.com/developerworks/library/bd-hivetool/index.html
• https://www.csgsolutions.com/blog/what-is-etl/
• https://software.intel.com/sites/default/files/article/402274/etl-big-data-
with-hadoop.pdf