This ppt shows the process of automating a data pipeline to: Load data from AWS S3 bucket into a Snowflake table using Airflow running in an AWS EC2 instance. Then data is then visualized using Tableau. Technologies used: Airflow, Python, SQL, AWS (EC2, S3), Snowflake and Tableau.

1. S E T T I N G U P A D A T A P I P E L I N E F O R
A N A L Y S I S O F S A L E S D A T A O F V A R I O U S
T R A N S P O R T V E H I C L E S

2. The upcoming slides show the process of automating a data pipeline to:
• Load data from amazon S3 bucket into a snowflake table using Airflow running in an AWS EC2 instance.
• Then data is then visualized using Tableau.
• Technologies used: Airflow, Python, SQL, AWS (EC2, S3), Snowflake and Tableau.

3. A R C H I T E C T U R E
• Use S3KeySensor package to mark the presence of source file being present in S3 bucket.
• Create staging area for file to land in snowflake.
• Create table schema to hold the records.
• Copy the records into from S3 to Snpowflake.
• Then data is then visualized using Tableau.
• Technologies used: Airflow, Python, SQL, AWS (EC2, S3), Snowflake and Tableau.

4. S T E P 1 : I N S T A L L P A C K A G E S & D E P E N D E N C I E S

5. S T E P 2 : S E T - U P E C 2 , S 3 & U P L O A D S O U R C E F I L E

6. S T E P 3 : C R E A T E D A G

7. S T E P 3 : A D D T A S K 1 T O D A G , T O S E E K T H E S O U R C E F I L E I N S 3

8. S T E P 4 : S E T - U P S N O W F L A K E D B

9. S T E P 5 : A D D T A S K 2 T O D A G - T O C R E A T E T A B L E S T R U C T U R E I N D B

10. S T E P 6 : A D D T A S K 3 T O D A G - T O C O P Y D A T A F R O M S 3 T O D B

11. S T E P 6 : T R I G G E R T H E D A G , T O R U N A L L T A S K S S E Q U E N T I A L L Y

12. V E R I F Y : D A T A G O T L O A D E D I N T A B L E

13. C R E A T E D D A S H B O A R D U S I N G T A B L E A U