Azure ETL using Data factory service

1. #AzConfDev
Explore ETL using Azure Data Factory
Azure Data Factory Walkthrough

2. #GlobalAzure
Priyatharshini Sundaram
Software Engineer III @PayPal, MCT 5X Azure
Certified, 1X GCP Certified, Azure Architect
,Developer & DevOps Engineer Expert , Data
Engineer
https://www.linkedin.com/in/priyatharshini-s-75b002110/

3. Agenda
Components of a modern data Engineering & Warehousing
High-level design of ETL in Azure
Overview of Data Factory & Components
Demo - Create ADF Pipeline to Copy data into Synapse
Analytics

4. Components of a modern data Engineering &
Warehousing
Azure Data
Factory
Azure Data
Lake
Azure
Databricks
Azure
HDInsight
Azure Synapse
Analytics

5. High-level design of ETL in Azure

6. Azure Data Factory
• Code free ETL/ELT orchestrator for hybrid
data ingestion.
• Currently, It has more than 85 pre-built
connectors, which helps user to establish a
connection with source and destination from
various platforms like other cloud & on-
premise.
• Work is defined as a pipeline operation – runs
continuously as CD.
• Can either run in cloud or self-hosted network.
• You can perform data
processing/transformation using ADF though
recommended only for small/medium size
data.

7. Data Ingestion using Azure Data Factory
Types of data ingestion in Azure
ADF
Heterogenous
PolyBase
70
100
80
File based
SSIS
Heterogenous

8. Data Ingestion using Azure Data Factory (Cont…)
components of Azure Data Factory
Linked Service
Data
Lake Store
Azure
Databricks
Activities
Pipeline
Triggers
Dataset
@ Parameters
IR Integration
Runtime
CF Control
Flow

9. Customer’s Stories

10. Demo – Create ADF Pipeline to Copy data into
Synapse Analytics
Create ADF pipeline to perform ETL and store data into Synapse Analytics

11. Reference
If you’re interested to explore more ,Please check out below topics @Microsoft
Learn Modules (docs.microsoft.com/Learn)
• /certifications/azure-data-fundamentals
• /azure-data-fundamentals-explore-data-warehouse-analytics
• /azure-for-the-data-engineer
• /azure/data-factory/v1/data-factory-samples
• / azure/data-factory/v1/data-factory-build-your-first-pipeline

12. Q & A

13. Feedback

14. THANKYOU!
Priyatharshini Sundaram
https://www.linkedin.com/in/priyath
arshini-s-75b002110/