#MIcrosoft #Azure #DataFactory

1. ETL Patterns in the Cloud with
Azure Data Factory
Mark Kromer
Senior Program Manager
Microsoft Azure Data Management
@kromerbigdata

2. ETL Patterns in the Cloud
Important factors for success
1. What is ETL?
• More than Extract, Transform, Load
• Scheduling, Monitoring, Maintenance, Source Control, CI/CD, Operationalize
2. Platform as a Service (ADF) vs. Infrastructure as a Service (IaaS/SSIS)
• Self-managed vs. Provider-Managed
3. ELT or ETL?
• Difference is primarily highly-parsed semantics
• However: In the cloud, common pattern == stage data in low-cost, inexpensive storage
4. Not typically performant to process data in-flight
• Particularly crossing boundaries (on-prem, vnets, data centers, regions)
5. Scale is very important in Cloud ETL
• Cloud projects assume elastic scale. ETL is not immune to this expectation.
6. Flexible Schema is very important in Cloud ETL
1. Assume “Big Data tenets” aka “data chaos”: Your data sources will change shape, size and
volume. Often!

3. Azure Data Factory
Workflow Pipelines/Control Flow

4. Built-in source
control support

7. ADF Integration Runtime
Activity Dispatch/Monitor
Data Movement
SSIS Package Execution

8. Azure Data Factory Service
Cloud Apps, Svcs & DataOn Premises Apps & Data
UX & SDK

9. Azure (US West)
Public Internet Border
HP Inc (Global)
HP Prod Firewall Border
HP Hadoop Cluster
Integration Fabric On-prem
Data Factory
(Orchestration
micro-service)
443
Storage (Azure)
443
SQL Data
Warehouse
UAM Server
443
ADF Foo On-prem
“IR”
Customer 1
Customer 1 firewall border
Azure Data Factory “Integration Runtime” deployed on premises for
transformation and then moved to cloud

10. SSIS in ADF

12. on premises
in Azure
Deployment via SSMS
Once connected, you can deploy
projects/packages to SSIS PaaS from your
local file system/SSIS on premises

13. Execution via SSMS
You can select some packages to execute
on SSIS PaaS

14. Monitoring via SSMS
on premises
in Azure
You can see package execution
error messages

15. Execute SSIS Packaged in ADF Pipeline

16. ADF Mapping Data Flows

17. What is ADF Mapping Data Flow?
Transform Data, At Scale, in the Cloud,
Zero-Code
Cloud-first, scale-out ELT
Code-free dataflow pipelines
Serverless scale-out transformation
execution engine
Maximum Productivity for Data
Engineers
Does NOT require understanding of Spark /
Scala / Python / Java
Resilient Data Transformation Flows
Built for big data scenarios with
unstructured data requirements
Operationalize with Data Factory
scheduling, control flow and monitoring

18. Code-free Data Transformation At Scale
Does not require understanding of Spark, Big Data Execution
Engines, Clusters, Scala, Python …
Focus on building business logic and data transformation
Data cleansing
Aggregation
Data conversions
Data prep
Data exploration

19. ADF Data Flow Workstream
Stage Data in Azure
(ADLS, Blob, SQL
DB/DW)
Transform Data in
Visual Data Flow
Land Data in Azure
Staging Area (ADLS,
Blob, SQL DB/DW)

20. Build your logical data flows adding data
transformations in a guided experience

21. Microsoft Azure Data Factory Continues to Extend Data Flow
Library with a Rich Set of Transformations and Expression
Functions

22. Debug mode provides row-level context
and visible results in inspector pane

23. Interactive Expression Builder – Build data transform
expressions, not Spark code

24. Deep Monitoring Introspection of Data Transformations

25. Debug Data Flows with Data Preview and Data Sampling

26. Cloud ETL Patterns
with ADF

28. MODEL & SERVE
Azure Analysis ServicesAzure SQL Data
Warehouse
Power BI
Modernize your enterprise data warehouse at scale
A Z U R E D A T A F A C T O R Y
On-premises data
Oracle, SQL, Teradata,
fileshares, SAP
Cloud data
Azure, AWS, GCP
SaaS data
Salesforce, Workday,
Dynamics
INGEST STORE PREP & TRAIN
Azure Data Factory Azure Blob Storage
Azure Databricks
Polybase
Microsoft Azure also supports other Big Data services like Azure HDInsight, Azure SQL Database and Azure Data Lake to allow customers to tailor the above architecture to meet their unique needs.
Orchestrate with Azure Data Factory

29. Lift your SQL Server Integration Services (SSIS) packages to Azure
On-Premise data sources
SQL DB Managed Instance
SQL Server
VNET
Azure Data Factory
SSIS Cloud ETL
SSIS Integration Runtime
Cloud data sources
Cloud
On-premises
Microsoft
SQL Server
Integration Services

30. Author, orchestrate and monitor with Azure Data Factory
Hybrid and Multi-Cloud Data Integration
Azure Data Factory
PaaS Data Integration
DATA SCIENCE
AND MACHINE
LEARNING
MODELS
ANALYTICAL
DASHBOARDS
USING POWER BI
DATA DRIVEN
APPLICATIONS
On-Prem SaaS Apps Public Cloud

31. Nightly ETL Data Flows - Codeless

32. Build Resilient Data Flows with Schema Drift
Handling of Flexible Schemas

33. Data Engineer derives columns using template expression
patterns based on name and type matching. No need to define
static field names.

34. Sink all incoming fields along with new
derived field

35. Slowly Changing Dimension Scenario

36. Load Star Schema DW Scenario

37. Data Lake Data Science Scenario

38. Sponsors