Data Factory in Microsoft Fabric (Presented at the Microsoft BI Professionals Denmark meeting #82 on April 15th, 2024)

1. Data Factory
in Microsoft Fabric
Cathrine Wilhelmsen • MsBIP #82 • April 15th, 2024

2. @cathrinew
cathrinew.net

3. Microsoft Fabric

4. What is MicrosoftFabric?
End-to-end Analytics:
• Software-as-a-Service
• Lake-centric architecture
• Role-tailored tools

5. Data
Factory
Data
Engineering
Data
Warehouse
Data
Science
Real-Time
Analytics
Power
BI
Data
Activator
OneLakestoring onecopy ofall yourdata inanopenformat
Built-insecurity, governanceand compliance

6. Data Factory

7. The Evolution of Data Factory
2014 2017 2019 2023
2015

8. What is Data Factory in Fabric?
No-code / low-code data integration experience with:
• Scale and power of Azure Data Factory
• Ease-of-useof Power Query
• Intelligenceof Copilot

9. Ingest Data Transform Data

10. Orchestrate Workflows
Ingest Data Transform Data

11. Orchestrate Workflows
Ingest Data Transform Data

12. Orchestrate Workflows
Pipelines
Ingest Data
Dataflows / Pipelines
Transform Data
Dataflows

13. Orchestrate Workflows
Pipelines
Ingest Data
Dataflows / Pipelines
Transform Data
Dataflows

14. Pipelines

15. What are Pipelines?
Pipelines are what
you execute or run
They define workflows:
what to do in which order

16. What are Activities?
Activities are individual
steps inside pipelines
They can be chained
or run in parallel

17. Copy Data Activity
Source Destination

18. Copy Data Activity: Complex Data
Source
Convert
File Formats
Convert
Data Types
Map
Columns
Flatten
Hierarchies
Split &
Merge Files
Compress &
Decompress
Destination

19. What are Connections?
Connections define how
to connect and authorize
Choose between built-in
connection to workspace
or 36+ external sources

20. What are Runs?
Runs are individual
executions of pipelines
Run on-demand or
on a defined schedule

21. Let’s demo!

22. Orchestrate Workflows
Pipelines
Ingest Data
Dataflows / Pipelines
Transform Data
Dataflows

23. Orchestrate Workflows
Pipelines
Ingest Data
Dataflows / Pipelines
Transform Data
Dataflows

24. Dataflows

25. What are Dataflows (Gen2)?
Power Query scaled up and out:
• No-code / low-code transformations
• 160+ source connectors
• 300+ transformation functions
• New destinations in Microsoft Fabric and Azure
• New Fast Copy capability

26. Let’s just demo!

27. Decision Guide for Ingest and Transform
Copy Activity Dataflow
Use cases:
Data Ingestion
Data Conversion
Data Ingestion
Data Transformation
Userinterface: Wizard,Canvas Power Query
Sources: 30+ connectors 160+connectors
Destinations: 18+connectors
Workspace:Lakehouse / Warehouse
External:Azure SQL Database,Azure Data
Explorer,Azure SynapseAnalytics
Transformations: Lightweight conversions 300+ transformationfunctions

28. Questions?

29. Thank you!
hi@cathrinew.net cathrinew.net @cathrinew