Download as PDF, PPTX
Uploaded byCathrine Wilhelmsen
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PASS Data Community Summit 2023)
The document discusses visual data transformation using Azure Data Factory and Azure Synapse Analytics, highlighting data flows as a low-code tool for large data manipulation. It covers capabilities, use cases, orchestration, and pricing models for data flows, comparing them to Power Queries. The session aims to simplify data processing without requiring extensive programming knowledge, offering demos and lessons learned.
More Related Content
Similar to Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PASS Data Community Summit 2023)
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PASS Data Community Summit 2023)
- 1. Visually Transform Datain Azure Data Factory or Azure Synapse Analytics Cathrine Wilhelmsen
- 2. #PASSDataSummit Data Warehousing BigData and Analytics
- 3. #PASSDataSummit Data Warehousing BigData and Analytics
- 4. #PASSDataSummit Data Warehousing BigData and Analytics
- 5. Visually Transform Datain Azure Data Factory or Azure Synapse Analytics Cathrine Wilhelmsen She / Her Solutions Architect Evidi
- 6. #PASSDataSummit Learning Pathway: The Battleof the Data Transformation Tools Visually Transform Data in Azure Data Factory or Azure Synapse Analytics Cathrine Wilhelmsen Power up Your Transformations Game with Power Query in Power BI and Fabric Marthe Moengen Azure Databricks and Notebooks in Fabric – A Transformation Dream Come True? Emilie Rønning The Battle of the Data Transformation Tools Cathrine Wilhelmsen, Marthe Moengen, Emilie Rønning
- 7. #PASSDataSummit Session Description Do youneed to clean, convert, aggregate, prepare, or transform large amounts of data, but don't want to spend your time learning a new programming language or writing lots of code? If so, Data Flows in Azure Data Factory or Azure Synapse Analytics could be the tool for you! By using Data Flows, you can build both simple and complex data transformations in a visual editor. These Data Flows are executed on an underlying spark cluster for optimal scale-out performance for big data analytics, without you having to worry about any nitty-gritty details. We will look at the capabilities and use cases for Data Flows, where they best fit into your architecture, and how they compare to Power Queries (called Dataflows Gen2 in Microsoft Fabric). Then, we will work through a few different Data Flows demos to dig deeper into the various transformations available, as well as the expression language and how to use the visual expression builder. Finally, we will cover how to orchestrate and monitor our Data Flows, discuss lessons learned, and explain the pricing model.
- 8. Cathrine Wilhelmsen She / Her SolutionsArchitect Evidi hi@cathrinew.net cathrinew.net @cathrinew I love data and coding, as well as teaching and sharing knowledge Microsoft Data Platform MVP Organizing Fabric February Renovating a house
- 9.
- 10. #PASSDataSummit What is AzureData Factory? Standalone service for: • Data Integration • Workflow Orchestration • Scheduling
- 11. #PASSDataSummit What is AzureSynapse Analytics? Unified analytics platform: • Data Integration • Data Lake • Data Warehousing • Big Data Analytics • Time-Series Analytics • Data Science
- 12.
- 13.
- 14.
- 15.
- 16. #PASSDataSummit Copy Data Activity Thecore activity * Supports 100+ connectors Powerful built-in capabilities * Cathrine's opinion
- 17. #PASSDataSummit Copy Data Activity:Binary Files Source Sink
- 18. #PASSDataSummit Copy Data Activity:Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping
- 19. #PASSDataSummit Copy Data Activity:Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Convert file formats
- 20. #PASSDataSummit Copy Data Activity:Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Zip or unzip files
- 21. #PASSDataSummit Copy Data Activity:Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Map columns implicitly or explicitly
- 22.
- 23.
- 24.
- 25.
- 26. #PASSDataSummit What are DataFlows? • Data transformation at scale • Visual editor, low-code experience • Runs on serverless, managed Spark clusters
- 27. #PASSDataSummit Why use DataFlows? Transform big data without writing code Modify complex structures using the expression language instead of Python, Scala, etc.
- 28. #PASSDataSummit Why use DataFlows? Optimized for data warehousing scenarios Slowly changing dimensions, fact table loading, fuzzy lookups, data quality validation etc.
- 29. #PASSDataSummit Why use DataFlows? Can handle flexible schemas and schema drift Column pattern matching, rule-based mappings, byNames, byPosition, etc.
- 30.
- 31. #PASSDataSummit What are transformations? •One step in the data flow • Executed sequentially • Order generally doesn’t matter for performance
- 32. #PASSDataSummit Which transformations exist? •Inputs / outputs (Blue) • Multiple inputs / outputs (Purple) • Schema modifiers (Green) • Row modifiers (Orange) • Formatters (Teal) • Flowlets (Turquoise)
- 33.
- 34.
- 35. #PASSDataSummit What does orchestrationmean? Defining Workflows Which activities to run in which order? Configuring Alerts and Error Handling How to handle unexpected results and failures? Adding Triggers When to execute pipelines?
- 36.
- 37.
- 38.
- 39.
- 40. #PASSDataSummit Activity Dependencies: LogicalAND AND AND AND
- 41.
- 42. #PASSDataSummit Triggers Execute last publishedpipeline: On a set Schedule In a Tumbling Window When Event happens Now
- 43. #PASSDataSummit Triggers: Schedule Execute oneor more pipelines on a set schedule: • Every Wednesday at 06:00 • Last day of the month at 18:00 • Every Monday at 04:00 and Friday at 20:00
- 44. #PASSDataSummit Triggers: Tumbling Window Executea single pipeline for each time slice: • For every 15 minutes • For every 1 hour • For every 24 hours
- 45. #PASSDataSummit Triggers: Storage orCustom Events Execute one or more pipelines when: • Blob is Created • Blob is Deleted • Custom Event Happens
- 46. #PASSDataSummit Triggers: Now Execute asingle pipeline immediately
- 47.
- 48.
- 49. #PASSDataSummit Azure Data FactoryData Flows Basic: $0.274 per vCore-hour General Purpose Standard: $0.343 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 50. #PASSDataSummit Azure Synapse AnalyticsData Flows Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 51. #PASSDataSummit Data Flows: RoundingUp Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 52. #PASSDataSummit Data Flows: ClusterSize Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 53. #PASSDataSummit Data Flows: ClusterSize Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 54. #PASSDataSummit Data Flows: ClusterSize Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 55. #PASSDataSummit Data Flows: ClusterSize Basic: $0.257 per vCore-hour General Purpose: 4 (+4 Driver Cores) Standard: $0.325 per vCore-hour Memory Optimized: 4 (+4 Driver Cores) * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 56. #PASSDataSummit Data Flows: ClusterSize Basic: $0.257 per vCore-hour = $2.056 per hour General Purpose: 4 (+4 Driver Cores) Standard: $0.325 per vCore-hour = $2.6 per hour Memory Optimized: 4 (+4 Driver Cores) * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 57. #PASSDataSummit Data Flows: ClusterSize Basic: $0.257 per vCore-hour = $69.9 per hour General Purpose: 256 (+16 Driver Cores) Standard: $0.325 per vCore-hour = $88.4 per hour Memory Optimized: 256 (+16 Driver Cores) * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 58.
- 59. #PASSDataSummit Kamil Nowinski’s CheatSheet github.com/Azure-Player/CheatSheets
- 60. Keeping Up withData Flows Resources: aka.ms/dflinks Videos: aka.ms/dataflowvideos
- 61.
- 62. #PASSDataSummit «Overkill for smalldata» - Cathrine
- 63. #PASSDataSummit «Smaller datasets fortesting will save your butt» - Cathrine
- 64. #PASSDataSummit «Turn. It. Off.When. Finished.» - Cathrine
- 65.
- 66. Session evaluation Your feedbackis important to us! PASSDataCommunitySummit.com/evaluation Evaluate this session at:
- 67. #PASSDataSummit Coming up nextin our Learning Pathway: The Battle of the Data Transformation Tools Visually Transform Data in Azure Data Factory or Azure Synapse Analytics Cathrine Wilhelmsen Power up Your Transformations Game with Power Query in Power BI and Fabric Marthe Moengen Azure Databricks and Notebooks in Fabric – A Transformation Dream Come True? Emilie Rønning The Battle of the Data Transformation Tools Cathrine Wilhelmsen, Marthe Moengen, Emilie Rønning
- 68. Thank you! Special thanksto Mark Kromer Cathrine Wilhelmsen cathrine@fabricfebruary.com cathrinew.net @cathrinew