This document discusses visual data transformations in Azure Data Factory. It introduces Mapping Data Flows, which allow transforming data at scale visually without code. It explores the capabilities of Mapping Data Flows, the various transformations available, using expressions, debugging transformations, and handling schema drift. The document includes a demonstration of transforming sample data using Mapping Data Flows.

1. Creating Visual Transformations in Azure Data Factory@cathrinew
Azure Data Factory v2 came with many new capabilities and
improvements. One of biggest game-changers is the Mapping
Data Flows feature, allowing you to transform data at scale -
without having to write a single line of code!
In this session, we will first go through the capabilities and
use cases for Mapping Data Flows. Then, we will explore the
various transformations available, as well as the expression
language and how to use the visual expression builder.
Finally, we will look at how to debug, monitor, and optimize
our data transformations.
Cathrine Wilhelmsen

3. Sponsors – Thank you!

4. Visual Transformations
in Azure Data Factory
Cathrine Wilhelmsen
dataMinds Connect · Oct 8, 2019

5. @cathrinew
cathrinew.net

6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What?
When?
Why?

9. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Collect
Store
Transform
Integrate
Prepare

10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Data Factory?

12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data

13. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data

14. Mapping
Data Flows

15. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Mapping Data Flows?

16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What do Mapping Data Flows look like?

17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use Mapping Data Flows?

18. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
When to use Mapping Data Flows?

19. Transformations

20. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Data Flow Transformations?
18

21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Source
1 or more sources per data flow

22. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Sink
1 or more sinks per data flow

23. Multiple
Inputs / Outputs

24. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Branch

25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Join

26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Conditional Split

27. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Exists

28. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Union

29. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup

30. Schema Modifier

31. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Derived Column

32. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Select

33. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Aggregate

34. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Surrogate Key

35. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Pivot

36. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Unpivot

37. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Window

38. Row Modifier

39. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Filter

40. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Sort

41. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Alter Row

42. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Kamil Nowinski's ADF Cheat Sheet
github.com/SQLPlayer/CheatSheets/blob/master/ADFDF-Cheat-Sheet-sqlplayer.pdf

43. Expression
Language

44. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expression Language

45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expression Language: Two Flavors
||
&&
==
>
>=
<
<=
!=

46. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Visual Expression Builder

47. Debug
and Execute

48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Debug Mode?

49. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you execute a Data Flow?

50. Schema Drift

51. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Schema Drift?

52. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you handle Schema Drift?

53. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Column Pattern Matching?

54. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Column Pattern Matching work?

55. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Column Pattern Matching work?

56. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Column Pattern Matching work?

57. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Rule-Based Mapping?

58. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Rule-Based Mapping work?

59. DEMO
Let's transform
some data!

60. Pricing

61. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How much do Mapping Data Flows cost?
Compute Optimized: $0.075 per vCore-hour
General Purpose: $0.109 per vCore-hour
Memory Optimized: $0.148 per vCore-hour

62. Use Cases

63. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Advanced Analytics on Big Data
https://azure.microsoft.com/en-us/solutions/architecture/advanced-analytics-on-big-data/

64. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Modern Data Warehouse
https://azure.microsoft.com/en-us/solutions/architecture/modern-data-warehouse/

65. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Ingest
Azure
Data Factory
Serve
Azure SQL Data
Warehouse
Visualize
Power BI
Store
Azure Data Lake Storage Gen2
Data Pipeline Orchestration and Monitoring
Azure Data Factory
Sources
Cloud
SaaS
Prepare
Wrangling
Data Flows
Transform
Mapping
Data Flows
On-Premises

66. Continued
Learning

67. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Resources:
aka.ms/dflinks

68. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Videos:
aka.ms/adfvideos

69. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Mark Kromer on Twitter:
@KromerBigData @mssqldude

70. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

71. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Announcement Blog Post
https://azure.microsoft.com
/en-us/blog/azure-data-
factory-mapping-data-
flows-are-now-generally-
available/

72. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!