Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Creating Visual Transformations in Azure Data Factory@cathrinew
Azure Data Factory v2 came with many new capabilities and
...
Sponsors – Thank you!
Visual Transformations
in Azure Data Factory
Cathrine Wilhelmsen
dataMinds Connect · Oct 8, 2019
@cathrinew
cathrinew.net
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and ...
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and ...
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What?
When?
Why?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Collect
Store
Transform
Integrate
Prepare
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Data Factory?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data
Mapping
Data Flows
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Mapping Data Flows?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What do Mapping Data Flows look like?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use Mapping Data Flows?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
When to use Mapping Data Flows?
Transformations
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Data Flow Transformations?
18
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Source
1 or more sources per data flow
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Sink
1 or more sinks per data flow
Multiple
Inputs / Outputs
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Branch
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Join
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Conditional Split
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Exists
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Union
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup
Schema Modifier
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Derived Column
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Select
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Aggregate
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Surrogate Key
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Pivot
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Unpivot
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Window
Row Modifier
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Filter
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Sort
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Alter Row
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Kamil Nowinski's ADF Cheat Sheet
github.com/SQLPlayer/CheatSheets/blob/maste...
Expression
Language
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expression Language
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expression Language: Two Flavors
||
&&
==
>
>=
<
<=
!=
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Visual Expression Builder
Debug
and Execute
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Debug Mode?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you execute a Data Flow?
Schema Drift
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Schema Drift?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you handle Schema Drift?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Column Pattern Matching?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Column Pattern Matching work?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Column Pattern Matching work?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Column Pattern Matching work?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Rule-Based Mapping?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Rule-Based Mapping work?
DEMO
Let's transform
some data!
Pricing
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How much do Mapping Data Flows cost?
Compute Optimized: $0.075 per vCore-hou...
Use Cases
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Advanced Analytics on Big Data
https://azure.microsoft.com/en-us/solutions/a...
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Modern Data Warehouse
https://azure.microsoft.com/en-us/solutions/architectu...
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Ingest
Azure
Data Factory
Serve
Azure SQL Data
Warehouse
Visualize
Power BI
...
Continued
Learning
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Resources:
aka.ms/dflinks
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Videos:
aka.ms/adfvideos
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Mark Kromer on Twitter:
@KromerBigData @mssqldude
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Announcement Blog Post
https://azure.microsoft.com
/en-us/blog/azure-data-
f...
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Upcoming SlideShare
Loading in …5
×

of

Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 1 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 2 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 3 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 4 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 5 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 6 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 7 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 8 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 9 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 10 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 11 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 12 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 13 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 14 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 15 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 16 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 17 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 18 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 19 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 20 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 21 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 22 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 23 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 24 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 25 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 26 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 27 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 28 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 29 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 30 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 31 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 32 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 33 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 34 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 35 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 36 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 37 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 38 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 39 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 40 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 41 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 42 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 43 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 44 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 45 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 46 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 47 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 48 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 49 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 50 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 51 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 52 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 53 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 54 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 55 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 56 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 57 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 58 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 59 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 60 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 61 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 62 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 63 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 64 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 65 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 66 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 67 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 68 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 69 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 70 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 71 Creating Visual Transformations in Azure Data Factory (dataMinds Connect) Slide 72
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

3 Likes

Share

Download to read offline

Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

Download to read offline

Creating Visual Transformations in Azure Data Factory (Presented at dataMinds Connect on October 8th, 2019)

Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

  1. 1. Creating Visual Transformations in Azure Data Factory@cathrinew Azure Data Factory v2 came with many new capabilities and improvements. One of biggest game-changers is the Mapping Data Flows feature, allowing you to transform data at scale - without having to write a single line of code! In this session, we will first go through the capabilities and use cases for Mapping Data Flows. Then, we will explore the various transformations available, as well as the expression language and how to use the visual expression builder. Finally, we will look at how to debug, monitor, and optimize our data transformations. Cathrine Wilhelmsen
  2. 2. Sponsors – Thank you!
  3. 3. Visual Transformations in Azure Data Factory Cathrine Wilhelmsen dataMinds Connect · Oct 8, 2019
  4. 4. @cathrinew cathrinew.net
  5. 5. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  6. 6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  7. 7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What? When? Why?
  8. 8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Collect Store Transform Integrate Prepare
  9. 9. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  10. 10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Azure Data Factory?
  11. 11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  12. 12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  13. 13. Mapping Data Flows
  14. 14. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What are Mapping Data Flows?
  15. 15. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What do Mapping Data Flows look like?
  16. 16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Why use Mapping Data Flows?
  17. 17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) When to use Mapping Data Flows?
  18. 18. Transformations
  19. 19. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What are Data Flow Transformations? 18
  20. 20. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Source 1 or more sources per data flow
  21. 21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Sink 1 or more sinks per data flow
  22. 22. Multiple Inputs / Outputs
  23. 23. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Branch
  24. 24. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Join
  25. 25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Conditional Split
  26. 26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Exists
  27. 27. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Union
  28. 28. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup
  29. 29. Schema Modifier
  30. 30. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Derived Column
  31. 31. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Select
  32. 32. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Aggregate
  33. 33. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Surrogate Key
  34. 34. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Pivot
  35. 35. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Unpivot
  36. 36. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Window
  37. 37. Row Modifier
  38. 38. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Filter
  39. 39. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Sort
  40. 40. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Alter Row
  41. 41. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Kamil Nowinski's ADF Cheat Sheet github.com/SQLPlayer/CheatSheets/blob/master/ADFDF-Cheat-Sheet-sqlplayer.pdf
  42. 42. Expression Language
  43. 43. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Expression Language
  44. 44. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Expression Language: Two Flavors || && == > >= < <= !=
  45. 45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Visual Expression Builder
  46. 46. Debug and Execute
  47. 47. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Debug Mode?
  48. 48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How do you execute a Data Flow?
  49. 49. Schema Drift
  50. 50. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Schema Drift?
  51. 51. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How do you handle Schema Drift?
  52. 52. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Column Pattern Matching?
  53. 53. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How does Column Pattern Matching work?
  54. 54. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How does Column Pattern Matching work?
  55. 55. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How does Column Pattern Matching work?
  56. 56. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Rule-Based Mapping?
  57. 57. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How does Rule-Based Mapping work?
  58. 58. DEMO Let's transform some data!
  59. 59. Pricing
  60. 60. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How much do Mapping Data Flows cost? Compute Optimized: $0.075 per vCore-hour General Purpose: $0.109 per vCore-hour Memory Optimized: $0.148 per vCore-hour
  61. 61. Use Cases
  62. 62. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Advanced Analytics on Big Data https://azure.microsoft.com/en-us/solutions/architecture/advanced-analytics-on-big-data/
  63. 63. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Modern Data Warehouse https://azure.microsoft.com/en-us/solutions/architecture/modern-data-warehouse/
  64. 64. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Ingest Azure Data Factory Serve Azure SQL Data Warehouse Visualize Power BI Store Azure Data Lake Storage Gen2 Data Pipeline Orchestration and Monitoring Azure Data Factory Sources Cloud SaaS Prepare Wrangling Data Flows Transform Mapping Data Flows On-Premises
  65. 65. Continued Learning
  66. 66. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Resources: aka.ms/dflinks
  67. 67. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Videos: aka.ms/adfvideos
  68. 68. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Mark Kromer on Twitter: @KromerBigData @mssqldude
  69. 69. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  70. 70. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Announcement Blog Post https://azure.microsoft.com /en-us/blog/azure-data- factory-mapping-data- flows-are-now-generally- available/
  71. 71. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!
  • AnitaAllen12

    Nov. 25, 2021
  • AlpaBuddhabhatti

    Feb. 11, 2021
  • AyshaHamisi

    Jul. 9, 2020

Creating Visual Transformations in Azure Data Factory (Presented at dataMinds Connect on October 8th, 2019)

Views

Total views

1,013

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

52

Shares

0

Comments

0

Likes

3

×