Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

184 views

Published on

Creating Visual Transformations in Azure Data Factory (Presented at dataMinds Connect on October 8th, 2019)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

  1. 1. Creating Visual Transformations in Azure Data Factory@cathrinew Azure Data Factory v2 came with many new capabilities and improvements. One of biggest game-changers is the Mapping Data Flows feature, allowing you to transform data at scale - without having to write a single line of code! In this session, we will first go through the capabilities and use cases for Mapping Data Flows. Then, we will explore the various transformations available, as well as the expression language and how to use the visual expression builder. Finally, we will look at how to debug, monitor, and optimize our data transformations. Cathrine Wilhelmsen
  2. 2. Sponsors – Thank you!
  3. 3. Visual Transformations in Azure Data Factory Cathrine Wilhelmsen dataMinds Connect · Oct 8, 2019
  4. 4. @cathrinew cathrinew.net
  5. 5. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  6. 6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  7. 7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What? When? Why?
  8. 8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Collect Store Transform Integrate Prepare
  9. 9. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  10. 10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Azure Data Factory?
  11. 11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  12. 12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  13. 13. Mapping Data Flows
  14. 14. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What are Mapping Data Flows?
  15. 15. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What do Mapping Data Flows look like?
  16. 16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Why use Mapping Data Flows?
  17. 17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) When to use Mapping Data Flows?
  18. 18. Transformations
  19. 19. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What are Data Flow Transformations? 18
  20. 20. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Source 1 or more sources per data flow
  21. 21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Sink 1 or more sinks per data flow
  22. 22. Multiple Inputs / Outputs
  23. 23. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Branch
  24. 24. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Join
  25. 25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Conditional Split
  26. 26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Exists
  27. 27. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Union
  28. 28. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup
  29. 29. Schema Modifier
  30. 30. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Derived Column
  31. 31. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Select
  32. 32. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Aggregate
  33. 33. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Surrogate Key
  34. 34. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Pivot
  35. 35. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Unpivot
  36. 36. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Window
  37. 37. Row Modifier
  38. 38. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Filter
  39. 39. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Sort
  40. 40. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Alter Row
  41. 41. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Kamil Nowinski's ADF Cheat Sheet github.com/SQLPlayer/CheatSheets/blob/master/ADFDF-Cheat-Sheet-sqlplayer.pdf
  42. 42. Expression Language
  43. 43. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Expression Language
  44. 44. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Expression Language: Two Flavors || && == > >= < <= !=
  45. 45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Visual Expression Builder
  46. 46. Debug and Execute
  47. 47. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Debug Mode?
  48. 48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How do you execute a Data Flow?
  49. 49. Schema Drift
  50. 50. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Schema Drift?
  51. 51. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How do you handle Schema Drift?
  52. 52. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Column Pattern Matching?
  53. 53. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How does Column Pattern Matching work?
  54. 54. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How does Column Pattern Matching work?
  55. 55. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How does Column Pattern Matching work?
  56. 56. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Rule-Based Mapping?
  57. 57. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How does Rule-Based Mapping work?
  58. 58. DEMO Let's transform some data!
  59. 59. Pricing
  60. 60. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How much do Mapping Data Flows cost? Compute Optimized: $0.075 per vCore-hour General Purpose: $0.109 per vCore-hour Memory Optimized: $0.148 per vCore-hour
  61. 61. Use Cases
  62. 62. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Advanced Analytics on Big Data https://azure.microsoft.com/en-us/solutions/architecture/advanced-analytics-on-big-data/
  63. 63. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Modern Data Warehouse https://azure.microsoft.com/en-us/solutions/architecture/modern-data-warehouse/
  64. 64. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Ingest Azure Data Factory Serve Azure SQL Data Warehouse Visualize Power BI Store Azure Data Lake Storage Gen2 Data Pipeline Orchestration and Monitoring Azure Data Factory Sources Cloud SaaS Prepare Wrangling Data Flows Transform Mapping Data Flows On-Premises
  65. 65. Continued Learning
  66. 66. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Resources: aka.ms/dflinks
  67. 67. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Videos: aka.ms/adfvideos
  68. 68. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Mark Kromer on Twitter: @KromerBigData @mssqldude
  69. 69. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  70. 70. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Announcement Blog Post https://azure.microsoft.com /en-us/blog/azure-data- factory-mapping-data- flows-are-now-generally- available/
  71. 71. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!

×