Successfully reported this slideshow.
Your SlideShare is downloading. ×

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azure Synapse Analytics (DataMinutes #2)

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 53 Ad

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azure Synapse Analytics (DataMinutes #2)

Download to read offline

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azure Synapse Analytics (Presented at DataMinutes #2 on January 21st, 2022)

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azure Synapse Analytics (Presented at DataMinutes #2 on January 21st, 2022)

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azure Synapse Analytics (DataMinutes #2) (20)

Advertisement

More from Cathrine Wilhelmsen (20)

Recently uploaded (20)

Advertisement

Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azure Synapse Analytics (DataMinutes #2)

  1. 1. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azure Synapse Analytics Cathrine Wilhelmsen DataMinutes #2 • January 21st, 2022
  2. 2. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Session Abstract Azure Data Factory and Synapse Pipelines pricing is easy, right? No upfront costs. Pay only for what you use. It's a wonderful world for data integration developers. Just a few clicks and your solution is ready! But what do you present to management when they ask for cost estimates? "I guess we just have to wait for the next invoice" is rarely an acceptable answer. It is definitely not an acceptable answer if that invoice turns out to be unexpectedly high. In this session, we will look at the bits and pieces that make up the Azure Data Factory and Synapse Pipelines pricing models, how to monitor consumption and manage costs, and cover a few important gotchas to be aware of…
  3. 3. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net
  4. 4. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipelines are awesome, right?
  5. 5. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) No upfront costs! Just start developing! Pay only for usage!
  6. 6. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) “But how much will it cost per month?”
  7. 7. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) “I guess we just have to wait for the invoice!"
  8. 8. Pipeline Pricing
  9. 9. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Operations Orchestration and Execution Data Flows
  10. 10. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Operations (Pretty cheap, but everything has a cost) Orchestration and Execution Data Flows
  11. 11. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Operations: Azure Data Factory Read / Write: $0.50 per 50 000 entities modified Operations: Create, Read, Update, Delete Monitoring: $0.25 per 50 000 run records retrieved Operations: Get, List * Prices in USD from January 2022
  12. 12. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Operations: Azure Synapse First 1 Million Operations: Free! Operations: Create, Read, Update, Delete, Get, List Additional Operations: $0.25 per 50 000 operations Operations: Create, Read, Update, Delete, Get, List * Prices in USD from January 2022
  13. 13. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Operations Orchestration and Execution (Be aware of the biggest gotchas) Data Flows
  14. 14. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Integration Runtimes Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  15. 15. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Orchestration Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  16. 16. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Orchestration Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per run: Activity runs, debug runs, trigger executions $0.001 $0.001 $0.0015 Data Movement per DIU-hour: Copy Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  17. 17. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Data Movement Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  18. 18. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Data Movement Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  19. 19. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Data Movement Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  20. 20. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Data Movement Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. Auto = 4+
  21. 21. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Data Movement Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 Minimum = 2 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  22. 22. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Pipeline Activity Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  23. 23. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Pipeline Activity Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebook, Databricks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. 200x 500x
  24. 24. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: External Activity Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebooks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  25. 25. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: External Activity Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebooks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. 4000x 10000x
  26. 26. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebooks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  27. 27. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Rounding Up Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebooks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  28. 28. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Rounding Up Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebooks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  29. 29. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Rounding Up Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebooks etc. $0.00025 $1 $0.0001 * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  30. 30. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Rounding Up Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebooks etc. $0.00025 $1 $0.0001 0.0167 hours = 1 minute * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  31. 31. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Pricing: Rounding Up Azure IR Azure Managed VNet IR Self-Hosted IR Orchestration per 1000 runs: Activity runs, debug runs, trigger executions $1 $1 $1.50 Data Movement per DIU-hour: Copy Data Activity $0.25 $0.25 $0.10 Pipeline Activity per hour: Lookup, Get Metadata, Delete etc. $0.005 $1 $0.002 External Activity per hour: Stored Procedure, Notebooks etc. $0.00025 $1 $0.0001 0.0333 hours = 2 minutes * Prices in USD from January 2022. All activities are prorated by the minute and rounded up.
  32. 32. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Operations Orchestration and Execution Data Flows (Can be expensive, but worth it for the right use cases)
  33. 33. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Data Flows Basic: $0.286 per vCore-hour General Purpose Standard: $0.346 per vCore-hour Memory Optimized * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
  34. 34. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Data Flows: Rounding Up Basic: $0.286 per vCore-hour General Purpose Standard: $0.346 per vCore-hour Memory Optimized * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
  35. 35. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Data Flows: Cluster Size Basic: $0.286 per vCore-hour General Purpose Standard: $0.346 per vCore-hour Memory Optimized * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
  36. 36. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Data Flows: Cluster Size Basic: $0.286 per vCore-hour General Purpose Standard: $0.346 per vCore-hour Memory Optimized * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
  37. 37. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Data Flows: Cluster Size Basic: $0.286 per vCore-hour General Purpose Standard: $0.346 per vCore-hour Memory Optimized * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
  38. 38. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Data Flows: Cluster Size Basic: $0.286 per vCore-hour = $2.288 per hour General Purpose: 4 (+4 Driver Cores) Standard: $0.346 per vCore-hour = $2.786 per hour Memory Optimized: 4 (+4 Driver Cores) * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
  39. 39. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Data Flows: Cluster Size Basic: $0.286 per vCore-hour = $77.792 per hour General Purpose: 256 (+16 Driver Cores) Standard: $0.346 per vCore-hour = $94.112 per hour Memory Optimized: 256 (+16 Driver Cores) * Prices in USD from January 2022. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
  40. 40. Lessons Learned
  41. 41. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Just because you can… doesn’t mean you should…
  42. 42. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Example
  43. 43. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Example = $0.0000835 = $0.0167 = $0.000008325
  44. 44. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Example $0.053
  45. 45. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Example $0.053 $1.59
  46. 46. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Example $0.053 $159
  47. 47. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Example $0.053 $3816
  48. 48. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Example $0.053 $45792
  49. 49. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Again Just because you can… doesn’t mean you should…
  50. 50. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Everything has a price We need to understand the pricing model to create solutions optimized for performance and cost
  51. 51. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) “I guess we just have to wait for the invoice!"
  52. 52. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Here’s our estimate! azure.microsoft.com/pricing/details/data-factory azure.microsoft.com/pricing/details/synapse-analytics
  53. 53. © 2022 Cathrine Wilhelmsen (hi@cathrinew.net) Thank you! hi@cathrinew.net @cathrinew cathrinew.net

×