Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Unleash the Power of Azure Data Factory - SQL User Group

109 views

Published on

Big data requires service that can orchestrate and operationalize processes to refine the enormous stores of raw data into actionable business insights. Azure Data Factory is a managed cloud service that's built for these complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Unleash the Power of Azure Data Factory - SQL User Group

  1. 1. Unleash the power of Azure Data Factory Sergio Zenatti Filho, Associate Director Data & Analytics, Satalyst
  2. 2. Sergio Zenatti Filho Associate Director Data &Analytics at Satalyst I am Data and Analytics Director with over 16 years experience in the delivery of Business Intelligence and Analytics Solutions. I worked internationally around Australia, New Zealand and Brazil, in sectors that include Mining, Oil & Gas, Government, Healthcare, Financial Services, Telecom, Automotive and dairy. I enjoy learning new technology and help people to learn. Place your photo here /sergiozenatti @SergioZenatti https://zenatti.net/
  3. 3. Thanks to all sponsors
  4. 4. SQL Saturday Perth - 2018 http://www.sqlsaturday.com/761
  5. 5. WE ARE MOVING !!!!!
  6. 6. Explore Everything PASS Has to Offer FREE ONLINE WEBINAR EVENTS FREE 1-DAY LOCAL TRAINING EVENTS LOCAL USER GROUPS AROUND THE WORLD ONLINE SPECIAL INTEREST USER GROUPS DATA PROFESSIONAL CONFERENCE VOLUNTEERING OPPORTUNITIES PASS COMMUNITY NEWSLETTER BA INSIGHTS NEWSLETTERFREE ONLINE RESOURCES November 7-9 in Seattle, WA
  7. 7. PASS Summit is the largest conference for technical professionals who leverage the Microsoft Data Platform. Get the skills and community connections you need to advance your company and career. Continue the learning. Save $150 USD Register for PASS Summit and as a participant in [enter group here], you can save $150 USD off your conference registration price. To access your exclusive discount, use code* LSDISQ4O5 when you register and your savings will calculate at time of registration. *Unique discount code cannot be applied retroactively and cannot be combined with any other offers G R O U P R E F E R R A L D I S C O U N T C O D E
  8. 8. Session objectives
  9. 9. Azure Data Factory • Cloud-based data integration service. Currently ADF v1 (GA) and ADF v2 (Preview). • Used to automate the movement and transformation of data. • Connect to on-premises and cloud data sources. • Integration with HDInsight, Azure Batch, Azure Functions, Logic Apps and more. • Globally deployed data movement as a service (Integration Runtime).
  10. 10. ADF v1, v2 and new Features • ADF v1 is designed for the batch data processing of time series data. • ADF v2 is general-purpose hybrid data integration service with very flexible execution patterns. ADF v2 – New Features
  11. 11. ADF Components Activities Type: • Data Movement • Data Transformation • Control
  12. 12. ADF Activities Options • Data Movement • Data Transformation • Control
  13. 13. Portal, PowerShell, .NET, Python, REST and ARM
  14. 14. ADF – Real Life Scenario - Components • Azure SQL Data Warehouse – Expensive to run if it is not using all the time POST https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group- name}/providers/Microsoft.Sql/servers/{server-name}/databases/{database-name}/pause?api-version=2014-04-01-preview HTTP/1.1 GET https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group- name}/providers/Microsoft.Sql/servers/{server-name}/databases/{database-name}?api-version=2014-04-01 HTTP/1.1 POST https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group- name}/providers/Microsoft.Sql/servers/{server-name}/databases/{database-name}/resume?api-version=2014-04-01-preview HTTP/1.1
  15. 15. ADF – Real Life Scenario - Components • Azure Analysis Services – Process data model after data load POST https://<region>.asazure.windows.net/servers/<my server>/models/<my db>/refreshes GET https://<region>.asazure.windows.net/servers/<my server>/models/<my db>/refreshes/<run id>
  16. 16. ADF – Data Integration Scenario • Process around 15k XML files a day with total size around 6-7 GB • Infrastructure cost: max $5k a month • Process files once every hour • ADF, Logic Apps, Azure Data Lake, Azure SQL DW and Azure Analysis Services
  17. 17. ADF – Learning New Azure Data Factory self-paced hands-on lab for UI Orchestrating Big Data with Azure Data Factory V1 https://www.edx.org/course/orchestrating-big-data-azure-data-microsoft-dat223-3x-3 https://azure.microsoft.com/en-us/blog/new-azure-data-factory-self-paced-hands-on-lab-for-ui/ https://azure.microsoft.com/en-us/documentation/learning-paths/data-factory/ Learning path for Azure Data Factory V1 and V2
  18. 18. ADF - Overview • Azure Data Factory (ADF) Overview • Understand how ADF Works • ADF v1, v2 and new features • Understand ADF components • Be able to deploy, build and manage a simple ADF implementation • Next step Start Using it J
  19. 19. Thank you for your time! Sergio Zenatti Filho Data & Analytics Architect sergiozenatti @SergioZenatti zenatti.net

×