Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Auckland SQL Saturday - Azure Data Lake

65 views

Published on

Azure Data Lake Store and Analytics

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Auckland SQL Saturday - Azure Data Lake

  1. 1. Harness the power of big data using Azure Data Lake Sergio Zenatti Filho
  2. 2. Thanks to our Gold Sponsors: SQL Sat Auckland Sergio Zenatti Filho Associate Director Data & Analytics - Satalyst I am Data and Analytics Director with over 16 years experience in the delivery of Business Intelligence and Analytics Solutions. I worked internationally around Australia, New Zealand and Brazil, in sectors that include Mining, Oil & Gas, Government, Healthcare, Financial Services, Telecom, Automotive and dairy. I enjoy learning new technology and help people to learn. Place your photo here /sergiozenatti @SergioZenatti zenatti.net
  3. 3. Thanks to our Gold Sponsors: SQL Sat Auckland Thanks to all sponsors
  4. 4. Thanks to our Gold Sponsors: SQL Sat Auckland Session Objectives • What is Data Lake • Azure Data Lake Store • Azure Data Lake Analytics • U-SQL • Demo
  5. 5. Thanks to our Gold Sponsors: SQL Sat Auckland What is Data Lake Ingest all data regardless of requirements Store all data in native format without schema definition Do analysis Hadoop, Spark, R, Azure Data Lake Analytics (ADLA) Interactive queries Batch queries Machine Learning Data warehouse Devices
  6. 6. Thanks to our Gold Sponsors: SQL Sat Auckland The 3 Azure Data Lake Services
  7. 7. Thanks to our Gold Sponsors: SQL Sat Auckland Azure Data Lake Store • A hyper-scale repository for Big Data analytics workloads; • Hadoop File System (HDFS) for the cloud; • Unlimited storage and can host petabyte files; • Store any data in its native format; • Enterprise-grade access control and encryption;
  8. 8. Thanks to our Gold Sponsors: SQL Sat Auckland Azure Data Lake Store
  9. 9. DEMO Data Lake Store
  10. 10. Thanks to our Gold Sponsors: SQL Sat Auckland Azure Data Lake Analytics • An on-demand analytics job service in the cloud; • Run massively parallel data transformation and processing programs in U-SQL, R, Python, and .NET; • No infrastructure to manage, you can process data on demand, scale instantly, and only pay per job; • Integrates with Visual Studio to develop, debug and tune code faster; Azure Data Lake Analytics Unit (AU): is a unit of computation made available to your U-SQL job. Each AU gives your job access to a set of underlying resources such as CPU and memory.
  11. 11. Thanks to our Gold Sponsors: SQL Sat Auckland Azure Data Lake Analytics - Query U-SQL Query Query Query Query W rite Azure Storage Blobs SQL in VMs Azure SQL DB Azure Data Lake Analytics Query Azure SQL Data Warehouse Query Write Azure Data Lake Storage
  12. 12. Thanks to our Gold Sponsors: SQL Sat Auckland U-SQL • It’s a framework for Big Data; • Familiar syntax to millions of SQL & .NET developers; • Built on the same distributed runtime that powers the big data systems inside Microsoft; • Querying multiple Azure Data Sources (Federated Query);
  13. 13. Thanks to our Gold Sponsors: SQL Sat Auckland Cognitive Capabilities in U-SQL • Image Tagging • Emotion Extraction • Face Detection • Optical Character Recognition • Key Phrases Extraction • Sentiment Analysis
  14. 14. DEMO Data Lake Analytics
  15. 15. Thanks to our Gold Sponsors: SQL Sat Auckland ADLA – Real Scenario • Process around 15k XML files a day with total size around 6-7 GB • Infrastructure cost: max $5k a month • Process files once every hour • ADF, Logic Apps, Azure Data Lake, Azure SQL DW and Azure Analysis Services
  16. 16. Thanks to our Gold Sponsors: SQL Sat Auckland What Next? • https://mva.microsoft.com/en-us/training-courses/data-series-analytics- big-data-azure-data-lake-17759 • https://www.edx.org/course/processing-big-data-with-azure-data-lake- analytics • https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake- analytics-data-lake-tools-get-started
  17. 17. Thanks to our Gold Sponsors: SQL Sat Auckland Q&A
  18. 18. Thank you for your time! Sergio Zenatti Filho Associate Director Data & Analytics, Satalyst sergiozenatti @SergioZenatti zenatti.net

×