Intro to Azure Data Platform

2. Karen has 20+ years of data and information architecture experience on large, multi-project programs. She is a frequent speaker on data modeling, data-driven methodologies and cloud data architectures She wants you to love your data. Karen Lopez, Data Evangelist InfoAdvisors (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

4. This session Very top-level information – mostly about branding and components Terminology Architectures May have time at the end for some quick demos Resources to learn and use Azure Data Platform services (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

5. Caveats Cloud technology names change—all the time. Cloud technology features change—all the time. Cloud technology feature names change—all the time. Screenshots are out of date the second they are taken. This is what makes it so much fun, by the way. (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

6. On-premises versus cloud technologies Computing Environment Licensing Model Maintainability Scalability Availability

9. Azure Blob Storage Access Tiers Hot Frequently accessed Higher cost Cool Infrequently accessed Lesser cost 30 days Archive Rarely accessed Greater latency Least storage cost 180 days (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

10. Replication with Azure Storage (Blob and Data Lake Gen2) Locally Redundant Storage (LRS) Data is replicated in the same storage scale unit Least expensive storage 99.999999999% durability Zone Redundant Storage (ZRS) Data is replicated within same region, but across 3 physically separate clusters Survives a single scale unit failure 99.9999999999% durability Geo-redundant storage (GRS) Data is replicated across regions Survives a regional failure 99.99999999999999% durability Read-access geo- redundant storage (RA-GRS) Data is replicated across regions All regions are readable Most expensive storage (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

12. Azure Data Lake Storage Gen2 (ADLS Gen2) Based on Azure Blob Storage Optimized for big data and analytics workloads Hadoop Distributed File System compatible Supports a hierarchical namespace (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

14. Why Use Azure Databricks? OPTIMIZED FOR PROCESSING DATA FULLY MANAGED BY MICROSOFT ENTERPRISE SECURITY VIA AAD ENTERPRISE SLAS INTEGRATED WITH AZURE DATA STORES AND POWER BI (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

15. Azure Databricks Platform Components Spark SQL and DataFrames SQL-familiar language for working with structured data DataFrames are collections of named columns like a table Streaming Real-time data processing and analytics MLlib Machine learning library GraphX Graphs and Graph processing Spark Core API for R, SQL, Scala, Java and Python (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

16. Azure Databricks Workspace Organizes objects into folders Web-based tool to work with commands, text, visualizations Dashboard for visualizations Library of code/runtime packages Experiment for Machine Learning runs (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

21. Cosmos DB APIs Core: SQL Graph: Gremlin Key Value: Table Document: Mongo DB Column Family: Cassandra https://docs.microsoft.com/azure/cosmos-db/introduction (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

24. What is a Request Unit (RU)? Microsoft-defined unit of measure Measures resource usage for reads and writes Varies based on configuration Resources can be provisioned based on RUs https://docs.microsoft.com/en-us/azure/cosmos-db/scaling-throughput (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

25. Global Distribution with Cosmos DB High availability within a region automatically Globally distributed data survives regional outages <10ms reads and writes possible 99.999% availability with multiple regions Automatic disaster recovery (within and across regions) (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

26. Partitioning in Cosmos DB Requires partition keys Logical partition Distributes data across replica sets Performance Impact Cost Impact https://docs.microsoft.com/azure/cosmos-db/partitioning-overview (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

27. Cosmos DB consistency levels Strong Bounded staleness Session Consistent prefix Eventual

29. What is Azure SQL Database (SQL DB)? Fully managed relational database Platform-as-a-Service (PaaS) High feature parity with SQL Server database engine Cloud first development Microsoft handles all upgrades and patching 99.5 % availability Scalable (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

30. Azure SQL DB Deployment Options Managed Instance • Dedicated compute & storage • Nearly 100% on-prem compatibility • Private IPs Single database • Set database-specific resource allocations • Best for new cloud developments • Public IPs Elastic Pools • Shared resources across one or many databases • Best for varied workloads across databases • Public IPs (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

31. Azure SQL DB DTUs vs. Cores Database Throughput Units (DTUs) • Metric for provisioning CPU, memory, reads, writes • Service tiers • Single database vCores • Choice of specific compute vCores • Service tiers • Single or pooled databases (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

32. What is an Elastic Pool? A configuration for Azure SQL Database Support for scalability and costs for highly unpredictable workloads Shared server and resources (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

38. Data Distribution Algorithms Hash distributed Used for large fact tables (>2 GB) Tables that have frequent inserts, updates and deletions Improves query performance Round-Robin distributed Used to optimize data loads Random distribution Default for SQL DW Can slow performance Ideal for data that isn’t JOINed Staging tables Replicated Tables Used for small tables (<2 GB) Used for small reference data Increase query performance Increases storage costs (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

43. What Are Azure Streaming Solution Components? Azure Stream Analytics ◦ Inputs ◦ Queries ◦ Outputs Azure Event Hubs Azure IoT Hubs Event Hub IoT Hub Stream Analytics (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

44. Streaming Solutions Overview Event Hub IoT Hub Stream Analytics Machine Learning Data Lake Cosmos DB SQL DB SQL DW Azure Functions Event Hubs Power BI …And more (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

46. What are IoT Hubs? Azure managed message hub Bi-directional communication Scalable to millions of devices Tiered performance and pricing 99.9% SLA for availability (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

47. IoT Hubs versus Event Hubs Device-to-cloud messaging HTTPS, AMQP, AMQP over webSockets MQTT, MQTT over webSockets File upload from devices Device provisioning service Cloud-to-device messaging IoT Edge Device Management…and more Device-to-cloud messaging HTTPS, AMQP, AMQP over webSockets IoT Hub Event Hub https://docs.microsoft.com/azure/iot-hub/iot-hub-compare-event-hubs (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

48. ASA Outputs Relational data stores Non-relational data stores Power BI Service Bus Other event hubs Azure Functions ..and more https://docs.microsoft.com/azure/stream-analytics/stream-analytics-define-outputs (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

51. Configuring Linked Services Data Sources + Data Services • Datasets • Credentials • Tested connections Reusable across pipelines • Define once • Use everywhere JSON • Automation • ARM Templates (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

52. Defining Datasets Definition of the format of data • Schema of a database table • Structure of files and blobs JSON-based • Properties of dataset structure • File names and paths (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

53. Azure Data Factory Sink Target data store End of ADF pipeline More limited than input datasets Depends on Integration Runtime https://docs.microsoft.com/azure/data-factory/concepts-pipelines-activities (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

57. Azure Data Factory Triggers Schedule • Clock-based • Recurrence • End time Tumbling Window • Interval-based Event • Response to event Manual • User/Code- based (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

61. Microsoft Purview – Azure service Supports both on- prem and multi- cloud resources Automated data and file discovery Understand and document data assets unified view of an organization's data, regardless of where it's stored enables users to govern their data using automated processes, policies, and rules. https://learn.microsoft.com/en-us/purview/ (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

62. Data Map Automated data discovery compute and repository Automatic and manual scans Data Classification (200+ built- in classifications, plus custom ones) Data profiling, not just names Manual classifications https://learn.microsoft.com/en-us/azure/purview/apply- classifications (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

63. Sources/Connectors Azure data services (All of them?) Databases ◦ RDS ◦ Db2 ◦ Big Query ◦ Hive ◦ MongoDb ◦ MySQL ◦ Oracle ◦ PostgreSQL ◦ SAP (DW and HANA) ◦ Snowflake ◦ Teradata Files ◦ S3 ◦ HDFS Apps ◦ Erwin ◦ Looker ◦ PowerBI ◦ Salesforce ◦ SAP ECC ◦ SAP S/4HANA https://learn.microsoft.com/en-us/azure/purview/microsoft-purview-connector-overview (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

69. Resources https://azure.microsoft.com/en-us/updates/ Roadmap for Azure services ◦ https://azure.microsoft.com/en- us/updates/?status=inpreview Preview services are often free or very low cost. https://azure.microsoft.com/en-us/solutions/space/ Newish focus on Space Exploration and Orbital Science https://docs.microsoft.com/en- us/azure/architecture/browse/ Reference Architectures Tutorials, Quickstarts, How-Tos, Samples (C) INFOADVISORS 2023 ALL RIGHTS RESERVED

Intro to Azure Data Platform

Intro to Azure Data Platform

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Intro to Azure Data Platform

Similar to Intro to Azure Data Platform(20)

More from Karen Lopez

More from Karen Lopez(16)

Recently uploaded

Recently uploaded(20)

Intro to Azure Data Platform