Deploy Microsoft Azure Data Solutions
•Download as PPTX, PDF•
0 likes•160 views
Quali vantaggi ci da Azure? Dal punto di vista dello sviluppo software, uno di questi è certamente la varietà dei servizi di gestione dei dati. Questo ci permette di cominciare a non essere SQL centrici ma utilizzare il servizio giusto per il problema giusto fino ad applicare una strategia di Polyglot Persistence (e vedremo cosa significa) nel rispetto di una corretta gestione delle risorse IT e delle pratiche di DevOps.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deploy Microsoft Azure Data Solutions
Similar to Deploy Microsoft Azure Data Solutions (20)
More from Marco Parenzan
More from Marco Parenzan (20)
Recently uploaded
Recently uploaded (20)
Deploy Microsoft Azure Data Solutions
- 2. Senior Solution Architect @ beanTech Microsoft Azure MVP Community Lead 1nn0va // Pordenone 1nn0va After Hour https://bit.ly/1nn0va-video Linkedin: https://www.linkedin.com/in/marcoparenzan/ Marco Parenzan
- 4. Da SQL Server a Azure SQL e oltre: le opportunità per i dati in Azure Dati relazionali in Azure Dati non relazionali Dati blob in Azure Architetture a Messaggi, Comandi ed Eventi in Azure Big Data in Azure Gestire i dati dell'IoT in Azure Servizi Cognitivi per i dati Sicurezza Agenda
- 6. • Polyglot Persistence • …non «spendere per spendere»… • No more SQL...only! • Microservizi • Ogni servizio porta con sé uno storage • «Next,Next,Next,Next,...» • Operations & DevOps • Quanto è «dato» e quanto è «servizio»? • Un confine sempre più labile • Questo è tutto ciò che è il cloud Alcuni temi
- 11. Pricing Azure Storage Blobs Pricing | Microsoft Azure Files and API Events External Tables Azure Blobs
- 12. Storage Account IoT Hub Devices Events Ingest The batch point of view...
- 13. With no any specific request... what is IoT all about? Efficiency Anomalies Batch Streaming
- 16. IoT Central IoT Hub Event Hub «Down the event stream»
- 17. …musing… DEMO
- 18. • Azure Stream Analytics • Azure Container Apps & co • Apache Spark (and again Synapse) Event Processing
- 19. …musing… DEMO
- 20. • ML.NET • Spark • ADX (and again Synapse) • Notebooks • Azure Functions • Cognitive Services • Anomaly Detector • Custom Vision Anomaly Detection
- 21. …musing… DEMO
- 22. • Azure love Blobs • Developers love Blobs • Developers love services that handles Blobs as first class citizens • No relational • No file system Lesson learned
- 24. Necessità: complex messages/entities, no mapping { "UserId": "123456", "UserName": "Mario Rossi", "EMail": "mario.rossi@somewhere.it", "StoreInfo": { "PreferredStore": { "PreferredStoreId": "12876", "PreferredStoreName": «Dovunque di Corso Vittorio" } } } { "UserId": "123456", "Reservations": [ { "LastUpdate": "", "CodiceArticolo": "", "StoreId": "", "AvailableParts": 4, "ProductWarehouseHash": "638AB6D83H" } ] } { "CodiceArticolo": "AK2457", "Descrizione": "Prodotto bello", "GiacenzaAttuale": 100, "Negozi": { "CodiceNegozio": { "GiacenzaAttuale": 20, "LastUpdate": "2021-04-26T10:20:30" } }, "LastUpdate": "2021-04-26T10:20:30" } { "CodiceArticolo": "AK2457", "Descrizione": «Prodotto", "Codici": { "B3812": { "Descrizione": "Prodotto bello", "Usage": "blablabla" } }, "DistintaBase": [ { "CodiceParte": "AK2457-P1" }, { "CodiceParte": "AK2457-P2" } ], "LastUpdate": "2021-04-26T10:20:30" }
- 25. Column-family Document Graph Turnkey global distribution Elastic scale out of storage & throughput Guaranteed low latency at the 99th percentile Comprehensive SLAs Five well-defined consistency models Table API Key-value Cosmos DB’s API for MongoDB Azure Cosmos Db
- 26. Modeling challenge #1: To embed or reference? { "menuID": 1, "menuName": "Lunch menu", "items": [ {"ID": 1, "ItemName": "hamburger", "ItemDescription":...} {"ID": 2, "ItemName": "cheeseburger", "ItemDescription":...} ] } { "menuID": 1, "menuName": "Lunch menu", "items": [ {"ID": 1} {"ID": 2} ] } {"ID": 1, "ItemName": “hamburger", "ItemDescription":...} {"ID": 2, "ItemName": “cheeseburger", "ItemDescription":...}
- 27. hash(City ID) Beijing Seattle Paris Redmon d Shanghai Tianjin Berlin Austi n Range 1 Range 2 Range 5 RU’s divided among Partitions
- 28. Multi-Master – Read/Write in any region Benefits • Write scalability around the world • Low latency (<10ms P99 for 1kb document) writes around the world • 99.999% High Availability around the world • Well-defined consistency models • Automatic conflict management
- 29. Common Change Feed Scenarios
- 30. Request Units Each request consumes # of RU Approx. 1 RU = 1 read of 1 KB document Approx. 5 RU = 1 write of a 1KB document Query: Depends on query & documents involved GET POST PUT Query … = = = =
- 31. Operation Type # Requests per sec # RU's per Request RU's Needed Write Single Document 10,000 10 100,000 Top Query #1 700 100 70,000 Top Query #2 200 100 20,000 Top Query #3 100 100 10,000 Total RU/s 200,000 RU/s Estimating Required RU’s
- 32. Storage Cost Avg Record Size (KB) 1 Number of Records 100,000,000 Total Storage (GB) 100 Monthly Cost per GB $0.25 Expected Monthly Cost for Storage $25.00 Throughput Cost Operation Type Number of Requests per sec Avg RU's per Request RU's Needed Create 100 5 500 Read 400 1 400 Total RU/sec 900 Hourly Cost per 100 RU/sec $0.008 Monthly Cost per 100 RU/sec $6.00 Expected Monthly Cost for Throughput $54.00 Total Monthly Cost [Total Monthly Cost] = [Monthly Cost for Storage] + [Monthly Cost for Throughput] = $25 + $54 = $79 per month * pricing may vary by region; for up-to-date pricing, see: https://azure.microsoft.com/pricing/details/cosmos-db/ Pricing Example
- 33. Service Bus Architettura generica per User Story • Microservizio responsabile del dominio applicativo e della sua consistenza e persistenza • API di frontend per gestire le richieste delle singole user stories specifiche verso il microservizio • Gestione in memory con Redis • In caso di fail lettura da Cosmos • In caso di update • Aggiornamento di Redis • Accodamento in serviceBus • Batch aggiorna Cosmos per abbassare RU API #1 Micro Service Cosmos DB REDIS Caching Update Batch API #n Read Write Command Read Invoke Invalidate Cache Write Invoke Read Command
- 35. • Azure love No SQL and Cosmos DB • Developers love No SQL and Json • Developer hates(?) Cosmos Db pricing? • Developers loves (fuffa style ) microservices • Everything has solution... Lesson learned
- 37. • È tutto Azure! • Le dashboard, se si può, si fanno in Power BI (8€/utente/mese, ricordatevi, anche solo per visualizzare) • Governance dello sviluppo • Premium Capacity • Developers • Deployment BI? Power BI? Cosa c’entra con Azure?
- 38. • Developer maleducati • Power BI developer MOLTO maleducati Scenario
- 39. • PowerShell • Azure DevOps • Git non è pbix-friendly • Ma è comunque un repo versionato • E le pipeline sono un ottimo strumento di automation Ops, Deployments
- 40. …musing… DEMO
- 41. • DR of a DataWarehouse: big problem • BIOps is not a practice Lesson learned
- 43. SQL virtual machines Managed instances Azure SQL Databases • SQL Server surface area (vast majority) • Native virtual network support • Fully managed service • SQL Server and OS server access • Expansive SQL and OS version support • Automated manageability features for SQL Server • Hyperscale storage (up to 100TB) • Serverless compute • Fully managed service • Resource sharing between multiple databases to price optimize • Simplified performance management for multiple databases • Fully managed service
- 44. Azure SQL Database — Everything built-in •Business continuity High availability Automated backups Long term backup retention Geo-replication •Scale Advanced security Automatic tuning Built-in monitoring Built-in intelligence
- 45. Azure SQL: multi-faceted service
- 46. On-demand flexible scale Operate at the true rhythm of your business Fully managed & intelligent Focus on your applications, not your infrastructure Cost-effective Pay for performance. Period. Adapts compute resources to the workload without sacrificing performance Automatically pauses and resumes Fully-managed and intelligent database service Built-in 99.99% availability Pay only for compute resources you consume, on a per-second basis Further optimize costs with configurable compute thresholds Best for unpredictable and intermittent workloads on single databases, such as: Dev/test E-commerce Line of Business
- 48. COnclusions
Editor's Notes
- Azure Cosmos DB offers the first globally distributed, multi-model database service for building planet scale apps. It’s been powering Microsoft’s internet-scale services for years, and now it’s ready to launch yours. You can add Azure locations to your database anywhere across the world, at any time, with a single click. Cosmos DB will seamlessly replicate your data and make it highly available. Cosmos DB allows you to scale throughput and storage elastically, and globally! You only pay for the throughput and storage you need – anywhere in the world, at any time.
- 27
- The number of RU’s each operation consumes depends on many factors which include: Document size Number of indexed fields Type of indexes Consistency model choice Not all queries will consume equal numbers of RU’s. Some operations are more computationally complex or require scans through more documents and therefore use more RU’s.
- Business continuity enables your business to continue operating in the face of disruption, particularly to its computing infrastructure. High availability of Azure SQL Database guarantees your databases are up and running 99.99% of the time, no need to worry about maintenance/downtimes. 2 read replicas - GP 3 replicas, 1 read-scale replica, zone-redundant HA - BC Primary read/write replica + up to 4 read replicas - HS Automated backups are created and use Azure read-access geo-redundant storage (RA-GRS) to provide geo-redundancy. Also Point in Time restore Long term backup retention enables you to store specific full databases for up to 10 years. Geo-replication by creating readable replicas of your database in the same or different data center (region). You can manually failover to readable replicas Also Auto-failover groups allows an application to recover in case of a data center outage Scale by easily adding more resources (CPU, memory, storage) without long provisioning. Advanced security detects threats and vulnerabilities in your databases and enables you to secure your data. Automatic tuning analyzes your workload and provides you the recommendations that can optimize performance of your applications by adding indexes, removing unused indexes, and automatically fixing the query plan issues. Built-in monitoring capabilities enable you to get the insights into performance of your databases and workload, and troubleshoot the performance issues. Built-in intelligence automatically identifies the potential issues in your workload and provides you the recommendations that can help you to fix the problems.
- Azure SQL Database serverless is our answer to cost-effectively resourcing these types of unpredictable and intermittent workloads. It is a dynamically scaling, on-demand version of Azure SQL Database that eliminates the complexity of resourcing and managing unpredictable workloads. Simply create a serverless database and connect your application, with no upfront resource configuration required. Serverless SQL databases automatically pause, resume and scale compute based upon your app’s requirements, making them particularly cost effective for variable or unpredictable workloads – or when you’re simply unsure of your requirements. With resources available on-demand, serverless SQL databases optimize costs with pay per-second billing that aligns with the app’s performance, so you only pay for the compute resources you use . Built upon the SQL Server architecture, serverless SQL databases are fully-managed, always up to date and highly available with a 99.99% uptime guarantee. Combined with built-in intelligence to optimize database performance and security, serverless SQL Databases help you be more productive so you can focus more on what you do best, building great apps faster and more cost effectively.