Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Move a successful onpremise oltp application to the cloud

441 views

Published on

Precon slides for PASS Summit 2019

Published in: Technology
  • Be the first to comment

Move a successful onpremise oltp application to the cloud

  1. 1. Move a Successful, Legacy OLTP Application to Azure Ike Ellis, MVP General Manager – Data & AI Practice Solliance
  2. 2. Please silence cell phones
  3. 3. everything PASS has to offer Free online webinar events Free 1-day local training events Local user groups around the world Online special interest user groups Business analytics training Get involved Free Online Resources Newsletters PASS.org Explore
  4. 4. Ike Ellis General Manager – Data & AI Practice Solliance /ikeellis @ike_ellis www.ikeellis.com • Founder of San Diego Power BI and PowerApps UserGroup • San Diego Tech Immersion Group • MVP since 2011 • Author of Developing Azure Solutions, Power BI MVP Book • Speaker at PASS Summit, SQLBits, DevIntersections, TechEd, Craft
  5. 5. Session Evaluations Submit by 5pm Friday, November 15th to win prizes. Download the GuideBook App and search: PASS Summit 2019 Follow the QR code link on session signage Go to PASSsummit.com 3 W A Y S T O A C C E S S
  6. 6. agenda • introduction of ike and dustin and the topic • introduction of topic and why are we here • preparation necessary to move the database to the cloud • first ~ move to an azure vm • second ~ move to azure sql database managed instance • third ~ move to pass offerings • what does the future look like
  7. 7. do you have something like this? huge sql database app 1 app 2 app 3
  8. 8. but really it’s like this: huge sql database app 1 app 2 app 3 excel ssrs reports ssis etl custom app data dump to vendor sql agent job that closes the month somebody hand made a lookup table for one report linked server for accounting process stored procs on a different server email notification for alerting power bi
  9. 9. not a cloud app: reason #1: too expensive • in the cloud, you pay for the following • compute • network • disk • memory • but the most expensive thing by far: • compute • when the database is that large, you are paying for a lot of compute for a database that is essentially hosting a lot of data at rest
  10. 10. not a cloud app: reason #2: too expensive huge sql database scales up as a unit. this is so big that if we need more power, we have to scale up the entire huge thing this tiny piece here is really important and should scale separately
  11. 11. not a cloud app: reason #3: bad messaging huge sql database app 1 app 2 this is an expensive and slow message bus that takes a lot of grooming. you’re getting overcharge for storage and compute
  12. 12. not a cloud app: reason #4: too hard to change huge sql database app 1 app 2 app 3 excel ssrs reports ssis etl custom app data dump to vendor sql agent job that closes the month somebody hand made a lookup table for one report linked server for accounting process stored procs on a different server email notification for alerting power biin order to change it, look at how many things have to be tested and verified and deployed together
  13. 13. not a cloud app: reason #4: too hard to deploy huge sql database app 1 app 2 app 3 excel ssrs reports ssis etl custom app data dump to vendor sql agent job that closes the month somebody hand made a lookup table for one report linked server for accounting process stored procs on a different server email notification for alerting power biin order to deploy this, look at how many things have to be tested and verified and deployed together
  14. 14. not a cloud app: reason #4: too hard to upgrade huge sql database app 1 app 2 app 3 excel ssrs reports ssis etl custom app data dump to vendor sql agent job that closes the month somebody hand made a lookup table for one report linked server for accounting process stored procs on a different server email notification for alerting power bi in order to upgrade this to a new version of sql, look at how many things have to be tested and verified and upgraded together
  15. 15. reason #5: not secure huge sql database pii data all up in this
  16. 16. reason #6: all kinds of performance problems huge sql database for far too many years the answer to what to do to solve all kinds of performance problems was: throw hardware at it until it goes away now we have 32 cores with an average cpu utilization of 2% the cloud will punish you for decisions like this
  17. 17. simple example of a cloud-scale application much smaller sql database app 1 cosmosdb app 2 much smaller sql database app 3
  18. 18. how do they integrate? much smaller sql database app 1 cosmosdb app 2 much smaller sql database app 3
  19. 19. why do we do this? • performance and scaling • cheaper • changing • deployment • security (limits the compromise) • ease of upgrading
  20. 20. example 1 of a cloud-scale application
  21. 21. example 2 of a cloud-scale application
  22. 22. #1 rule of a cloud scale data application no one ever touches the the database directly. no one. ever. not even once. everyone who wants data needs to go through the application layer. couple this rule with the rule: “one application, one purpose, one database” if you remember nothing else today, please remember this
  23. 23. follow this rule and your database will stay small, decoupled, vibrant, clean, upgradeable, and scale-able
  24. 24. but how do we get from here: huge sql database app 1 app 2 app 3 excel ssrs reports ssis etl custom app data dump to vendor sql agent job that closes the month somebody hand made a lookup table for one report linked server for accounting process stored procs on a different server email notification for alerting power bi
  25. 25. to here: much smaller sql database app 1 cosmosdb app 2 much smaller sql database app 3
  26. 26. that’s what we’re going to talk about today
  27. 27. there are four different ways to get from on-premise to a proper cloud application • lift and shift to a vm • find a natural severing point and only move part of the application to the cloud • decouple the large sql database and move it to an azure sql database product (managed instance, hyperscale) • rewrite the entire application from scratch into paas offerings
  28. 28. I propose instead of choosing between these ways that you do all four of them, so think of them as ordered steps that will take you where you want to go
  29. 29. preparation
  30. 30. first things first: let’s pay off some technical debt • before you can begin a cloud migration of a database, here are some great things you can do to get ready for the project
  31. 31. decouple from the server name • create a dns cname for your sql server • change the connectionstring for all applications, excel, ssis, ssrs, power bi, linked servers, backup tools, ssms, ssas, and everything else you can think of • make a list of every single location where you had to make the change • this way when the server changes location, you can just change the cname • if you really want to see if you succeeded at this or not, rename the server • https://docs.microsoft.com/en-us/sql/database-engine/install- windows/rename-a-computer-that-hosts-a-stand-alone-instance-of-sql- server?view=sql-server-ver15
  32. 32. decouple database names • look for three part object names in stored procedures in this database and other databases • worldwideimporters.dbo.customers
  33. 33. make the database smaller • every time i ask developers or business users if we can delete some data, the answer is always no • when we actually look at the data, there is plenty we can either delete or move • really old data • audit logs and audit records • temp tables • tables that are empty or have really old data in them or rarely used data • reporting tables that really should be in a data warehouse • indexes that no one ever uses • remove overlapping indexes
  34. 34. find unused indexes SELECT objects.name AS Table_name, indexes.name AS Index_name, dm_db_index_usage_stats.user_seeks, dm_db_index_usage_stats.user_scans, dm_db_index_usage_stats.user_updates FROM sys.dm_db_index_usage_stats INNER JOIN sys.objects ON dm_db_index_usage_stats.OBJECT_ID = objects.OBJECT_ID INNER JOIN sys.indexes ON indexes.index_id = dm_db_index_usage_stats.index_id AND dm_db_i ndex_usage_stats.OBJECT_ID = indexes.OBJECT_ID WHERE AND dm_db_index_usage_stats.user_lookups = 0 AND dm_db_index_usage_stats.user_seeks = 0 AND dm_db_index_usage_stats.user_scans = 0 ORDER BY dm_db_index_usage_stats.user_updates DESC
  35. 35. find overlapping indexesselect s.Name + N'.' + t.name as [Table] ,i1.index_id as [Index1 ID], i1.name as [Index1 Name] ,dupIdx.index_id as [Index2 ID], dupIdx.name as [Index2 Name] ,c.name as [Column] from sys.tables t join sys.indexes i1 on t.object_id = i1.object_id join sys.index_columns ic1 on ic1.object_id = i1.object_id and ic1.index_id = i1.index_id and ic1.index_column_id = 1 join sys.columns c on c.object_id = ic1.object_id and c.column_id = ic1.column_id join sys.schemas s on t.schema_id = s.schema_id cross apply ( select i2.index_id, i2.name from sys.indexes i2 join sys.index_columns ic2 on ic2.object_id = i2.object_id and ic2.index_id = i2.index_id and ic2.index_column_id = 1 where i2.object_id = i1.object_id and i2.index_id > i1.index_id and ic2.column_id = ic1.column_id ) dupIdx order by s.name, t.name, i1.index_id if the leading edge is the same, the indexes can likely be combined
  36. 36. implement filtered indexes my feeling is that every index should be filtered for legacy databases that have been around for more than 3 years create nonclustered index IDX_Data_Unprocessed_Filtered on dbo.Data(RecId) include(Processed) where Processed = 0;
  37. 37. make the database smaller: defrag and shrink • many of us have databases that have not been properly maintained • defrag! • get rid of the free space in datafiles • find the worst offenders: select f.type_desc as [Type] ,f.name as [FileName] ,fg.name as [FileGroup] ,f.physical_name as [Path] ,f.size / 128.0 as [CurrentSizeMB] ,f.size / 128.0 - convert(int,fileproperty(f.name,'SpaceUsed')) / 128.0 as [FreeSpaceMb] from sys.database_files f with (nolock) left outer join sys.filegroups fg with (nolock) on f.data_space_id = fg.data_space_id option (recompile)
  38. 38. make the database smaller: compression • introduce page compression • columnstor indexing
  39. 39. make the database smaller: get the blobs out • blobs don’t belong in a database • move them to azure blob storage first! • c# is probably how you got them in there, write a c# app to get them out and move them to abs • save the url to the file in the database as a varchar(100) • change the application to pull from abs instead of the database • benefits of getting the blobs • significantly reduces the size of the database • backed up automatically • much cheaper to save those files in abs than pay to have azure sql database store them • consistent load time no matter how many there are in there. sql can’t say that • with abs we don’t pay for compute time! just storage! • if you don’t do this, at least compress them where they are, but that’s a similar amount of work
  40. 40. find bad actors with the network • use profiler, extended events, wireshark, fiddler, chrome network tab • look for bad actors who are using too much network • it’s easy to find pages that are doing the exact same code twice • or have n+1 problems • or are bringing back too much data and not paging it • saving the network will make your app more cloud-ready
  41. 41. before the cloud migration, move your backup/restore process to azure • if you’re currently backing up your databases to an on-premise location, change that process to move it to azure blob storage • this is a great way to actually get your database to the cloud in bak form • azure blob storage can be both a backup destination and a restore source!
  42. 42. consider an on-premise upgrade of sql server • upgrades will get you on the latest version of the sql server optimizer and will prepare you to use an azure sql database version of the optimizer • will give you features for compression and data management that you might not have right now
  43. 43. download some things • download and install the latest version of the MAP Toolkit. • provides a readiness report to migrate the server • download and install the latest version of the Database Experimentation Assistant • download and install the Data Migration Assistant v3.3 or later • create an instance of sql server on azure vm’s by following the detail in the article How to provision a Windows SQL Server virtual machine in the Azure portal
  44. 44. azure express route
  45. 45. take care of azure active directory https://docs.microsoft.com/en-us/azure/active-directory/
  46. 46. options for extending active directory • ad connect from a machine that’s on prem and that connects to azure active directory • active directory domain controller on a vm • create groups in ad on prem that map to what you want to secure in azure • add rights for resource groups • create simple groups on prem in ad that are the exact names of the resource groups in your ad subscription • demo: resource groups
  47. 47. some things you should know about azure • I might be making all this stuff up • history of aws and rds • aws loves iaas • most of their paas offerings are just managed iaas • history of azure • azure loves paas • they try to be “more cloudy” • iaas story is lacking • history of azure sql server managed instance
  48. 48. now let’s do step 1! migrate sql server first to an azure virtual machine!
  49. 49. choose the right vm size • watch the cpu/memory ratio • use performance monitor to determine hardware requirements • test them all! • not all types are available in all azure regions, so choose your region carefully
  50. 50. it’s all about the disks (i/o) baby disk options: https://azure.microsoft.com/en-us/pricing/details/managed-disks/ • premium ssd managed disks • high performance, managed disks • ssd
  51. 51. ultra disks not available in all regions
  52. 52. storage space: ack! • stay away from storage spaces • taxes the cpu and that’s expensive • doesn’t really give you more iops
  53. 53. azure vm: other disk recommendations • remember premium ssd or ultra ssd. • Standard storage is only recommended for dev/test. • keep the storage account and vm in the same region. • disable azure geo-redundant storage (geo-replication) on the storage account.
  54. 54. remember your disk best practices • Use a minimum of 2 p30 disks (1 for log files and 1 for data files including tempdb). • For workloads requiring ~50,000 IOPS, consider using an Ultra SSD. • avoid using operating system or temporary disks for database storage or logging. • enable read caching on the disk(s) hosting the data files and tempdb data files. • do not enable caching on disk(s) hosting the log file. • important: Stop the SQL Server service when changing the cache settings for an azure vm disk. • stripe multiple azure data disks to get increased IO throughput • but be careful! you will load the cpu here! • format with documented allocation sizes. • place tempdb on the local ssd d: drive for mission critical sql server workloads (after choosing correct VM size).
  55. 55. ways to do the migration • backup/restore – VM • log shipping - VM • attach/detach - VM • Snapshot Replication – VM/SQL DB/MI • Transactional Replication – VM/SQL DB/MI
  56. 56. a particular way to do the migration • azure migrate • https://azure.microsoft.com/en-us/services/azure-migrate/ • azure site recovery • https://azure.microsoft.com/en-us/services/site-recovery/ • good if it’s a single server • not a cluster • you are going to keep everything the same • you will turn off the original server • keep everything the same with the same name • make sure nothing is using a hard-coded ip address
  57. 57. excellent resource for migrations https://docs.microsoft.com/en-us/azure/migrate/contoso-migration-overview https://docs.microsoft.com/en-us/azure/cloud-adoption- framework/migrate/azure-best-practices/contoso-migration-infrastructure https://docs.microsoft.com/en-us/azure/cloud-adoption- framework/migrate/azure-best-practices/contoso-migration-infrastructure#step- 2-manage-hybrid-identity written by my friend, dan patrick
  58. 58. zero downtime migration model • also gives you a way to rollback • can move read-only workloads to azure first like reporting, ssas, ssis
  59. 59. demo: alwayson availability groups
  60. 60. tutorials for using always on ags in azure https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual- machines-windows-portal-sql-availability-group-prereq https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual- machines-windows-portal-sql-availability-group-tutorial
  61. 61. step 2 – move to azure sql database managed instance
  62. 62. who is azure sql database managed instance for? customers looking to migrate a large number of apps from on-premise or iaas, self-built or isv provided, with as low migration effort as possible & cost being a crucial factor
  63. 63. • enable full isolation from other tenants without resource sharing • promote secure communication over private IP addresses with native vnet integration • enable your on-premise identities on cloud instances, through integration with azure active directory and ad connect • combine the best of sql server with the benefits of a fully-managed service • use familiar sql server features in sql database managed instance VNET support in SQL Database Managed Instance
  64. 64. competitive tco azure sql db offers the most cost-effective path to the cloud
  65. 65. convert on-premises cores to vcores to maximize value of investments 1 standard license core = 1 general purpose core 1 enterprise license core = 1 business critical core 1 enterprise license core = 4 general purpose cores (virtualization benefit) SQL Database vCore-based options SQL Server with Software Assurance SQL Server license trade-in values
  66. 66. reserve azure sql db resources in advance and save up to 33%1 • budget and forecast better with upfront payment for one-year or three-year terms • get prioritized compute capacity in azure regions • exchange or cancel reservations as your needs evolve • scale up or down within a performance tier and region with auto-fit • move saas apps between elastic pools and single databases and keep your reserved instance benefit License included With Azure Hybrid Benefit Up to 55% savings2 Reserved Instances with Azure Hybrid Benefit Up to 80% savings3
  67. 67. • discounted rates up to 55% off to support your ongoing development and testing • dev/test pricing available for vcore- based deployment options • eligible with active visual studio subscription
  68. 68. azure relational database platform Azure storage Azure compute Database Services Platform Intelligent Flexible Trusted
  69. 69. azure relational database platform Azure storage Azure compute Database Services Platform Single Elastic Pool Managed Instance Standard Business Critical
  70. 70. • networking can be a challenge • opening ports should be done sparingly • a jump box on your vnet will let you manage azure sql db mi • a place for all your tools • you can pause it to stop getting billed for it
  71. 71. demo “It’s just SQL”
  72. 72. azure db migration journey Assess Optimize Migrate Enables optimization during or post migration (fully managed service) IaaS (virtual machines) fall short here Enables rehosting or light refactoring for most apps Eliminates the need to rearchitect or rebuild your apps SQL
  73. 73. • Fully-fledged SQL instance with nearly 100% compat with on-premise • Built on the same infrastructure as SQL Database • Provides the same benefits (PaaS) • Contained within your VNet • Private IP addresses • Express Route / VPN connectivity SQL Database (PaaS) Elastic Pool Managed Instance Single database • Transparent • Frictionless • Competitive
  74. 74. put your dbs on autopilot and focus on your business… * - features coming soon mi has it built-in compute & storage provisioned on demand fast & online scaling full stack updates and patches backups with health checks point-in-time restore (configurable retention *) 99.99% availability with automatic failover disaster recovery with single geo secondary (multiple*)
  75. 75. It’s much easier with the mi compliance with all major industry standards threat detection with automatic alerting intelligent query processing automatic performance tuning* monitoring at scale with intelligent insights data discovery and classification* vulnerability assessment put your dbs on autopilot and focus on your business… * - features coming soon
  76. 76. • Fully-fledged SQL instance with nearly 100% compat with on-premise • Built on the same infrastructure as SQL Database • Provides the same benefits (PaaS) • Contained within your VNet • Private IP addresses • Express Route / VPN connectivity SQL Database (PaaS) Elastic Pool Managed Instance Single database • Transparent • Frictionless • Competitive
  77. 77. cross-db queries & transactions, linked servers to sql .net clr modules service broker change data capture transactional replication easily migrate from sql server & modernize * - features coming soon choice of instance collations* and instance time zone* R services* msdtc for distributed transactions filestream / filetable, polybase
  78. 78. dmvs, xevents, query store sql agent and db mail sysadmin privileges and resource governor easily migrate from sql server & modernize sql auditing, rls tde, always encrypted, and dynamic data masking built-in ha replaces on-prem setups replace mdw with oms monitoring network security with vnetss and private IPs integrated auth. with azure ad
  79. 79. • Fully-fledged SQL instance with nearly 100% compat with on-premise • Built on the same infrastructure as SQL Database • Provides the same benefits (PaaS) • Contained within your VNet • Private IP addresses • Express Route / VPN connectivity SQL Database (PaaS) Elastic Pool Managed Instance Single database • Transparent • Frictionless • Competitive
  80. 80. dedicated resources through customer isolation
  81. 81. isolation and connectivity of mi
  82. 82. • Fully-fledged SQL instance with nearly 100% compat with on-premise • Built on the same infrastructure as SQL Database • Provides the same benefits (PaaS) • Contained within your VNet • Private IP addresses • Express Route / VPN connectivity SQL Database (PaaS) Elastic Pool Managed Instance Single database • Transparent • Frictionless • Competitive
  83. 83. mi: service tiers Capability Service tier General Purpose (GA) Business Critical (Public Preview) Best for Apps with typical availability and common IO latency requirements Apps with highest availability and lowest IO latency requirements. Compute (vCores) 8, 16, 24, 32, 40, 64, 80 8, 16, 24, 32, 40, 64, 80 HA / Recovery Time Objective Remote storage based / Good Always On AG based / Better Storage type / size Fast remote (Azure Premium) / Up to 8 TB Super-fast local SSD / Up to 4 TB Read scale out (read-only replica) No Yes In-Memory OLTP No Yes Price competitive with AWS? Yes, ~33% lower (license included) Yes, ~46% lower (license included)
  84. 84. sql db deployment model overview Azure SQL Database Unit of Monetization Tiering Pricing vs. Competitors  Basic: designed for apps with light workloads  Standard: mid-level performance and business continuity  Premium: low IO latency workloads and higher business continuity  General Purpose  “Business Critical” Hybrid Benefits DTU – “Database Throughput Unit” – measure of database performance that blends CPU, memory and I/O.  vCore for compute  GBs for storage  IOPs for IO Yes, EE customers also get 4 cores in General Purpose SKU  Basic – very cheap because it priced to accommodate web customers  Standard – comparable pricing but not easily explainable to customer  Premium – expensive due to additional replicas and IOs  Priced lower compared to AWS  No Best for New apps, with a ‘one database per app pattern’ and resources guaranteed at DB level Modernizing large number of existing SQL Server apps from on-premises or IaaS New SaaS apps or modernizing existing apps to SaaS, resource sharing across DBs of existing LOB apps for higher efficiency eDTU – elastic “Database Throughput Unit” – measure of database performance that blends CPU, memory and I/O.
  85. 85. mi start provisioning in azure portal download template to automate quickStart walkthrough on docs https://docs.microsoft.com/en-us/azure/sql- database/sql-database-managed-instance-get-started
  86. 86. connecting to an mi Instance Driver/tool​ Version​ .NET Framework​ 4.6.1 (or .NET Core)​ ODBC driver​ v17​ PHP driver​ 5.2.0​ JDBC driver​ 6.4.0​ Node.js driver​ 2.1.1​ OLEDB driver​ 18.0.2.0​ SSMS​ 17.8.1 or higher
  87. 87. deployment resources portal deployment walk-through https://docs.microsoft.com/en-us/azure/sql-database/sql-database-managed-instance-create-tutorial-portal https://docs.microsoft.com/en-us/azure/sql-database/sql-database-managed-instance-vnet-configuration arm template deployment walk-through https://blogs.msdn.microsoft.com/sqlserverstorageengine/2018/07/02/deploy-azure-sql-managed-instance-network- environment-using-arm/ https://blogs.msdn.microsoft.com/sqlserverstorageengine/2018/05/15/creating-azure-sql-managed-instance-using- arm-templates/
  88. 88. enterprise-grade security that is easy to use PHYSICAL SECURITY NETWORK SECURITY CLUSTER SECURITY ACCESS MANAGEMENT THREAT PROTECTION CUSTOMER DATA INFORMATION PROTECTION 17 552 207 90 59 33 0 100 200 300 400 500 600 Vulnerabilities(2010–2018)
  89. 89. securing the network PHYSICAL SECURITY NETWORK SECURITY PLATFORM SECURITY ACCESS MANAGEMENT THREAT PROTECTION INFORMATION PROTECTION CUSTOMER DATA
  90. 90. Service dependencies Azure Azure Management Legend: Data Customer applications & tools On-Premises + Azure VNet Azure SQL Database Managed Instance VNet Management & Deployment Plane Azure Maintenance Corpnet certificate verification Certificate Authorities Internet TLS TLS TLS TLS Certificate
  91. 91. hub & spoke architecture with MI On-premises network Management subnet Active Directory subnet Hub virtual network Gateway Subnet JumpBox ExpressRout e Gateway DMZ Subnet NVA Management subnet Spoke 1 virtual network JumpBox Workload subnet Management subnet Spoke 2 virtual network JumpBox Workload subnet ExpressRout e circuit vnet peering vnet peering Managed Instance subnet
  92. 92. host your application in the cloud or keep on-premises
  93. 93. be empty: have specific route table: optional custom dns: no service endpoint: sufficient ip addresses: virtual network considerations https://docs.microsoft.com/en-us/azure/sql-database/sql-database-managed-instance-vnet-configuration#requirements
  94. 94. an mi instance must be deployed in an azure virtual network • allows for connecting directly from an on-premises network • allows for connecting linked servers or other on-premises data stores • allows for connecting to additional azure resources plan your deployment • managed instance requires a minimum of 16 IP addresses in a subnet and may use up to 256 IP addresses • if deploying multiple managed instances inside the subnet, you need to optimize the subnet size • the default values create a subnet that takes all the vnet address space, allowing for only mi inside the virtual network routes • Effective routes on the Managed Instance subnet are not supported • Routes can be user-defined (UDR) or Border Gateway Protocol (BGP) routes propagated to network interfaces through ExpressRoute or site-to-site VPN connections • For BGP routes, create a 0.0.0.0/0 Next Hop Internet route and apply it to the Managed Instance subnet network security groups (nsg) NSGs on the Managed Instance subnet are not supported virtual network guidance
  95. 95. considerations when creating a new virtual network for mi • calculate the subnet size • assess the needs for the rest of the vnet • disable service end points • create new arm virtual network virtual network guidance Name Any valid name Address space Any valid address range, such as 10.14.0.0/24 Subscription Your subscription Resource Group Any valid resource group (new or existing) Location Any valid location Subnet name Any valid subnet name, such as mi_subnet Subnet address range Any valid subnet address, such as 10.14.0.0/28. Use a subnet address space smaller than the address space itself to allow space to create other subnets in the same VNet, such as a subnet for hosting test / client apps or gateway subnets to connect from on- prem or other VNets. Service endpoints Disabled
  96. 96. create the required route table and associate it • Cceate new Route table • 0.0.0.0/0 Next Hop Internet** • associate route table with the mi subnet virtual network guidance Name Any valid name Subscription Your subscription Resource Group Select the resource group you created in the previous procedure Location Select the location you specified in the previous procedure Disable BCP route propagation Enabled
  97. 97. n-tier architecture in on-premise/customer data center Data center network Web tier Internet Load balancer Business tier Data tier Management Active Directory JumpBox SQL HA Cluster Load balancer Load balancer
  98. 98. n-tier architecture in azure Data center network Web tier subnet Internet Azure application gateway Business tier subnet Managed Instance subnet Management subnet JumpBox Load balancer Azure Active Directory
  99. 99. securing the platform PHYSICAL SECURITY NETWORK SECURITY PLATFORM SECURITY ACCESS MANAGEMENT THREAT PROTECTION INFORMATION PROTECTION CUSTOMER DATA Enhanced monitoring Collecting of low-fidelity anomalous activity Monitoring PERF for traits Red Team / Blue Team
  100. 100. virtual cluster Virtual cluster VNe t Node TDS endpoint (Private IP) SQL EngineSQL Management Node Agent LB ILB G W G W G W mymi.<clusterid>.database.windows.net TLS TLS TLS Windows Firewall SQL Management (Public IP) Nodeprimary node Node SQL Management (public IP) TDS endpoint (private IP) SQL
  101. 101. platform/service security assume breach… enhanced monitoring of our azure assets collection of low-fidelity anomalous activity (automated hunting) monitoring PERF for traits of crypto currency mining … and large set of other detections that we don’t talk about publicly attack team, sql red team, tries to get in, gain a foothold, escalate privileges, and maintain persistence sql blue team practices defense-in-depth when we detect something, e.g., failed login attempts, we defend… if it involves customer, we will notify many times this is the customer’s own security and compliance scanners!
  102. 102. configuring authentication PHYSICAL SECURITY NETWORK SECURITY PLATFORM SECURITY ACCESS MANAGEMENT THREAT PROTECTION INFORMATION PROTECTION CUSTOMER DATA Azure Active Directory Provides an alternative to SQL authentication. Helps to stop the proliferation of user identities across Azure SQL logical servers and SQL DBs Use benefits provided by Azure AD
  103. 103. azure active directory authentication Simplified Permission Management Central ID Management Can help you Eliminate Storing Passwords Flexible Configuration Supported in many Tools and Drivers Universal / Interactive Authentication
  104. 104. monitoring and auditing PHYSICAL SECURITY NETWORK SECURITY PLATFORM SECURITY ACCESS MANAGEMENT THREAT PROTECTION INFORMATION PROTECTION CUSTOMER DATA SQL Auditing Integration with Log Analytics and Event Hubs for SQL DB Singletons/pools (coming to M.I. soon) Advanced Threat Protection Vulnerability Assessment Threat Monitoring (3 classes of threats)
  105. 105. protecting the data PHYSICAL SECURITY NETWORK SECURITY PLATFORM SECURITY ACCESS MANAGEMENT THREAT PROTECTION CUSTOMER DATA INFORMATION PROTECTION Encryption-in-flight (aka. Transport Layer Security TLS) Encryption-at-rest (aka. Transparent Data Encryption TDE) Service- or User-managed keys Backup encryption Encryption-in-use (Always Encrypted)* Key management with Azure Key Vault
  106. 106. authentication SSMS SSDT Connection string-based authentication SQL package Azure SQL Database ADO.NET 4.6 Application Active Directory Azure Active Directory
  107. 107. general purpose Feature Description Number of vCores* 8, 16, 24 (Gen 4) 8, 16, 24, 32, 40, 64, 80 (Gen 5) SQL Server version / build SQL Server (latest available) Min storage size 32 GB Max storage size 8 TB Max storage per database Determined by the max storage size per instance Expected storage IOPS 500-7500 IOPS per data file (depends on data file). See Premium Storage Number of data files (ROWS) per the database Multiple Number of log files (LOG) per database 1 Managed automated backups Yes HA Based on remote storage and Azure Service Fabric Built-in instance and database monitoring and metrics Yes Automatic software patching Yes VNet - Azure Resource Manager deployment Yes VNet - Classic deployment model No Portal support Yes
  108. 108. business critical Feature Description Number of vCores* 8, 16, 24, 32 (Gen 4) 8, 16, 24, 32, 40, 64, 80 (Gen 5) SQL Server version / build SQL Server (latest available) Additional features In-Memory OLTP 1 additional read-only replica (Read Scale-Out) Min storage size 32 GB Max storage size •Gen 4: 1 TB (all vCore sizes Gen 5:1 TB for 8, 16 vCores •2 TB for 24 vCores •4 TB for 32, 40, 64, 80 vCores Max storage per database Determined by the max storage size per instance Number of data files (ROWS) per the database Multiple Number of log files (LOG) per database 1 Managed automated backups Yes HA Based on Always On Availability Groups and Azure Service Fabric Built-in instance and database monitoring and metrics Yes Automatic software patching Yes VNet - Azure Resource Manager deployment Yes VNet - Classic deployment model No Portal support Yes Business Critical service tier: collocated compute and storage Primary endpoint (read-write) Read-only endpoint Always On AG SQL SQL SQL SQL Super-fast SSD Primary replica Secondary replica Secondary replicaSecondary replica
  109. 109. built-in high availability P S S Write Write Ack Ack Read write Ack value DB Availability group
  110. 110. surface area of mi always on the latest and greatest SQL engine version your code can be sql deployment model aware if necessary note: current limitation being removed later this year MI is always on latest and greatest SQL engine version documentation page Your code can be SQL deployment model aware if necessary Current limitations (will be removed later this year) AT TIME ZONE
  111. 111. db compatibility based certification microsoft database compatibility level protection easy to use tools to help you access migration Microsoft Database Compatibility Level Protection Overall process Contact Microsoft
  112. 112. app compatibility: what’s is missing? Features with a better alternative in Azure OMS Retired features Features considered post-GA
  113. 113. clr considerations Managed Instance cannot access file shares and Windows folders Only CREATE ASSEMBLY FROM BINARY is supported CREATE ASSEMBLY FROM FILE is not supported ALTER ASSEMBLY can’t reference files
  114. 114. sql server agent Built into Managed Instance Azure SQL Database requires using on-premises SQL Server Agent, Azure Automation, Elastic Jobs, or PowerShell Always running Services cannot be stopped or restarted like they can with on-premises Option to auto-restart SQL Server if it stops unexpectedly is disabled Option to auto-restart SQL Server Agent if it stops unexpectedly is disabled Forwarding SQL Server events is disabled On-premises SQL Server Agent allows for forwarding events to another server but this is currently not an option for a Managed Instance Connection Alias local host server is predefined for a Managed Instance, whereas on-premises SQL Server Agent allows that to be configured if needed Creating jobs Creating jobs is as simple and easy as on-premises Jobs can be created using the UI or T-SQL Alert System Functions the same as on-premises for sending email alerts SQLCMD Cannot be called within a SQL Server Agent job Can be used to connect to a Managed Instance
  115. 115. service broker within instances Service broker is on by default for all user databases Cross-instance service broker is not supported CREATE ROUTE does not work with ADDRESS other than LOCAL ALTER ROUTE does not work with ADDRESS other than LOCAL
  116. 116. Fully supported in Managed Instance Functions the same as on-premises to set up and use Azure SQL Database does not have Database Mail support database mail
  117. 117. replication support Supported Snapshot replication. Same functionality as on-premises Transactional replication Unsupported Peer-to-peer replication Merge replication Heterogeneous replication Oracle publisher For comparison, Azure SQL Database only supports being a transactional replication push subscriber Some restrictions when used with a Managed Instance Updatable subscriptions are not permitted Publisher and distributor must be in the same location If publisher and distributor are in a Managed Instance, Azure file share must be used to store data and schema from the publication Connections to the Distributor must use SQL authentication Additions to support Managed Instance New fields have been added in replication-related tables in msdb job_login, job_password, storage_connection_string SSMS replication wizard supports using a Managed Instance
  118. 118. user db file layout considerations Data file default initial size is 16MB with 16MB auto growth These can and should be adjusted for your workload File size limit is 8TB in General Purpose Log file default initial size is 8MB with 16MB auto growth This can and should be adjusted for your workload Additional data files/filegroups can be added Only using an ALTER DATABASE statement and the FILENAME clause is not permitted Paths and File Names are chosen for you Different from Azure SQL Database where additional files are not allowed Multiple log files are not supported (and should not be needed) A backup with multiple files/filegroups can be restored Each user database has a FILESTREAM filegroup for In-Memory OLTP checkpoint files Multiple log files are not supported (and should not be needed)
  119. 119. tempdb data file considerations Tempdb Tuning Options Additional tempdb data files can be created if needed Well-known tempdb tuning ‘fixes’ are on by default Tempdb Resizing
  120. 120. backup/restore
  121. 121. backups are automatic Database backup schedule is the same as Azure SQL Database COPY_ONLY, URL-based backups can be used to perform manual full database backups Backup retention is 7 days by default
  122. 122. restore considerations Point-in-time restores are possible and must be performed manually using the Azure Portal Restoring automated backups from within SSMS is not allowed You can only restore using the Azure Portal COPY_ONLY, URL-based full backups can be restored using SSMS to a Managed Instance only Cannot be restored to on-premises as Managed Instance uses a higher build than on-premises instances Databases with multiple log files cannot be restored Secondary log files must be removed prior to backing up and restoring to a Managed Instance Can restore backups in a specific DB Compatibility Supports up to SQL 2005
  123. 123. Azure Database Migration Service (Azure DMS) Data Migration Assistant (DMA) SQL Database Managed Instance Azure Hybrid Benefit for SQL Server Data Migration Assistant (DMA) • Enables upgrades to SQL Server and Azure SQL Database Azure Hybrid Benefit for SQL Server • Maximizes current on-premises license investments to facilitate migration. Azure SQL Database Managed Instance • Facilitates lift and shift migration from on-premises SQL Server to PaaS.
  124. 124. provides guidance, tools, and partners in context of your migration scenario Azure migration center
  125. 125. • fully managed Azure service platform for seamless and frictionless data migration at scale • database migrations with minimal downtime • migrate sql server & 3rd party databases to azure sql database • built for scale and reliability expedite migration with azure database migration service azure db migration Service SQL DB
  126. 126. azure database migration service On-premises network Subnet used for DMS On-PremiseCustomer SubscriptionMicrosoft Subscription Azure Storage Account Service Bus Queue DMS Resource Provider CLI commands PowerShell Virtual Machine Azure Portal My M y
  127. 127. azure database migration journey assess migrate optimize
  128. 128. migrating databases using azure database migration services Seamless, end to end solution | Near-zero downtime | Resilient | Migrate at-scale from multiple sources On-premises SQL Server 2005 - 2017 Assessment SSMA & DMA Azure Database Migration Service
  129. 129. Assessment + Optimize Target Recommendation Which Azure target is best for me - SQL DB, MI or VM SKU Recommendation Which servicepricing tier should I pick within the target Detailed Assessment Is my database ready to migrate? Performance Validation Validate how my DB will perform in Azure Create config file with databases you want to assess Run target recommendation using DMA command line PowerBI report that gives an overview Have more databases to assess? Yes Consolidate multiple assessments into a single report No Collect DMV statistics from machine hosting your databases Run SKU recommendation using DMA command line Recommendations in JSON, CSV and HTML formats Provision databases in Azure with the recommendations Multiple databases at a time Create config file with databases you want to assess 1 machine at a time (no DB limit) Multiple databases at a time Run detailed assessment using DMA UX or command line 1 database at a time Get compatibility and feature parity issues Fix and rerun assessments to confirm readiness Capture workload from your source databases Replay capture on current and planned targets Generate analysis report Analysis report has query correctness and performance (source vs target)
  130. 130. Target Recommendation SKU Recommendation Create database config file Run target recommendation using DMA CLI Run SKU recommendation using DMA CLI Collect DMV statistics from your database Power BI report
  131. 131. database experimentation assistant
  132. 132. what to remember when migrate Latest driver versions provides best connectivity experience MI is PaaS - build connectivity resilience into your code to protect from transient faults Well-known SSMS tools supported – install latest version MI is always on latest and greatest SQL engine version but supports db compat levels from 100 and above Code can be MI aware, if necessary: SERVERPROPERTY (‘EngineEdition’) = 8 Current limitations (will be removed later this year) Time is UTC . Use AT TIME ZONE to add local time zone experience Instance collation is fixed (affects tempdb and system databases) You can use read-only replicas to load balance read-only queries Local and Geo-DR
  133. 133. azure database migration service Self-guided migration process Provides comprehensive assessments for pre-migration Allows for migration at scale from multiple sources to the target database with minimal downtime Requires an active Azure subscription and site-to-site connectivity Roadmap to support large amount of sources Convergence of DMA functionality and DMS Integration with Azure Databox
  134. 134. data migration assistant assess on-premises SQL Server instance(s) for migrating to Azure SQL database(s) discover issues that can affect an upgrade migrate an on-premises SQL Server instance to a modern SQL Server instance
  135. 135. Demo Data Migration Assistant
  136. 136. sql -> sql database mi online migration workflow Subnet used for MI Subnet used for DMS On-Premise Cloud network SQL DB Managed Instance Azure Blob Storage SMB Network Share provide Tail-Log backup, initiate cutover in DMS and change the application connection strings
  137. 137. resources Azure Database Migration Service https://azure.microsoft.com/services/database-migration/ Preview signup: https://aka.ms/dms-preview Feedback alias: dmsfeedback@microsoft.com Channel 9 Video: Oracle migrations; Azure SQL Database migrations Video: Online migrations to Azure SQL Database using Azure DMS Video: Migrate MySQL applications to Azure with minimal downtime using the Azure Database Migration Service Migration Guide datamigration.microsoft.com SQL Server Migration Assistant: https://docs.microsoft.com/sql/ssma/sql-server-migration-assistant Database Migration Assistant: https://blogs.msdn.microsoft.com/datamigration/dma/ Database Experimentation Assistant: Download Find a partner: http://migration/Pages/SearchPartners.aspx Data Migration Team Blog: blogs.msdn.microsoft.com/datamigration/
  138. 138. your data migration path https://azure.microsoft.com/en-us/resources/choosing-your-database-migration-path-to-azure/en-us/ five steps to migrate 1. initiate and discover 2. access 3. plan 4. transform and optimize 5. migrate, validate and remediate
  139. 139. migration to azure
  140. 140. business intelligence services not installed side-by-side with Managed Instance migrate your ssis packages to new ssis on azure data factory (PaaS service) migrate your olap models to azure analysis services … or run these services in Azure virtual machines for ssrs: run in a virtual machine, or switch to power bi SSIS/SSAS SSIS / SSAS PaaS Power BI SQL Server Reporting Services SQL
  141. 141. Lift your SQL server integration services (SSIS) packages to azure Azure Data Factory SSIS ETL Data sources SQL Database Managed Instance SQL Server SQL Data sources SSIS ETL SSIS Integration Runtime VNET Cloud On-premises SQL
  142. 142. inefficient data management and performance limitations Productivity and performance challenges Very large databases face slow queries and downtime Dedicated resources needed for database administration Inability to effectively scale for multitenancy
  143. 143. breakthrough productivity and performance with large workloads Azure SQL Database solutions Support for 100TB+ databases with Hyperscale Faster app development and reduced database ops with a fully managed service Multi-tenant apps easily built in a preferred environment
  144. 144. what is hyperscale? hyperscale is an all new storage layer under the database hyperscale is architected for the cloud from the ground up hyperscale is fully compatible with azure sql database hyperscale is no limits hyperscale is vldb size without the vldb headaches support for 100TB+
  145. 145. hyperscale your database SQL Database Hyperscale Storage Primary Compute Secondary Compute Secondary Compute Secondary Compute Log service Compute 1 TB data file 1 TB data file 1 TB data file Legend:
  146. 146. step 3: now that you’ve de-coupled the database using mi, let’s see if you can move to paas much smaller sql database app 1 cosmosdb app 2 much smaller sql database app 3 remember this is the goal how do we get there?
  147. 147. the main problems with legacy applications • too many dependencies • bleeding layers • code that is unreadable • code that no one wants to change • repeatedly introducing the same bugs
  148. 148. Michael feathers https://www.amazon.com/Working- Effectively-Legacy-Michael- Feathers/dp/0131177052
  149. 149. stinting / seam testing a seam is a place where you can alter behavior in your program without editing in that place. we split the method into two methods without changing functionality this allows us to cover one or both of the methods in test once covered in tests, we can begin to change functionality
  150. 150. testing strategies • imagine we’re pulling a puzzle apart, but leaving the pieces where they are • soon we have separate pieces, but now we can look at them individually • move the individual piece to a paas offering • can it be deployed separately
  151. 151. only convert that subset to paas good candidates are: • archival • authentication • notification services
  152. 152. concentrated re-write only rewrite a small target that you can rip out of the legacy app
  153. 153. let the developers choose their data store guide them towards the products you know develop your expertise
  154. 154. remember our main rules only the application touches the database all integration is done through an event hub the bi teams can consume the hub too…..they don’t need to touch the db
  155. 155. what’s a good boundary? think in terms of deployment what features are likely to be tested and deployed together keep the testing surface area small-enough but not too small deploy the database with the code version both of them think in terms of remediation. how are conflicts resolved?
  156. 156. the death of the dba, you are now an azure data architect • roles and responsibilities • day to day life • rise of your career
  157. 157. things you no longer need to do • sweat about disk space • worry about patch level and security patches • create an upgrade plan • care about what the ops guys are doing • wonder if you’re going to get priority on the san • purchase hardware • argue about capital expenses
  158. 158. things you absolutely have to do • follow every blog and twitter account for any product you’re responsible for • Life moves pretty fast. If you don't stop and look around once in a while, you could miss it. • learn as much as you can about a wide variety of products • constantly interact with the developers. it’s your day to watch them. • de-couple everything from your schema so you can normalize or de-normalize as necessary • consider partitioning • learn scripting because the number of database… • well, they’re about to explode
  159. 159. day to day life • create scripts for all manual activities • learn terraforming • learn scripting and automation • create framework to easily deploy dev/test environments • learn docker/Kubernetes • learn python probably • watch your data model • watch your performance • predict scale-up/scale-down • watch your bill • watch those developers
  160. 160. rise of your career • as you embrace the cloud, you will find lots of future opportunities • no longer is knowledge about only one database product acceptable • learn no-sql • learn what mpp and analytics teams are doing • learn about data pipelining • double your income
  161. 161. random slides: let’s talk marketing material in case you need to make a use case
  162. 162. constantly expanding But data is more complex than ever
  163. 163. Getting ahead means getting to the cloud 80% of organizations 19.6% faster
  164. 164. 1Pie Chart *IDC Worldwide DB Forecast Dec 2016 IDC Worldwide Database Server ForecastModernizing opportunity with SQL Server 37% of all units run Microsoft, >50% of units run 2008/R2 or older2 Non Relational Open source PaaS & IaaS Microsoft Oracle IBM SAP New SQL Other MySQL No SQL Priorities Eliminate time spent managing “long tail” of applications – lift and shift to managed cloud Free up limited IT resources to drive transformation Migrate business critical apps to cloud – extend and innovate IT optimization is key to digital transformation
  165. 165. Migrate to the cloud with Azure SQL Database Seamless and compatible Competitive TCO Built-in intelligence Breakthrough productivity & performance Industry-leading security The best and most economical cloud destination
  166. 166. near 100% uptime with dynamic scalability 30x faster transactions with in-memory OLTP 100x performance gains with in-memory analytics 100TB + Auto scaling up to 100TB with Hyperscale
  167. 167. Threat detection No app changes Always Encrypted Industry-leading security Client side Server side TCE-enabled ADO .NET library Master key Encrypted columnar key Apps Encrypted query Real-time data masking, partial masking. CreditCardNo XXXX-XXXX-XXXX-5796 XXXX-XXXX-XXXX-1978 SQL Database Table.CreditCardNo 4465-6571-7868- 5796 4468-7746-3848- 1978 4484-5434-6858- 6550 Alert Customer 1 Customer 2 Customer 3
  168. 168. Intelligent Performance learns unique database patterns and automatically tunes for improved performance Adaptive query processing Accelerate parallel queries and improve scaling of frequent queries with Intelligent Query Processing Tuning algorithm s SQL Database Database advisor Developer no admin skills DBA of many databases “My app gets auto-tuned” “I can do more in less time”
  169. 169. Seamless and compatible Compatibility challenges Migrating to the cloud is complex and time- consuming Maintaining security isolation from other tenants in the cloud Eliminating the costs of re-architecting apps for the cloud Azure SQL Database solutions
  170. 170. Azure SQL Database resource types Azure SQL Database Database-scoped deployment option with predictable workload performance Shared resource model optimized for greater efficiency of multi- tenant applications Best for apps that require resource guarantee at database level Best for SaaS apps with multiple databases that can share resources at database level, achieving better cost efficiency Best for modernization at scale with low friction and effort Elastic PoolSingle Managed Instance Instance-scoped deployment option with high compatibility with SQL Server and full PaaS benefits
  171. 171. Reduce capital and operational costs with a fully-managed service and achieve up to 406% ROI1 Financially-backed 99.99% availability SLA2 Promote business continuity with built-in capabilities Maximize your on-premises investments with Azure Hybrid Benefit for SQL Server and reserved capacity pricing Right-size on-premises workload requirements for the cloud with independent control of storage and compute Source: The Total Economic Impact™ of Microsoft Azure SQL Database Managed Instance, Forrester Consulting, September 2018 $(500,000) $- $500,000 $1,000,000 $1,500,000 $2,000,000 $2,500,000 $3,000,000 INITIAL YEAR 1 YEAR 2 YEAR 3 Cashflows Financial Analysis (risk-adjusted) Total costs Total benefits Cumulative net benefits Up to 212% ROI with Azure SQL Database
  172. 172. Customers can focus on their business We take care of your database chores Your work so far How SQL Database helps Hardware purchasing and management Built-in scale on-demand Protect data with backups (with health checks and retention) Built-in point-in-time restore High availability implementation Built-in 99.99% SLA and auto-failover Disaster recovery implementation Built-in geo-redundancy and geo-replication Ensure compliance with standards on your own Built-in easy to use features Secure your data from malicious users and mistakes Built-in easy to use features Role out updates and upgrades Built-in updates and upgrades Monitor, troubleshoot, and manage at scale Built-in easy to use features Tune and maintain for predictable performance Built-in easy to use features
  173. 173. Thank You Speaker name @yourhandle youremail@email.com

×