8. • Intelligent Query Processing
Gain Performance with no
application changes
• Data Classification and Auditing
Need to classify and audit
your data?
• Accelerated Database Recovery
Long running transactions
affect your availability
• Data Virtualization
Want external access with
no data movement?
• SQL Server Replication in one command
Is SQL Server on Linux and
Containers compatible?
• Additional capabilities, Migration Tools, and Database
Compatibility
Additional capabilities and
Migration
https://aka.ms/bobwardms
• https://github.com/microsoft/sqlworkshops/tree/master/SQLGroundToCloud/slides
(Note that each of these can be independent units)
16. ADD SENSITIVITY CLASSIFICATION TO
dbo.sales.price, dbo.sales.discount
WITH ( LABEL='Highly Confidential',
INFORMATION_TYPE='Financial' )
data_sensitivity_information
Who, what, and when accessed my
classified data?
21. SQL Server
T-SQL
Analytics Apps
ODBC NoSQL Relational databases Big data
PolyBase external tables
Distributed compute engine integrated with SQL Server
Query data where it lives using T-SQL
Distributed, scalable query performance
Manual/deploy with SQL Server
Auto deploy/optimize with Big Data Clusters
Intelligence over all data
“It’s all about
Data Virtualization”
23. Managed SQL Server, Spark,
and data lake
Store high volume data in a data lake and access
it easily using either SQL or Spark
Management services, admin portal, and
integrated security make it all easy to manage
SQL
Server
Data virtualization
Combine data from many sources without
moving or replicating it
Scale out compute and caching to boost
performance
T-SQL
Analytics Apps
Open
database
connectivity
NoSQL Relational
databases
HDFS
Complete AI platform
Easily feed integrated data from many sources to
your model training
Ingest and prep data and then train, store, and
operationalize your models all in one system
SQL Server External Tables
Compute pools and data pools
Spark
Scalable, shared storage (HDFS)
External
data sources
Admin portal and management services
Integrated AD-based security
SQL Server
ML Services
Spark &
Spark ML
HDFS
REST API containers
for models
24. SQL Platform Abstraction Layer
(SQLPAL)
RDBMS IS AS RS
Windows Linux
Windows Host Ext. Linux Host Extension
SQL Platform Abstraction Layer
(SQLPAL)
Host extension mapping to OS system calls
(IO, Memory, CPU scheduling)
Win32-like APIs
SQL OS API
SQL OS v2
All other systems
System resource &
latency sensitive code paths
Choice across OS and containers
25.
26. Container Container Container
Portable
Run anywhere Docker is supported
Lightweight
Reduced disk, CPU, and memory footprint
Consistent
Consistent image of SQL Server, scripts, and tools
Efficient
Faster deployment, reduced patching, and
less downtime
Container configuration
Database
Persisted storage
Switch for
simple
upgrades
31. String or binary data would be truncated
String or binary data would be truncated
in table '%.*ls', column '%.*ls’.
Truncated value: '%.*ls'
SELECT page_info.*
FROM sys.dm_exec_requests AS d
CROSS APPLY
sys.fn_PageResCracker(d.page_resource) AS r
CROSS APPLY sys.dm_db_page_info(r.db_id,
r.file_id, r.page_id,'DETAILED')
AS page_info;
33. Reduce
upgrade risks
Unified
application
certification
Upgrade to latest
SQL Database
Engine version
Upgrade your SQL Server Database Engine or
move instances to the cloud with no code changes
Applications tested and certified on a given SQL
Server version are also implicitly tested and
certified on that SQL Server version native
database compatibility level
Separate application and platform layer upgrade
cycles for less disruption
Microsoft fully supports Compatibility Certification
Compatibility Certification benefits
Upgrade to the latest SQL Server Database Engine
without changing your critical applications
Frictionless migration
with no code changes
Certify once, run on-premises and in the cloud with Compatibility
Certification
Upgrade & modernize your SQL Server database on-premises, in the cloud and on the edge with Compatibility
Certification that eliminates risks of application compatibility
34. http://aka.ms/bobwardms
http://aka.ms/bobsqldemos
http://aka.ms/sqllinuxbook
Use our free training at https://aka.ms/sqlworkshops
Learn from videos and demos at https://aka.ms/sqlchannel
Download and try SQL Server 2019 at http://aka.ms/ss19
Watch the video Modernizing SQL Server at
https://www.youtube.com/watch?v=5RPkuQHcxxsto to plan your migration
Read the what’s new for SQL 2019 documentation
Sign-up for the EAP program at https://aka.ms/eapsignup
37. Data Growth
Computing and Storage advances impact data collection abilities
Computing and Storage technologies allow greater data collection points
They also allow longer historical data storage, and as time goes on become
part of that storage lineage
Large Retailers are a classic example of data proliferation and leverage
38. Use-Cases
Every Industry classification benefits
from Big Data, Retail and Finance
leads the way
Industry Sector Primary Use-Cases
Retail Demand prediction
In-store analytics
Supply chain optimization
Customer retention
Cost/Revenue analytics
HR analytics
Inventory control
Finance Cyberattack Prevention
Fraud detection
Customer segmentation
Market analysis
Risk analysis
Blockchain
Customer retention
Healthcare Fiscal control analytics
Disease Prevention prediction and classification
Clinical Trials optimization
Patient load analysis
Episode analytics
Public Sector Revenue prediction
Education effectiveness analysis
Transportation analysis and prediction
Energy demand and supply prediction and control
Defense readiness predictions and threat analysis
Manufacturing Predictive Maintenance (PdM)
Anomaly Detection
Pattern analysis
Agriculture Food Safety analysis
Crop forecasting
Market forecasting
Pipeline Optimization
41. Physical Computer
Virtualization
Hardware Abstraction
Building on hardware, you can
create a complete “PC” on top of
a Hypervisor layer, which
abstracts out the hardware. You
still own the Operating System
and up
This allows for scale by ring-
fencing OS-level dependencies Operating System
Hypervisor
Virtual Hard
Disk
Python
Binaries
Code
Virtual Hard
Disk
Python
Binaries
Code
Virtual Hard
Disk
Python
Binaries
Code
42. Containers
Abstracting the OS, Allowing complete portability
Containers go one level further
than the Hypervisor, and
focusing on binaries and
applications
Storage and networking are a
consideration
Scale is achieved through
multiple containers
Physical Computer
Operating System
Container Runtime
Container Container
Binaries
Code
Binaries
Code
Container
Binaries
Code
Python Python
47. Control
Compute Pool
HDFS
Storage Pool
SQL Data Pool
Cluster
Orchestration
Master
SQL Server
Master
Controller
(Shared Services)
SQL Server Spark
SQL Server
SQL Server
App Pool
Job (SSIS)
(Web Apps)
ML Server
(CTP 3.2)
BDC
HDFS
SQL Server
48. kubectl
Control Watchdog
Controller
Tools and Process Flow
Cluster Orchestration
Master
azdata
Azure Data
Studio
Config Store
Grafana Interface
Kibana Interface
Deployment, Configuration, Upgrade, HA Monitoring and Metrics
Management Proxy
Kibana
Operator
Elastic Search
Grafana
InfluxDB
Telegraf
CollectD
Telegraf
CollectD
Fluentbit
Fluentbit
49. AG
Cluster Orchestration
Master
Controller
Operator
Replication Controller
SQL Server
SQL Server SQL Server
Listener
(Readable Secondary)
Listener
(Primary)
Using Cluster HA and SQL Server Availability Groups
HA Supervisor
Primary
HA Supervisor
Secondary
HA Supervisor
Secondary
Cluster HA SQL Server HA
(Becomes Primary)
52. Takeaways
• SQL Server 2019 Big Data cluster includes SQL Server together with the HDFS and Spark Compute
engine as one package for big data processing, Machine Learning and AI
• Spark is a distributed compute engine that provides a unified framework for E2E big data processing
pipeline including Machine learning and AI
• You can use SQL Server 2019 to create a secure, hybrid, machine learning architecture starting with
data preparation, training a machine learning model, operationalizing your Model and using it for
scoring
• Go Do > Practice with installing, configuring, and operating SQL Server 2019
• Go Do > Download this deck and practice a demo on Big Data Clusters on SQL Server
• Go Do > Follow a complete workshop
58. SQL virtual machines Managed instances Databases
• SQL Server surface area
(vast majority)
• Native virtual network
support
• Fully managed service
• SQL Server and OS
server access
• Expansive SQL And OS
version support
• Automated
manageability features
for SQL Server
• Hyperscale storage (up
to 100TB)
• Serverless compute
• Fully managed service
• Resource sharing
between multiple
databases to price
optimize
• Simplified performance
management for
multiple databases
• Fully managed service
59.
60. Provisioned General Purpose Hyperscale Business Critical
VM NA NA NA
Managed Instance
Single Database
Elastic Pool
Serverless General Purpose Hyperscale Business Critical
VM NA NA NA
Managed Instance
Single Database
Elastic Pool
65. What is Resource Provider?
Resource Provider brings the functionality of Azure Marketplace images
to SQL Server instances self-installed on Azure Virtual Machines
Key customer benefits
66. Key benefits of Resource Provider (RP) on self-installations
Azure VMs are now discoverable
on the new Azure SQL blade
in Azure Marketplace
Self-installed VMs registered
with RP now can access
automation features in
Azure Marketplace images
Self-installed VMs with RP
can be easily converted
to PAYG images
Self-installed VMs with RP
automatically indicate usage
of Azure Hybrid Benefit
Learn more aka.ms/sqlvm_rp and aka.ms/sqlvm_rp_documentation
67. How to take advantage of Resource Provider
Learn more aka.ms/sqlvm_rp and aka.ms/sqlvm_rp_documentation
1. Register the resource provider to your subscription (one-time only) via the Azure Portal or Azure CLI below:
# Register the new SQL resource provider to your subscription
az provider register --namespace Microsoft.SqlVirtualMachine
2. Once subscription registration is complete, register your VM with the SQL VM Resource Provider:
# Register your existing SQL Server VM with the new resource provider
az sql vm create -n <VMName> -g <ResourceGroupName> -l <VMLocation>
Ways to get Resource Provider*
Recommended approach
71. Products and Services Organization Size Industry Country
The Allscripts prescription
for agility: lift and shift to
the cloud
Allscripts is a leading healthcare software manufacturer, serving
physician practices, hospitals, health plans, and Big Pharma. To
transform its applications frequently and host them securely and
reliably, Allscripts has started to use Microsoft Azure. In just three
weeks, the company lifted and shifted dozens of acquired
applications running on 1,000 virtual machines to Azure with
Azure Site Recovery. Allscripts also develops software in Azure
and realizes dramatic time-to-market gains and savings by using
the Azure platform as a service offering.
United States
Partner
professional
services
10,000 employees
Microsoft Azure
Azure Site Recovery
Azure SQL Database
Azure Virtual Machines
Azure Cosmos DB
72. Challenge
Had multiple mainframe applications for different
types of data, and needed to consolidate them into a
single system so that different parts of the
organization can easily get a holistic view of data
from different sources
Impact
Moved the entire database and the Discovery Hub analytics
implementation to Azure SQL Database Managed Instance in
less than two weeks, and since then we’ve seen a 49 percent
cost reduction and 25 to 30 percent performance gains.
“We determined Azure SQL Database
Managed Instance was the best choice
for us in terms of scalability, cost, and
performance.… We’ve seen a 49
percent cost reduction and 25 to 30
percent performance gains..”
Nipun Sharma: Analytics Architect, Business
Technology and Systems
Komatsu Australia
73. Products and Services Organization Size Industry Country Partner
AccuWeather weathers the
storm with Azure
The company first moved its API business to Azure. AccuWeather sells a host of weather-related APIs that
news outlets and other companies plug into their enterprise resource planning, customer relationship
management, and other business applications. The company receives more than 40 billion API calls a day for
weather information, and Azure provides a more scalable, cost-effective way to manage this service.
Next the company moved its big data storage and processing, using services such as Azure Blob
storage, Azure Data Factory, and Azure SQL Database.
When AccuWeather set out to build an automated, scalable weather prediction service for business
customers, it turned to Azure as well. To create highly accurate forecasts, Radich’s team needed
sophisticated machine learning tools that it could customize using R and Python code.
“With Azure, we have access to sophisticated data science services in a real-time, on-demand basis, which helps us innovate faster.”
- Chris Patti: Chief Technology Officer, AccuWeather
USA
Professional
Services
Medium (50 -
999 employees)
Azure API Management
Azure App Service
Azure Data Factory
Azure Machine Learning
Azure SQL Database
Azure Storage
74. Challenge
Flex Time by Paychex handles 13 million timecards
and 3 million distinct punches on any given day for
over 650,000 customers. With an increasing volume
of customers, however, managing that environment
became very cumbersome. Guaranteeing uptime
meant constant investments in new hardware, which
was both time-consuming and capital-intensive.
Impact
Paychex engineers are freed from managing
infrastructure, licensing, and database spin up and
spin down. Built-in intelligent features like automatic
tuning optimize database performance, leaving them
to focus on what they do best: understand how
clients are accessing the app and provide value in
their continuous delivery pipeline.
“Azure SQL Database allows us to
execute upgrades and releases without
our customers ever knowing it, which is
critical for an application that runs
24/7/365.”
Dave Wilson: Sr. Director of IT Infrastructure and
Architecture
75.
76. On-demand flexible scale
Operate at the true rhythm of
your business
Fully managed & intelligent
Focus on your applications, not
your infrastructure
Cost-effective
Pay for performance. Period.
Adapts compute resources to the
workload without sacrificing performance
Automatically pauses and resumes
Fully-managed and intelligent
database service
Built-in 99.99% availability
Pay only for compute resources you
consume, on a per-second basis
Further optimize costs with configurable
compute thresholds
Best for unpredictable and intermittent
workloads on single databases, such as:
Dev/test E-commerce
Line of Business
77. Workloads with unpredictable and intermittent
usage patterns or performance requirements
Workloads where the requirements are
unknown and you can delegate compute sizing
to the service
Common scenarios
Scale up or down to meet workload
requirements, instead of pre-provisioning
Bill on a per-second basis
Optimize compute provisioning and billing for your workload
Serverless databases…
Workloads with regular and substantial
compute utilization
Multiple databases with bursty usage
patterns that can be consolidated into a
single server and use elastic pools for
better price optimization
Common scenarios
Provision compute resources upfront
Bill on an hourly basis
Databases with provisioned compute…
78.
79. Azure Storage
~2ms for all data access
HS
Azure Storage
RBPEX
Page Servers
RBPEX
<0.5ms for hot data access
~2ms otherwise
Proportional
to compute
size
Full coverage
BC
Files
<0.5ms for all data access
Attached SSD
Remote Storage
80. Resources
Storage Premium remote storage (5-10ms latency)
Single database: 5 GB – 4 TB
Managed Instance: 32 GB - 8 TB
Compute
(vCores)
Gen4: 1 to 24 vCore
Gen5: 1 to 80 vCore
I/O Single database: 500 IOPS per vCore with 7000
maximum IOPS
Managed Instance: Depends on size of file
HA 1 replica, no read-scale
In-Memory Not supported
81. Resources
Storage Local SSD storage (1-2 ms latency)
Single database: 5 GB – 4 TB
Managed Instance: 32 GB - 4 TB
Compute
(vCores)
Gen4: 1 to 24 vCore
Gen5: 1 to 80 vCore
I/O 5000 IOPS per core with 200,000 maximum IOPS
HA 3 replicas + 1 read-scale replica, zone-redundant
HA
In-Memory Supported
82. Resources
Storage Local SSD storage for local buffer pool
cache/data
Azure remote storage as final long-term data
store
Supports up to 100TB storage
Compute
(vCores)
Gen4: 1 to 24 vCore
Gen5: 1 to 80 vCore
I/O TBD
HA 2 replicas (up to 5 at GA)
In-Memory Not supported
83. Service tier General purpose Business critical Hyperscale
Best for Most budget-oriented workloads
Critical business applications with high
IO requirements.
VLDB OLTP and HTAP workloads
with highly scalable storage and
read-scale requirements
Deployment
option
Single /
Elastic Pools
Managed
Instance
Single /
Elastic Pools
Managed
Instance
Single
Compute tiers
Gen4: 1 to 24 vCore
Gen5: 2 to 80 vCore
Gen4: 4 to 24 vCore
Gen5: 4 to 80 vCore
Gen4: 1 to 24 vCore
Gen5: 2 to 80 vCore
Gen4: 4 to 24 vCore
Gen5: 4 to 80 vCore
Gen4: 1 to 24 vCore
Gen5: 2 to 80 vCore
Storage
Premium remote Local SSD Local SSD Cache
32GB – 8TB per
instance
32GB – 8TB per
instance
32GB – 4TB per
instance
32GB – 8TB per
instance
Scale from 5GB to 100TB of storage in
1GB increments
In-Memory Not supported Supported Not supported
Read-write IO ~2ms for all data access <0.5ms for all data access
<0.5ms for hot data access
~2ms otherwise
Availability 2 read replicas
3 replicas, 1 read-scale replica, zone-
redundant HA
Primary read/write replica + up to 4
read replicas
Backups RA-GRS, 7-35 days (7 days by default) RA-GRS, 7-35 days (7 days by default)
LRS, ZRS, RA-GRS, 7-35 days (7 days
by default)
https://azure.microsoft.com/en-us/pricing/details/sql-database/
84.
85. In SQL MI you control the instance. In SQL DB you do not.
As a result a number of features are available in MI that are not in DB – for example:
Because you control the instance, a number of features are not in MI that are in DB - for example:
For a detailed comparison see https://docs.microsoft.com/en-us/azure/sql-database/sql-database-features
Only a detailed analysis of requirements can tell you which is the best fit:
𝑆𝑄𝐿𝐷𝐵𝑀𝐼 ≠ 𝑆𝑄𝐿𝐷𝐵 + +
Instance
88. Performance
VLDB operations without
VLDB headaches
Cloud native
Architected for cloud
SQL
Seamless compatibility
Fully compatible with Azure
SQL Database
No limits
Scale compute and storage
Storage
Scalable new storage
architecture
Large database
Support for 100TB+
91. Desire to build more intelligent applications with AI and Machine Learning
What, when, and how can I migrate to the cloud?
Want to leverage new security and performance features in Azure they’ve heard
about
Need Service Broker, do not want to rearchitect the solution to use an alternative
Applications need better availability and less downtime
92. Customer concerns
Looking for comparison of SQL database hosting options in Azure.
What tools can help a evaluate databases against target platforms and
identify potential issues?
Can we have network isolation and secure channel access from on-
premises?
94. Service tier General purpose Business critical Hyperscale
Best for Most budget-oriented workloads
Critical business applications with high
IO requirements.
VLDB OLTP and HTAP workloads
with highly scalable storage and
read-scale requirements
Deployment
option
Single /
Elastic Pools
Managed
Instance
Single /
Elastic Pools
Managed
Instance
Single
Compute tiers
Gen4: 1 to 24 vCore
Gen5: 2 to 80 vCore
Gen4: 4 to 24 vCore
Gen5: 4 to 80 vCore
Gen4: 1 to 24 vCore
Gen5: 2 to 80 vCore
Gen4: 4 to 24 vCore
Gen5: 4 to 80 vCore
Gen4: 1 to 24 vCore
Gen5: 2 to 80 vCore
Storage
Premium remote Local SSD Local SSD Cache
32GB – 8TB per
instance
32GB – 8TB per
instance
32GB – 4TB per
instance
32GB – 8TB per
instance
Scale from 5GB to 100TB of storage in
1GB increments
In-Memory Not supported Supported Not supported
Read-write IO ~2ms for all data access <0.5ms for all data access
<0.5ms for hot data access
~2ms otherwise
Availability 2 read replicas
3 replicas, 1 read-scale replica, zone-
redundant HA
Primary read/write replica + up to 4
read replicas
Backups RA-GRS, 7-35 days (7 days by default) RA-GRS, 7-35 days (7 days by default)
LRS, ZRS, RA-GRS, 7-35 days (7 days
by default)
https://azure.microsoft.com/en-us/pricing/details/sql-database/
95.
96. Take an inventory of on-premises licenses
to determine potential for conversion
Convert on-premises cores to vCores to
maximize value of investments
License trade-in values
General purpose or
Hyperscale vCore
4x General purpose or
Hyperscale vCores
Business critical
vCore
SQL Server cores
with SA license
vCores on Azure
SQL Database
SQL Server
Enterprise Edition
SQL Server
Standard Edition
98. Migrating to Azure SQL
Demo: Database discovery and assessment
Demo: Migrating to Azure SQL
Demo Advanced Data Security
99. • Azure Migrate
• Data Migration Assistant
Discover and assess
what and where to
migrate to Azure
• Azure Database Migration Services
Migrate to Azure SQL
Managed Instance with
little to no downtime
• SQL Data Discovery and Classification
• Vulnerability Assessment
Improve database
security with Advanced
Data Security
• Online secondary read replica
Obtain read-scale out
for read only queries
https://aka.ms/azuresqlslides
103. Azure Migration Program (AMP)
Available to all Azure customers, scaled through specialized migration partners
Learn more | Azure.com/AMP
Reduce cost and risk | Accelerate migration velocity | Deliver customer success
Offers and incentives
Best practice guidance Technical skill building
Infrastructure and
data foundations
Migration planning
and execution
Azure
trainers
FastTrack for
Azure engineers
Specialized
partners
Digital
resources
Migration
Tools
104. Desire to build more intelligent applications with AI and Machine Learning
What, when, and how can I migrate to the cloud?
Want to leverage new security and performance features in Azure they’ve heard
about
Need Service Broker, do not want to rearchitect the solution to use an alternative
Applications need better availability and less downtime
105. Tailspin Toys - Preferred migration solution
Azure Migrate
Data Migration Assistant
Azure Database Migration Service
Azure Site Recovery
Partner or Azure Migrate Program engagement
Azure Hybrid Benefit
Prepay for Reserved Capacity
108. Tailspin Toys - Preferred solution – Gaming
services
Azure Site Recovery to migrate VMs
Multi-region deployments to address latency
VM scale sets for scalability of VMs
Availability Zones for high-availability
Availability Zones for VMs
Auto-failover groups in SQL MI (in public preview)
Geo-restore and geo-replication
129. Use our free training at https://aka.ms/sqlworkshops
Learn from videos and demos at https://aka.ms/sqldbonyoutube
See and reuse useful Azure SQL slides at https://aka.ms/azuresqlslides
https://azure.com/migration/
131. 1. Discovery: The original statement of the problem from the customer
2. Envisioning: A "blue-sky" description of what success in the project would
look like. Often phrased as "I can..." statements
3. Architecture Design Session: An initial layout of the technology options and
choices for a preliminary solution
4. Proof-Of-Concept (POC): After the optimal solution technologies and
processes are selected, a POC is set up with a small representative example of
what a solution might look like, as much as possible. If available, a currently-
running solution in a parallel example can be used
5. Implementation: Implementing a phased-in rollout of the completed
solution based on findings from the previous phases
6. Handoff: A post-mortem on the project with a discussion of future
enhancements
Project Process
132.
133.
134.
135.
136.
137.
138. mediaacq
www.superstock.com/
www.gettyimages.com/creativeimages/royaltyfree
Avoid intellectual property theft: Do not use photos
or graphics copied from the web in presentations.
When in doubt, consult with CELA.
https://microsoft.sharepoint.com/teams/BrandCentral/
www.microsoft.com/en-us/news
http://news.xbox.com/media/
https://www.featureddevices.com/
https://microsoft.sharepoint.com/teams/BrandCentral/P
ages/Presentations.aspx