Expert summit SQL Server 2016

SQL Server 2016
New innovations
Lukasz Grala
Microsoft MVP Data Platform

Operational
Database
Management
Systems
Data
Warehouse
Database
Management
Systems
Business
Intelligence
and Analytics
Platforms
x86 Server
Virtualization
Cloud
Infrastructure
as a Service
Enterprise
Application
Platform as a
Service
Public Cloud
Storage
Leader in 2014 for Gartner Magic Quadrants
Microsoft platform leads the way on-premises and cloud

Operational Database Management System – October 2015

Performance Security Availability Scalability
Operational analytics
Insights on operational data;
Works with in-memory OLTP and
disk-based OLTP
In-memory OLTP
enhancements
Greater T-SQL surface area,
terabytes of memory supported,
and greater number of parallel
CPUs
Query data store
Monitor and optimize query plans
Native JSON
Expanded support for JSON data
Temporal database
support
Query data as points in time
Always encrypted
Sensitive data remains encrypted
at all times with ability to query
Row-level security
Apply fine-grained access control
to table rows
Dynamic data masking
Real-time obfuscation of data to
prevent unauthorized access
Other enhancements
Audit success/failure of database
operations
TDE support for storage of in-
memory OLTP tables
Enhanced auditing for OLTP with
ability to track history of record
changes
Enhanced AlwaysOn
Three synchronous replicas for
auto failover across domains
Round robin load balancing of
replicas
Automatic failover based on
database health
DTC for transactional integrity
across database instances with
AlwaysOn
Support for SSIS with AlwaysOn
Enhanced database
caching
Cache data with automatic,
multiple TempDB files per instance
in multi-core environments
Mission-critical performance

Operational Analytics
What is operational
analytics and what does it
mean to you?
Operational analytics with
disk-based tables
Operational analytics with
In-Memory OLTP
IIS Server
BI analysts

Traditional operational/analytics architecture
Key issues
Complex implementation
Requires two servers (capital
expenditures and operational
expenditures)
Data latency in analytics
More businesses demand;
requires real-time analytics
IIS Server
BI analysts

Minimizing data latency for analytics
Benefits
No data latency
No ETL
No separate data warehouse
Challenges
Analytics queries are resource intensive and
can cause blocking
Minimizing impact on operational workloads
Sub-optimal execution of analytics on
relational schema
IIS Server
BI analysts

Operational
analytics
disk-based tables

Operational
analytics for
in-memory tables
In-Memory OLTP table
Updateable
CCI
TailDRT
Hash index
Like Delta
rowgroup

Using Availability Groups instead of data warehouse
Key points
Mission Critical Operational Workloads
typically configured for High Availability
using AlwaysOn Availability Groups
You can offload analytics to readable
secondary replica
Secondary
Replica
Secondary
Replica
Secondary
Replica
Primary
Replica
Always on Availability Group

ALTER TABLE Sales.SalesOrderDetail
ALTER INDEX PK_SalesOrderID
REBUILD
WITH (BUCKET_COUNT=100000000)
T-SQL surface area: New
{LEFT|RIGHT} OUTER JOIN
Disjunction (OR, NOT)
UNION [ALL]
SELECT DISTINCT
Subqueries (EXISTS, IN, scalar)
ALTER support
Full schema change support: add/alter/drop
column/constraint
Add/drop index supported
Surface area improvements
Almost full T-SQL coverage including scaler user-defined
functions
Improved scaling
Increased size allowed for durable tables; more sockets
Other improvements
MARS support
Lightweight migration reports

Improved scaling
In-memory OLTP engine
has been enhanced to
scale linearly on servers
up to 4 sockets
Other enhancements include:

Data Source=MSSQL; Initial Catalog=AdventureWorks;
Integrated Security=SSPI;
MultipleActiveResultSets=True
Setup MARS connection for
memory optimized tables using
the
MultipleActiveResultsSets
=True in your connection string
Using multiple active result sets (MARS)

In SQL Server 2016 CTP2, the
storage for memory-optimized
tables will be encrypted as part
of enabling TDE on the database
Simply follow the same steps as
you would for a disk-based
database
Support for Transparent Data Encryption
Windows Operating System
Level Data Protection
SQL Server
Instance Level
User Database
Level Database Encryption Key
Service Master Key
DPAPI encrypts the Service Master Key
Master
Database Level Database Encryption Key
Service Master Key Encrypts the Database
master Key for the master Database
Database Master Key of the master Database
creates a certificate in the master database
The certificate encrypts the database
Encryption Key in the user database
The entire user database is secured by the
Datbase Encryption Key (DEK) of the user
database by using transparent database
encryption
Created at a time of SQL Server setup
Statement:
CREAT MASTER KEY…
Statement:
CREATE CERTIFICATE…
Statement:
CREATE DATABASE ENCRYPTION KEY…
Statement:
ALTER DATABSE… SET ENCRYPTION

New Transaction Performance Analysis Overview report
New report replaces the
need to use the
Management Data
Warehouse to analyze
which tables and stored
procedures are
candidates for in-
memory optimization

Query Store
Your flight data recorder
for your database

Have You Ever…?
…had your system down/slowed down and everyone waiting for you to
magically fix the problem ASAP?
…upgraded an application to the latest SQL Server version
and had an issue with a plan change slowing your application down?
…had a problem with your Azure SQL Database and been unable to
determine what was going wrong?

With Query Store…
I CAN get full history of query execution
I CAN quickly pinpoint the most expensive queries
I CAN get all queries that regressed
I CAN easily force better plan from history with a single line of T-SQL
I CAN safely do server restart or upgrade

Durability latency controlled by DB option
DATA_FLUSH_INTERNAL_SECONDS
Compile
Execute
Plan Store
Runtime
Stats
Query
Store
Schema
Query data store
Collects query texts (+ all relevant properties)
Stores all plan choices and performance
metrics
Works across restarts / upgrades / recompiles
Dramatically lowers the bar for perf.
Troubleshooting
New Views
Intuitive and easy plan forcing

Monitoring Performance By Using the Query Store
The query store feature
provides DBAs with
insight on query plan
choice and performance

Live query statistics
Collect actual metrics about query while running
View CPU/memory usage, execution time, query
progress, etc.
Enables rapid identification of potential
bottlenecks for troubleshooting query
performance issues.
Allows drill down to live operator level statistics:
Number of generated rows
Elapsed time
Operator progress
Live warnings, etc.

Java Script Object
Notation (JSON)

JSON became ubiquitous
Compact and simple data exchange format
The choice on the web
Recommended scenario
I CAN accept JSON, easily parse and store it as relational
I CAN export relational easily as JSON
I CAN correlate relational and non-relational

[
{
"Number":"SO43659",
"Date":"2011-05-31T00:00:00"
"AccountNumber":"AW29825",
"Price":59.99,
"Quantity":1
},
{
"Number":"SO43661",
"Date":"2011-06-01T00:00:00“
"AccountNumber":"AW73565“,
"Price":24.99,
"Quantity":3
}
]
Number Date Customer Price Quantity
SO43659 2011-05-31T00:00:00 AW29825 59.99 1
SO43661 2011-06-01T00:00:00 AW73565 24.99 3
SELECT * FROM myTable
FOR JSON AUTO
SELECT * FROM
OPENJSON(@json)
Data exchange with JSON

CREATE TABLE SalesOrderRecord (
Id int PRIMARY KEY IDENTITY,
OrderNumber NVARCHAR(25) NOT NULL,
OrderDate DATETIME NOT NULL,
JSalesOrderDetails NVARCHAR(4000)
CONSTRAINT SalesOrderDetails_IS_JSON
CHECK ( ISJSON(JSalesOrderDetails)>0 ),
Quantity AS
CAST(JSON_VALUE(JSalesOrderDetails, '$.Order.Qty') AS int)
)
GO
CREATE INDEX idxJson
ON SalesOrderRecord(Quantity)
INCLUDE (Price);
JSON and relational
JSON is plain text
ISJSON guarantees
consistency
Optimize further with
computed column and
INDEX

SELECT t.Id, t.OrderNumber, t.OrderDate,
JSON_VALUE(t.JSalesOrderDetails,
'$.Order.ShipDate')
AS ShipDate
FROM SalesOrderRecord AS t
WHERE ISJSON(t.JSalesOrderDetails) > 0
AND t.Price > 1000
Ability to load JSON text in tables
Extract values from JSON text
Index properties in JSON text
stored in columns
-- and more
Querying JSON

No new data type
If you need to store it raw, store it as NVARCHAR
What is new:
Easy export: FOR JSON
Easy import: OPENJSON
Easy handling: ISJSON, JSON_VALUE
How to handle JSON?

SELECT
OrderNumber AS 'Order.Number',
OrderDate AS 'Order.Date'
FROM SalesOrder
FOR JSON PATH
For JSON path
[
{
"Order":{
"Number":"SO43659",
"Date":"2011-05 31T00:00:00“
},
},
{
"Order":{
"Number":"SO43660",
"Date":"2011-06-01T00:00:00“
},
}
]
Query Result (JSON array)

SELECT SalesOrderNumber,
OrderDate,
UnitPrice,
OrderQty
FROM Sales.SalesOrderHeader H
INNER JOIN Sales.SalesOrderDetail D
ON H.SalesOrderID = D.SalesOrderID
FOR JSON AUTO
For JSON AUTO
[
{
"SalesOrderNumber":"SO43659",
"OrderDate":"2011-05-31T00:00:00",
"D":[
{"UnitPrice":24.99, "OrderQty":1
}
]
},
{
"SalesOrderNumber":"SO43659" ,
"D":[
{ "UnitPrice":34.40 },
{ "UnitPrice":134.24, "OrderQty":5
}
]
}
]
Query Result

CREATE TABLE OrderRecord (
Id int PRIMARY KEY IDENTITY,
OrderDetails NVARCHAR(25)
CONSTRAINT OrderDetails_IS_JSON
CHECK (ISJSON(OrderDetails)>1),
Quantity AS
CAST(JSON_VALUE(OrderDetails,'$.
Order.Qty') AS int)
)
JSON functions (CTP3)
CREATE INDEX idxJson
ON OrderRecord(Quantity);
ISJSON (json_text) JSON_VALUE (json_text,json_path)

OPENJSON
OPENJSON (@json, N'$.Orders.OrdersArray')
WITH (
Number varchar(200) N'$.Order.Number',
Date datetime N'$.Order.Date',
Customer varchar(200) N'$.AccountNumber',
Quantity int N'$.Item.Quantity'
)
{"Orders": { "OrdersArray":
[
{
"Order": {
"Number":"SO43659",
"Date":"2011-05-31T00:00:00“
},
"Item": {
"Price":2024.9940,
"Quantity":1
}
},
{
"Order":{
“Number":"SO43661",
"Date":"2011-06-01T00:00:00“
},
"Item": {
"Price":2024.9940,
"Quantity":3
}
}
]} }
Number Date Customer Quantity
SO43659 2011-05-31T00:00:00 AW29825 1
SO43661 2011-06-01T00:00:00 AW73565 3

SQL Server and Azure DocumentDB
Schema-free NoSQL
document store
Scalable transactional
processing for rapidly
changing apps
Premium relational
DB capable to
exchange data with
modern apps & services
Derives unified insights from
structured/unstructured data
JSON
JS
JSJSON

Key takeaways
Temporal: data audit
and time-based
analysis
Query Store:
performance
troubleshooting and
tuning
JSON: interop with
modern services and
applications
Available in the cloud
and on-prem
QUERY STORE
JSON

Real data sources are dynamic
Historical data may be critical to business success
Traditional databases fail to provide required insights
Workarounds are…
Complex, expensive, limited, inflexible, inefficient
SQL Server 2016 makes life easy
No change in programming model
New Insights
Why Temporal
Time Travel Data Audit
Slowly Changing
Dimensions
Repair record-level
corruptions

No change in programming model New Insights
INSERT / BULK INSERT
UPDATE
DELETE
MERGE
DML SELECT * FROM temporal
Querying
How to start with temporal
CREATE temporal
TABLE PERIOD FOR
SYSTEM_TIME…
ALTER regular_table
TABLE ADD
PERIOD…
DDL
FOR SYSTEM_TIME
AS OF
FROM..TO
BETWEEN..AND
CONTAINED IN
Temporal
Querying

Temporal table (actual data)
Insert / Bulk Insert
* Old versions
Update */ Delete *
How system-time works?
History Table

Temporal table (actual data)
Temporal Queries *
(Time travel,etc.)
How system-time works?
History Table
Regular queries
(current data)
* Include Historical
Version

DepNum DepName MngrID From To
A001 Marketing 5 2005 2008
A002 Sales 2 2005 2007
A003 Consulting 6 2005 2006
A003 Consulting 10 2009 2012
DepNum DepName MngrID
A001 Marketing 5
A001 Marketing 6
A002 Sales 2
A002 Sales 5
A003 Consulting 6
A003 Consulting 10
DepNum DepName MngrID From To
A001 Marketing 6 2008 ∞
A002 Sales 5 2007 ∞
A001
A002
A003
period of validity current time
∞
∞
Department (current)
Department (history)
Department (current + history)
2005 2015
SELECT * FROM DepartmentSELECT * FROM Department FOR SYSTEM_TIME
BETWEEN '2006.01.01' AND '2007.01.01'
SELECT * FROM Department FOR SYSTEM_TIME
CONTAINED IN ('2007.01.01', '2009.01.01')
SELECT * FROM Department FOR SYSTEM_TIME
AS OF '2006.01.01'
Getting insights from temporal – AS OF
A001
A002
A003
“Get actual row versions”AS OFBETWEEN..ANDCONTAINED IN

SELECT * FROM Department
FOR SYSTEM_TIME
AS OF '2010.01.01' Facts:
1. History is much bigger than actual data
2. Retained between 3 and 10 years
3. “Warm”: up to a few weeks/months
4. “Cold”: rarely queried
Solution:
History as a stretch table:
PeriodEnd < “Now - 6 months”
Azure SQL Database

Provides correct information
about stored facts at any
point in time, or between 2
points in time.
There are two orthogonal
sets of scenarios with
regards to temporal data:
System(transaction)-time
Application-time
SELECT * FROM Person.BusinessEntityContact
FOR SYSTEM_TIME BETWEEN @Start AND @End
WHERE ContactTypeID = 17
Performance
Temporal database support - BETWEEN

Prevents Data
Disclosure
Client-side encryption of
sensitive data using keys that
are never given to the
database system.
Queries on
Encrypted Data
Support for equality
comparison, incl. join, group
by and distinct operators.
Application
Transparency
Minimal application changes
via server and client library
enhancements.
Allows customers to securely store sensitive data outside of their trust boundary.
Data remains protected from high-privileged, yet unauthorized users.
Benefits of Always Encrypted

dbo.Patients
Jane Doe
Name
243-24-9812
SSN
USA
Country
Jim Gray 198-33-0987 USA
John Smith 123-82-1095 USA
dbo.Patients
Jane Doe
Name
1x7fg655se2e
SSN
USA
Jim Gray 0x7ff654ae6d USA
John Smith 0y8fj754ea2c USA
Country
Result Set
Jim Gray
Name
Jane Doe
Name
1x7fg655se2e
SSN
USA
Country
Jim Gray 0x7ff654ae6d USA
John Smith 0y8fj754ea2c USA
dbo.Patients
SQL Server
Query
Trusted
Apps
SELECT Name FROM
Patients WHERE SSN=@SSN
@SSN='198-33-0987'
Result Set
Jim Gray
Name
SELECT Name FROM
Patients WHERE SSN=@SSN
@SSN=0x7ff654ae6d
Column
Encryption
Key
Enhanced
ADO.NET
Library
Column
Master
Key
Client side
Always Encrypted
Help protect data at rest and in motion, on-premises & cloud
ciphertext

Randomized encryption
Encrypt('123-45-6789') = 0x17cfd50a
Repeat: Encrypt('123-45-6789') = 0x9b1fcf32
Allows for transparent retrieval of encrypted
data but NO operations
More secure
Deterministic encryption
Encrypt('123-45-6789') = 0x85a55d3f
Repeat: Encrypt('123-45-6789') = 0x85a55d3f
Allows for transparent retrieval of encrypted
data AND equality comparison
E.g. in WHERE clauses and joins, distinct,
group by
Two types of encryption
available
Randomized encryption uses a
method that encrypts data in a less
predictable manner
Deterministic encryption uses a
method which always generates
the same encrypted value for any
given plain text value
Types of Encryption for Always Encrypted

Security
Officer
1. Generate CEKs and Master Key
2. Encrypt CEK
3. Store Master Key Securely
4. Upload Encrypted CEK to DB
CMK Store:
Certificate Store
HSM
Azure Key Vault
…
Encrypted
CEK
Column
Encryption Key
(CEK)
Column
Master Key
(CMK)
Key Provisioning
CMK
Database
Encrypted CEK

Param
Encryption
Type/
Algorithm
Encrypted
CEK Value
CMK Store
Provider
Name CMK Path
@Name Non-DET/
AES 256
CERTIFICATE_
STORE
Current User/
My/f2260…
EXEC sp_execute_sql
N'SELECT * FROM Customers WHERE SSN = @SSN'
, @params = N'@SSN VARCHAR(11)', @SSN=0x7ff654ae6d
Param
Encryption
Type/
Algorithm
Encrypted
CEK Value
CMK Store
Provider
Name CMK Path
@SSN DET/ AES
256
CERTIFICATE_
STORE
Current User/
My/f2260…
Enhanced
ADO.NET
Plaintext
CEK
Cache
exec sp_describe_parameter_encryption
@params = N'@SSN VARCHAR(11)'
, @tsql = N'SELECT * FROM Customers WHERE SSN = @SSN'
Result set (ciphertext)
Name
Jim Gray
Result set (plaintext)
using (SqlCommand cmd = new SqlCommand(
"SELECT Name FROM Customers WHERE SSN =
@SSN“
, conn))
{
cmd.Parameters.Add(new SqlParameter(
"@SSN", SqlDbType.VarChar, 11).Value =
"111-22-3333");
SqlDataReader reader =
cmd.ExecuteReader();
Client - Trusted SQL Server - Untrusted
Encryptionmetadata
Name
0x19ca706fbd9
Encryptionmetadata
CMK Store
Example

Select columns to
be encrypted
Analyze schema
and application
queries to detect
conflicts (build
time)Set up the keys:
master & CEK
Static schema
analysis tool
(SSDT only)
UI for selecting columns (no
automated data classification)
Key setup tool to automate
selecting CMK, generating and
encrypting CEK and uploading
key metadata to the database
Setup (SSMS or SSDT)
User Experience: SSMS or SSDT (Visual Studio)

Existing App – Setup
User Experience: SSMS or SSDT (Visual Studio)
UI for selecting columns
(no automated data
classification)
Select candidate
columns to be
encrypted
Analyze schema and
application queries to
detect conflicts and
identify optimal
encryption settings
Set up the keys
Encrypt selected
columns while
migrating the
database to a target
server (e.g. in Azure
SQL Database
Key Setup tool to
streamline selecting CMK,
generating and encrypting
CEK and uploading key
metadata to the database
Encryption tool creating
new (encrypted) columns,
copying data from old
(plain text) columns,
swapping columns and re-
creating dependencies
Select desired
encryption settings
for selected columns
UI for configuring
encryption settings on
selected columns
(accepting/editing
recommendations from
the analysis tool)
Schema/workload analysis
tool analyzing the schema
and profiler logs

Data remains encrypted
during query
Summary: Always encrypted
Protect data at rest and in motion, on-premises & cloud
Capability
ADO.Net client library provides
transparent client-side encryption, while
SQL Server executes T-SQL queries on
encrypted data
Benefits
Apps TCE-enabled
ADO .NET library
SQL ServerEncrypted
query
Columnar
key
No app
changes
Master
key

Row-Level Security.
SQL Server 2016
SQL Database

Fine-grained access control over specific rows in a
database table
Help prevent unauthorized access when multiple users
share the same tables, or to implement connection
filtering in multitenant applications
Administer via SQL Server Management Studio or SQL
Server Data Tools
Enforcement logic inside the database and schema
bound to the table.
Protect data privacy by ensuring the right access across rows
SQL Database
Customer 1
Customer 2
Customer 3
Row-level security

Fine-grained
access control
Keeping multi-tenant
databases secure by limiting
access by other users who
share the same tables.
Application
transparency
RLS works transparently at
query time, no app changes
needed.
Compatible with RLS in other
leading products.
Centralized
security logic
Enforcement logic resides
inside database and is
schema-bound to the table it
protects providing greater
security. Reduced application
maintenance and complexity.
Store data intended for many consumers in a single database/table while at the same time
restricting row-level read & write access based on users’ execution context.
Benefits of row-level security

CREATE SECURITY POLICY mySecurityPolicy
ADD FILTER PREDICATE dbo.fn_securitypredicate(wing, startTime, endTime)
ON dbo.patients
Predicate function
User-defined inline table-valued function (iTVF) implementing security logic
Can be arbitrarily complicated, containing joins with other tables
Security predicate
Applies a predicate function to a particular table (SEMIJOIN APPLY)
Two types: filter predicates and blocking predicates
Security policy
Collection of security predicates for managing security across multiple tables
RLS Concepts

Dynamic Data Masking
SQL Server 2016
SQL Database

Configuration made easy in the new Azure
portal
Policy-driven at the table and column level, for
a defined set of users
Data masking applied in real-time to query
results based on policy
Multiple masking functions available (e.g. full,
partial) for various sensitive data categories
(e.g. Credit Card Numbers, SSN, etc.)
SQL Database
SQL Server 2016 CTP2
Table.CreditCardNo
4465-6571-7868-5796
4468-7746-3848-1978
4484-5434-6858-6550
Real-time data masking;
partial masking
Prevent the abuse of sensitive data by hiding it from users

Audit success/failure of
database operations
Enhanced auditing for
OLTP with ability to track
history of record
changes
Transparent Data
Encryption support for
storage of In-memory
OLTP Tables
Backup encryption now
supported with
compression
Other security enhancements

disk-based OLTP
In-memory OLTP
enhancements
CPUs
Query data store
Native JSON
Temporal database
support
Query data as points in time
Always encrypted
Row-level security
to table rows
Other enhancements
operations
memory OLTP tables
changes
Enhanced AlwaysOn
replicas
database health
AlwaysOn
Enhanced database
caching

Greater scalability:
Load balancing readable secondaries
Increased number of auto-failover targets
Log transport performance
Improved manageability:
DTC support
Database-level health monitoring
Group managed service account
AG_Listener
New York
(Primary)
Asynchronous data
Movement
Synchronous data
Movement
Unified HA Solution
Enhanced AlwaysOn Availability Groups
AG
Hong Kong
(Secondary)
AG
New Jersey
(Secondary)
AG

DR Site Computer2
Computer3
Computer4
Computer5
READ_ONLY_ROUTING_LIST=
(('COMPUTER2', 'COMPUTER3',
'COMPUTER4'), 'COMPUTER5')
Primary Site
Computer1
(Primary)
Readable Secondary load balancing

Ability to rebuild online indexes in a single partition
provides partition-level control for users who need
continual access to the database.
Allows database administrators to specify whether or
not to terminate processes that block their ability to
lock tables.
SQL Server 2016 CTP2 now provides 100% uptime with
enhanced online database operations when conducting
ALTER or TRUNCATE operations on tables.
Enhanced online operations

disk-based OLTP
In-memory OLTP
enhancements
CPUs
Query data store
Native JSON
Temporal database
support
query data as points in time
Always encrypted
Row-level security
to table rows
Other enhancements
operations
memory OLTP tables
changes
Enhanced AlwaysOn
replicas
database health
AlwaysOn
Enhanced database
caching

Supports caching data with automatic, multiple TempDB
files per instance in multi-core environments
Reduces metadata and allocation contention for TempDB
workloads, improving performance and scalability
Enhanced database caching

Enhanced support for Windows Server
Hardware Acceleration for TDE Encryption/Decryption
Parallelizing the Decryption Built-in Function to Improve Read Performance
Results in dramatically better response times for queries with encrypted data columns

Switches
Networking
Compute
Rack 2
Edge
Storage
Networking
Compute
Rack 1
Edge
Storage
Management Switches
Networking
Compute
Rack 3
Edge
Storage
Switches
Networking
Compute
Rack 4
Edge
Storage
Cloud Platform System advantages
Partnership with Dell.
Combination of optimized
hardware/software.
Designed specifically to reduce the
complexity and risk of implementing a
self-service cloud.
CPS can go from delivery to live within
days—not months—and lets service
providers and enterprises move up the
stack to focus on delivering services to
users

Access any data Scale and manage Powerful Insights Advanced analytics
PolyBase
Insights from data across SQL
Server and Hadoop with simplicity
of T-SQL
Enhanced SSIS
Designer support for previous SSIS
versions
Support for Power Query
Enterprise-grade
Analysis Services
Enhanced performance and
scalability for analysis services
Single SSDT in Visual Studio
2015 (CTP3)
Build richer analytics solutions as
part of your development projects
in Visual Studio
Enhanced MDS
Excel add-in 15x faster; more
granular security roles; archival
options for transaction logs; and
reuse entities across models
Mobile BI
Business insights for your on-
premises data through rich
visualization on mobile devices
with native apps for Windows, iOS
and Android
Enhanced Reporting
Services
New modern reports with rich
visualizations
R integration (CTP3)
Bringing predictive analytic
capabilities to your relational
database
Analytics libraries (CTP3)
Expand your “R” script library with
Microsoft Azure Marketplace
Deeper insights across data

PolyBase
of T-SQL
Enhanced SSIS
versions
Enterprise-grade
Analysis Services
2015
in Visual Studio
Enhanced MDS
Mobile BI
and Android
Enhanced Reporting
Services
visualizations
database

Query relational
and non-relational
data, on-premises
and in Azure
Apps
T-SQL query
SQL Server Hadoop
PolyBase
Query relational and non-relational data with T-SQL

Prerequisites for installing PolyBase
64-bit SQL Server Evaluation edition
Microsoft .NET Framework 4.0.
Oracle Java SE RunTime Environment (JRE) version 7.51 or
higher
NOTE: Java JRE version 8 does not work.
Minimum memory: 4GB
Minimum hard disk space: 2GB

Using the installation wizard for PolyBase
Run SQL Server Installation Center. (Insert SQL Server
installation media and double-click Setup.exe.)
Click Installation, then click New Standalone SQL Server
installation or add features
On the feature selection page, select PolyBase Query
Service for External Data.
On the Server Configuration Page, configure the PolyBase
Engine Service and PolyBase Data Movement Service to run
under the same account.

-- Run sp_configure ‘hadoop connectivity’
-- and set an appropriate value
sp_configure
@configname = 'hadoop connectivity',
@configvalue = 7;
GO
RECONFIGURE
GO
-- List the configuration settings for
-- one configuration name
sp_configure @configname='hadoop connectivity';
GO
Option values
0: Disable Hadoop connectivity
1: Hortonworks HDP 1.3 on Windows Server
Azure blob storage (WASB[S])
2: Hortonworks HDP 1.3 on Linux
3: Cloudera CDH 4.3 on Linux
4: Hortonworks HDP 2.0 on Windows Server
5: Hortonworks HDP 2.0 on Linux
6: Cloudera 5.1 on Linux
7: Hortonworks 2.1 and 2.2 on Linux
Hortonworks 2.2 on Windows Server
Choose Hadoop data source with sp_configure

Start the PolyBase services
After running for sp_configure,
you must stop and restart the
SQL Server engine service
Run services.msc
Find the services shown below and
stop each one
Restart the services

-- Using credentials on database requires enabling
-- traceflag
DBCC TRACEON(4631,-1)
-- Create a master key
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo';
CREATE CREDENTIAL WASBSecret ON DATABASE WITH
IDENTITY = 'pdw_user', Secret = 'mykey==';
-- Create an external data source (Azure Blob Storage)
-- with the credential
CREATE EXTERNAL DATA SOURCE Azure_Storage WITH
( TYPE = HADOOP,
LOCATION
='wasb[s]://mycontainer@test.blob.core.windows.net/pat
h’,
CREDENTIAL = WASBSecret
)
Type methods for providing
credentials
Core-site.xml in installation path of
SQL Server -
<SqlBinRoot>PolybaseHadoopConf
Credential object in SQL Server for
higher security
NOTE: The syntax for a database-scoped
credential (CREATE CREDENTIAL … ON
DATABASE) is temporary and will change
in the next release. This new feature is
documented only in the examples in the
CTP2 content, and will be fully
documented in the next release.
Configure PolyBase for Azure blob storage

-- Create an external data source (Hadoop)
CREATE EXTERNAL DATA SOURCE hdp2 with (
TYPE = HADOOP,
LOCATION ='hdfs://10.xxx.xx.xxx:xxxx',
RESOURCE_MANAGER_LOCATION='10.xxx.xx.xxx:xxxx')
CTP2 supports the following
Hadoop distributions
Hortonworks HDP 1.3, 2.0, 2.1, 2.2 for
both Windows and Linux
Cloudera CDH 4.3, 5.1 on Linux
Create a reference to a Hadoop cluster

-- Create an external file format
-- (delimited text file)
CREATE EXTERNAL FILE FORMAT ff2 WITH (
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS (FIELD_TERMINATOR ='|',
USE_TYPE_DEFAULT = TRUE))
CTP2 supports the following file
formats
Delimited text
Hive RCFile
Hive ORC
Define the external file format

-- Create an external table pointing to file
stored in Hadoop
CREATE EXTERNAL TABLE [dbo].[CarSensor_Data] (
[SensorKey] int NOT NULL,
[CustomerKey] int NOT NULL,
[GeographyKey] int NULL,
[Speed] float NOT NULL,
[YearMeasured] int NOT NULL
)
WITH (LOCATION='/Demo/car_sensordata.tbl',
DATA_SOURCE = hdp2,
FILE_FORMAT = ff2,
REJECT_TYPE = VALUE,
REJECT_VALUE = 0
The external table provides a T-
SQL reference to the data source
used to:
Query Hadoop or Azure blob storage
data with Transact-SQL statements
Import and store data from Hadoop
or Azure blob storage into your SQL
Server database
Create an external table to the data source

-- Create statistics on an external table.
CREATE STATISTICS StatsForSensors ON
CarSensor_Data(CustomerKey, Speed)
In CTP2, you can optimize query
execution against the external
tables using statistics
Optimize queries by adding statistics

SELECT DISTINCT C.FirstName, C.LastName,
C.MaritalStatus
FROM Insurance_Customer_SQL
INNER JOIN (
SELECT * FROM SensorData_ExternalHDP WHERE
Speed > 35
UNION ALL
SELECT * FROM SensorData_ExternalHDP2 WHERE
Speed > 35
) AS SensorD
ON C.CustomerKey = SensorD.CustomerKey
External tables
referring to data
in 2 HDP Hadoop
clusters
SQL Server table
Query Capabilities (1)
Joining relational and external data

SELECT DISTINCT C.FirstName, C.LastName, C.MaritalStatus
FROM Insurance_Customer_SQL -- table in SQL Server
…
OPTION (FORCE EXTERNALPUSHDOWN) – push-down computation
CREATE EXTERNAL DATA SOURCES ds-hdp WITH .( TYPE = Hadoop,
LOCATION = “hdfs://10.193.27.52:8020”,
Resources_Manager_Location = ‘10.193.27.52:8032’);
Query Capabilities (2)
Push-Down Computation

SSIS improvements for SQL Server 2016
CTP2
AlwaysOn support
Incremental deployment of
packages
Improved project upgrade support
CTP3
Designer improvements
One designer multi-version support
OData V4 support
Power Query as a data source
AlwaysOn
Availability Groups
Secondary for
SSISDB
New York
(Primary)
New Jersey
(Secondary)
SSIS
DB
SSIS
DB
SQL Server 2012
SSIS Project X
SQL Server 2016
SSIS Project X
Improved project
upgrade

AlwaysOn for SSIS Catalog
Steps to configure AlwaysOn
for SSIS catalog
Create Integrations Services catalog
Add SSISDB to an AlwaysOn
Availability Group
Enable SSIS support for AlwaysOn

private static void Main(string[] args)
{
// Connection string to SSISDB
var connectionString = "Data Source=.;Initial Catalog=SSISDB;Integrated
Security=True;MultipleActiveResultSets=false";
using (var sqlConnection = new SqlConnection(connectionString))
{
sqlConnection.Open();
var sqlCommand = new SqlCommand
{
Connection = sqlConnection,
CommandType = CommandType.StoredProcedure,
CommandText = "[catalog].[deploy_packages]"
};
var packageData = Encoding.UTF8.GetBytes(File.ReadAllText(@"C:TestPackage.dtsx"));
// DataTable: name is the package name without extension and package_data is byte
array of package.
var packageTable = new DataTable();
packageTable.Columns.Add("name", typeof(string));
packageTable.Columns.Add("package_data", typeof(byte[]));
packageTable.Rows.Add("Package", packageData);
// Set the destination project and folder which is named Folder and Project.
sqlCommand.Parameters.Add(new SqlParameter("@folder_name", SqlDbType.NVarChar,
ParameterDirection.Input, "Folder", -1));
sqlCommand.Parameters.Add(new SqlParameter("@project_name", SqlDbType.NVarChar,
ParameterDirection.Input, "Project", -1));
sqlCommand.Parameters.Add(new SqlParameter("@packages_table", SqlDbType.Structured,
ParameterDirection.Input, packageTable, -1));
var result = sqlCommand.Parameters.Add("RetVal", SqlDbType.Int);
result.Direction = ParameterDirection.ReturnValue;
sqlCommand.ExecuteNonQuery();
}
}
Deployment options in CTP2
Integration Services Deployment
Wizard
SQL Server Management Studio
deploy_packages stored procedure
Object model API
Deployment options in CTP3
SQL Server Data Tools for Business
Intelligence
Deploy packages to Integration Services server

Enterprise grade
Analysis Services

Scale your tabular
models with support
for high end servers
More memory
More cores
Enhanced Analysis Services
Deliver high performance and scalability for your BI solutions
High-end server hardware
Capability
Parallel partition processing
NUMA optimization for tabular models
On-demand loading and paging
Tabular and MOLAP modeling enhancements
Benefits

SSAS Enterprise Readiness: MOLAP
Netezza as a data source
Performance improvements
(unnatural hierarchies, ROLAP
distinct count, drill-through,
semi-additive measures)
Detect MOLAP index
corruption using DBCC

High-end Server Hardware
SSAS Enterprise Readiness: Tabular
Processing optimizations
Parallel partition processing
Support for Power Query in AS engine

Support for BI directional cross filtering (M:M)
Calculated tables, Date Table
Translations
Tabular scripting language
New DAX functions (DATEDIFF, GEOMEAN,
PERCENTILE, PRODUCT, XIRR, XNPV, many more..)
Query engine optimizations
New DAX functions for high perf. reporting
Direct Query perf. optimizations
DimDepartmentGr…
DimDepartmentGroupKey…
ParentDepartmentGr…
DepartmentGroupNa…
DimAccount
AccountKey
ParenAccountKey
AccountCodeAlter…
ParenAccountCod…
AccountDescription
FactFinance
FinanceKey
DateKey
OrganizationKey
DepartmentGroup…
ScenarioKey
DimOrganization
OrganizationKey
ParnentOrganization
PercentageOfOwn…
OrganizationName
CurrencyKey
DimAccount
ScenarioKey
ScenarioName
SSAS Rich Modeling Platform: Tabular

Unified Visual
Studio Data Tools
(CTP3)

Capability
SSDT-BI and SSMS for Visual Studio 2015
Rich data modeling enhancements
New DAX analytical functions
MDX Query Plan tool for performance optimizations and troubleshooting
Benefits{JSON}
Powerful Insights
Rich BI Application Platform
Develop and deliver BI solutions faster and at lower costs

Master Data
Services
improvements

Performance and Scale
Overall performance and scale
improvements
Target of 15x performance
improvement for Excel add-in
Increased performance for bulk
entity based staging operations
Security Improvements
New security roles for more
granular permissions around read,
write and delete
Multiple system administrators
Manageability and Modeling
Archival of Transaction Logs
Reuse of entities across models
Support for compound keys
Allow the Name and Code attributes
to be renamed
MDS Improvements

Mobile BI apps for SQL Server – formally Datazen
For on-premises
implementations - optimized
for SQL Server.
Rich, interactive data
visualization on all major
mobile platforms
No additional cost for
SQL Server Enterprise Edition
customers 2008 or later and
Software Assurance

Data visualization and publishing
Create beautiful visualizations
and KPIs with a touch-based
designer
Connect to the Mobile BI for
SQL Server server to
access SQL Server data
Publish for access by others

Datazen architecture overview
Server, Publisher and Mobile apps
SQL Server
Analysis Services
File
Data sources
Authentication
Internet boundary
Viewer
Apps
Publisher
App
Web
browser
Datazen Enterprise
Server
Enterprise environment

Mobile BI for SQL Server summary
Beautiful visualizations
KPI repository
Create once - publish to any
device
Native apps for all platforms
Perfect scaling to any form
factor
Custom branding
Collaborate on the go

Modern reports with SQL Server Reporting Services
Report consumption from
modern browsers
Improved parameters
Modern themes
New chart types

PolyBase
of T-SQL
Enhanced SSIS
versions
Enterprise-grade
Analysis Services
2015
in Visual Studio
Enhanced MDS
Mobile BI
and Android
Enhanced Reporting
Services
visualizations
database
Deeper insights across data (CTP3)

R integration with
database engine
(CTP3)

Example Solutions
Fraud detection
Salesforecasting
Warehouse efficiency
Predictive maintenance
Extensibility
Microsoft Azure
Machine Learning Marketplace
New R scripts
010010
100100
010101
010010
100100
010101
010010
100100
010101
010010
100100
010101
Built-in advanced analytics (CTP3)
In-database analytics
Built-in to SQL Server
Analytic Library
T-SQL Interface
Relational Data
010010
100100
010101
010010
100100
010101
Data Scientist
Interact directly
with data
Data Developer/DBA
Manage data and
analytics together
?
R
RIntegration

Capability
Extensible In-database analytics, integrated with
R, exposed through T-SQL
Centralize enterprise library for analytic models
Benefits
SQL Server
Analytical Engines
Full R Integration
Fully Extensible
Data Management Layer
Relational Data
T-SQL Interface
Stream Data In-Memory
Analytics Library
Share and Collaborate
Manage and Deploy
R +
Data Scientists
Business
Analysts
Publish algorithms, and
interact directly with Data
Analysis through TSQL, tools,
and vetted algorithms
DBAs
Manage storage and
analytics together
Advanced Analytics (CTP3)
Faster analytics with In-database processing

Hybrid solutions Simplicity Consistency
Stretch Database
Stretch operational tables in a secure manner
into Azure for cost effective historic data
availability works with Always Encrypted and
Row Level Security
Power BI with on-premises data
New interactive query with Analysis Services.
Customer data stays behind your firewall
Hybrid Scenarios with SSIS
Azure Data Factory integration with SSIS,
package lineage and impact analysis and
connect SSIS to cloud data source
Enhanced Backup to Azure
Faster restore times and 50% reduction in
storage, support larger DBs with Block blobs
and custom backup schedule with local staging
Easy migration of on-premises SQL
Server
Simple point and click migration to Azure
Simplified Add Azure
Replica Wizard
Automatic listener configuration for AlwaysOn
in Azure VMs
Common development, management
and identity tools
Including Active Directory, Visual Studio, Hyper-
V and System Center
Consistent Experience from SQL
Server on-premises to Microsoft
Azure IaaS and PaaS

Capability
Stretch large operational tables
from on-premises to Azure with
the ability to query
Benefits
BI integration
for on-premises
and cloud
Cold/closed data
Orders
In-memory
OLTP table
Hot/active data
Order history
Stretched table
Trickle data movement and
remote query processing
On-premises Azure
Stretch SQL Server into Azure
Securely stretch cold tables to Azure with remote query processing

Stretch database architecture
How it works
Creates a secure linked server
definition in the on-premises SQL
Server
Linked server definition has the
remote endpoint as the target
Provisions remote resources and
begins to migrate eligible data, if
migration is enabled
Queries against tables run against
both the local database and the
remote endpoint
Remote
Endpoint
Remote
Data
Azure
InternetBoundary
Local
Database
Local
Data
Eligible
Data

-- Enable local server
EXEC sp_configure 'remote data archive' , '1';
RECONFIGURE;
-- Provide administrator credential to connect to
-- Azure SQL Database
CREATE CREDENTIAL <server_address> WITH
IDENTITY = <administrator_user_name>,
SECRET = <administrator_password>
-- Alter database for remote data archive
ALTER DATABASE <database name>
SET REMOTE_DATA_ARCHIVE = ON (SERVER = server name);
GO
-- Alter table for remote data archive
ALTER TABLE <table name>
ENABLE REMOTE_DATA_ARCHIVE
WITH ( MIGRATION_STATE = ON );
GO;
High level steps
Configure local server for remote
data archive
Create a credential with
administrator permission
Alter specific database for remote
data archive
Alter table for remote data archive
Typical workflow to enable Stretch Database

On-premises
integration with
Power BI

Live dashboards
and exploration
Analysis Services
on-premises
Tabular model
Interactive queryAS Connector
Capability
Publish on-premises Analysis Services data
models for consumption in Power BI
Benefits
SQLServer vNext
Connect live to on-premises Analysis Services data

SSIS Improvements for Azure services (CTP3)
Capability
Stretch large operational tables
from on-premises to Azure with
the ability to query
Benefits
SQL

Managed backup
Granular control of the backup
schedule;
Local staging support for faster
recovery and resilient to transient
network issues;
Support for system databases;
Support simple recovery mode.
Backup to Azure block blobs
Cost savings on storage;
Significantly improved restore
performance; and
More granular control over Azure
storage.
Azure Storage snapshot
backup
Fastest method for creating
backups and running restores
Uses SQL Server db files on Azure
Blob Storage
Enhanced backup to Azure

Hybrid solutions Simplicity Consistency
Stretch Database
Stretch operational tables in a secure manner
into Azure for cost effective historic data
availability works with Always Encrypted and
Row Level Security
PowerBI with on-premises data
New interactive query with Analysis Services.
Customer data stays behind your firewall
Hybrid Scenarios with SSIS
Azure Data Factory integration with SSIS,
package lineage and impact analysis and
connect SSIS to cloud data source
Enhanced Backup to Azure
Faster restore times and 50% reduction in
storage, support larger DBs with Block blobs
and custom backup schedule with local staging
Easy migration of on-premises SQL
Server
Simple point and click migration to Azure
Simplified Add Azure
Replica Wizard
Automatic listener configuration for AlwaysOn
in Azure VMs
Common development, management
and identity tools
Including Active Directory, Visual Studio, Hyper-
V and System Center
Consistent Experience from SQL
Server on-premises to Microsoft
Azure IaaS and PaaS

User DB System Objects SQL Settings
Migration Wizard
On Premise
EZ Button – Migrate to Azure
Simple Single Click Migration Experience
Capability
Publish on-premises Analysis Services data
models for consumption in Power BI
Benefits

Method 1 Method 2 Method 3
Target
DB
Azure SQL Database
SQL Server
SSMS
2. Import
1. DeployOR
.bacpac
1. Export
Source
DB
Target
DB
Azure SQL Database
SQL Server
SQL Azure
Migration
Wizard
2. Execute
1. Generate
T-SQL
Source
DB
Target
DB
Azure SQL Database
SQL Server
SSMS
6.
Export/
Import
or Deploy
SQL
Azure
Migrati
on
Wizard
4. Copy
Copy
DB
Source
DB
5. Publish
1. Import
Visual
Studio
Database
Project
2. Transform
3. Edit, Build
& Test
*.sql
4. Publish (schema only)
Migrate a compatible database
using SSMS
Migrate a near-compatible
database using SAMW
Update the database schema offline
using Visual Studio and SAMW, and then
deploy it with SSMS
Migration methodologies

Migration Cookbook
Migrate an on-premises SQL
Server database to Azure SQL
Database (v12).
The Migration Cookbook
describes various approaches you
can use to migrate an on-
premises SQL Server database to
the latest Azure SQL Database
Update (v12).
Download:
http://aka.ms/azuresqlmigration

Simplified
AlwaysOn with
replicas on Azure

Today this requires manually
configuring the Listener
SQL Server 2016 CTP2
Simplified Add Azure Replica Wizard
Automatic Listener Configuration

Consistent tools
and experience

Consistent Tools
Consistency across:
On-premises, Private Cloud, Public Cloud
SQL Server local, VM, Azure SQL Database
Scalability, availability, security, identity, backup and restore, and replication
Plethora of data sources
Reporting, integration, processing, and analytics
All of this support Hybrid Cloud

Options:
SQL Server on physical machines
SQL Server in on-premises VM (private
cloud)
SQL Server in Azure VM (public cloud)
Azure SQL Database (public cloud)
SQL Server, Azure VMs, Azure SQL DB
Shared
Lower Cost
Dedicated
Higher
Cost
Higher Administration Lower Administration
Off Premises
Hybrid Cloud
Physical
SQL Server
Physical Machines(raw Iron)SQL
Virtual
SQL Server Private Cloud
Virtualized Machines +
Appliances
Infrastructure
as a service
SQL Server in Azure VM
Virtualized machinesSQL
Platform as
a service
Software as
a services
Azure SQL Database
Virtualized Databases
SQL
On Premises

The Microsoft
data platform
Internal &
external
DashboardsReports Ask Mobile
Information
managementOrchestration
Extract, transform,
load Prediction
Relational Non-relational Analytical
Apps
Streaming


© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on
the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Expert summit SQL Server 2016

More Related Content

What's hot

Viewers also liked

Similar to Expert summit SQL Server 2016

More from Łukasz Grala

Recently uploaded

Expert summit SQL Server 2016

Editor's Notes