Niko Neugebauer gave a presentation on the columnstore improvements in SQL Server 2016. Some of the key improvements discussed include hybrid transactional/analytical processing (HTAP), new T-SQL syntax for defining columnstore indexes, high availability features like readable secondaries, improved data loading and batch processing performance, new maintenance features, and expanded monitoring capabilities. The presentation provided examples and demonstrations of many of these new columnstore features in SQL Server 2016.

1. Columnstore
Improvements
in SQL Server 2016
Niko Neugebauer, OH22

2. Please silence
cell phones

3. Explore Everything PASS Has to Offer
FREE ONLINE WEBINAR EVENTS FREE 1-DAY LOCAL TRAINING EVENTS
LOCAL USER GROUPS
AROUND THE WORLD
ONLINE SPECIAL INTEREST
USER GROUPS
BUSINESS ANALYTICS TRAINING
VOLUNTEERING OPPORTUNITIES
PASS COMMUNITY NEWSLETTER
BA INSIGHTS NEWSLETTERFREE ONLINE RESOURCES

4. Session Evaluations
ways to access
Go to passSummit.com Download the GuideBook App
and search: PASS Summit 2016
Follow the QR code link displayed
on session signage throughout the
conference venue and in the
program guide
Submit by 5pm
Friday November 6th to
WIN prizes
Your feedback is
important and valuable. 3

5. Niko Neugebauer
Consultant, OH22
• Microsoft Data Platform Professional
• OH22 (http://www.oh22.net)
• Data Platform MVP
• Founder of a couple of Portuguese PASS Chapters
• Creator of CISL – Free & Open Source
Columnstore Indexes Script Library
(https://github.com/NikoNeugebauer/CISL)
Place your
photo here
/WebCaravela @NikoNeugebauer www.nikoport.com

6. Our Today’s Columnstore Agenda p. 1
• HTAP aka Operational Analytics
• Data Warehousing (CCI)
• T-SQL
• High Availability
• Batch Mode Changes
• Data Loading (+Minimal Logging)
• Performance (Batch, SSIS)

7. Our Today’s Columnstore Agenda p. 2
• Memory Managment
• Change Tracking
• Maintenance
• Monitoring Improvements (DMV, Extended Events, Performance Counters)
• Configuration Improvement (Test it!)
• Indexed Views
• New Trace Flags
• Upcoming Attraction 

8. HTAP
Hybrid Transactional
/Analytical Processing

9. Analytics
Operational

10. What is it all about:
• Predictive Analytics, Prescriptive Analytics, Business
Analytics, Machine Learning, Data Mining
But technically it all goes down to the mining 
one thing we treasure quite a lot: the Data
Analytics

11. Analytics – is the process of discovery & communication
of meaningful patterns in data.
Reporting - extraction of the aggregated information for
further analysis.
Querying - data extraction process
Analytics Faces

12. Traditional Operational/Analytics
architecture
Key issues
Complex implementation
Requires two servers (capital
expenditures and operational expenditures)
Data latency in analytics
More businesses demand;
requires real-time analytics
IIS Server
BI analysts

13. • Costs
• Integration Problems (data types, constraints, network
problems, etc)
• Delay for getting the actual data
Traditional Operational Analytics
Problems:

14. Nothing substitutes analytics queries performance,
possible using schemas customized (Star/Snowflake) and/or
pre-aggregated cubes
Modern Operational Analytics notes:

15. Minimizing data latency for analytics
Benefits
No data latency
No ETL
No separate data warehouse
Challenges
Analytics queries are resource intensive and
can cause blocking
Minimizing impact on operational workloads
Sub-optimal execution of analytics on
relational schema
IIS Server
BI analysts

16. Optimizing data latency for analytics
BI analysts
What is operational analytics and what does it
mean to you?
Operational analytics with disk-based tables
Operational analytics with In-Memory OLTP
IIS Server

17. There are now 2 types of Operational Analytics:
Operational Analytics for Disk-Based
Operational Analytics for InMemory
Operational Analytics: SQL Server 2016

18. Updatable Columnstore SQL Server 2014

19. Disk-based Operational Analytics

20. Operational Analytics Query Processing

21. Operational
Analytics
Demo

22. Filtered Operational Analytics
Create a filtered Nonclustered Columnstore Index!

23. Operational
Analytics
Demo with a Filtered Index

24. In-Memory Operational Analytics

25. In-Memory Operational Analytics
sys.sp_memory_optimized_cs_migration –
compresses data from InMemory OLTP Table into
In-Memory Columnstore

26. Operational
Analytics
Demo Memory-Optimized

27. Compression
Delay

28. Compression Delay
Warm Data Hot Data
Recent DataArchive Data
Compressed Row Groups Delta-Stores

29. The Problem:
When we move a row from Delta-Store into the
compressed Row Group, it becomes compressed.
Once it is updated, the row is marked as deleted in
the Deleted Bitmap/Deleted Buffer and the new
value is inserted in Delta-Store.

30. The Solution is the Compression Delay
Default delay is 0 min
alter index PK_NCCi_test_inline
on dbo.ncci_test_inline
set (COMPRESSION_DELAY = 60 Minutes);

31. Data
Warehouse
Clustered Columnstore

32. Clustered Columnstore (DWH)
Nonclustered Indexes
Primary & Foreign Keys + Unique Constraints
Nonclustered Secondary Rowstore Locking

33. T-SQL
Columnstore in SQL 2016

34. New T-SQL
• Inline Table definition of Clustered Columnstore
Index
• Inline Table definition of Nonclustered
Columnstore Index
• In-Memory OLTP & Columnstore Indexes

35. New T-SQL for Disk-Based Columnstore
CREATE TABLE [dbo].[FactOnlineSales_CCI](
[OnlineSalesKey] [int] NOT NULL,
[StoreKey] [int] NOT NULL,
[ProductKey] [int] NOT NULL,
[PromotionKey] [int] NOT NULL,
[CurrencyKey] [int] NOT NULL,
[CustomerKey] [int] NOT NULL,
INDEX PK_FactOnlineSales_CCI CLUSTERED COLUMNSTORE
);
CREATE TABLE [dbo].[FactOnlineSales_NCCI](
[OnlineSalesKey] [int] NOT NULL,
[StoreKey] [int] NOT NULL,
[ProductKey] [int] NOT NULL,
[PromotionKey] [int] NOT NULL,
[CurrencyKey] [int] NOT NULL,
[CustomerKey] [int] NOT NULL,
INDEX PK_FactOnlineSales_CCI NONCLUSTERED COLUMNSTORE (OnlineSalesKey, CustomerKey)
);

36. New T-SQL for In-Memory Columnstore
CREATE TABLE [dbo].[FactOnlineSales_Hekaton](
[OnlineSalesKey] [int] NOT NULL,
[StoreKey] [int] NOT NULL,
[ProductKey] [int] NOT NULL,
[PromotionKey] [int] NOT NULL,
[CurrencyKey] [int] NOT NULL,
[CustomerKey] [int] NOT NULL,
Constraint PK_FactOnlineSales_Hekaton PRIMARY KEY NONCLUSTERED ([OnlineSalesKey]),
INDEX CCI_FactOnlineSales_Hekaton
CLUSTERED COLUMNSTORE
) WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);
ALTER TABLE dbo. FactOnlineSales_Hekaton
ADD INDEX CCI_FactOnlineSales_Hekaton
CLUSTERED COLUMNSTORE;

37. HA
High Availability

38. High Availability

39. High Availability
Readable Secondaries for Availability Groups through
Snapshot &
Read Committed Snapshot
Isolation Levels support

40. Transactional Replication
We can now control, how the HTAP
Columnstore Indexes are being replicated –
if they should be created on the Subscriber or not.

41. Transactional Replication

42. Further Columnstore Improvements
• Data Loading Improvements
• Batch Mode
• Performance Improvements
• Change Tracking
• Maintenance Improvements
• Monitoring Improvements (DMV, Extended Events, Performance Counters)

43. Data Loading Improvements
• Parallel Data Loading finally enabled
• SIMD support
• Delta-Stores are not Page-Compressed!!!
• Under NCCI Delta-Stores are automatically increasing their max size with
each iteration under pressure
• Compression Delay
• SSIS
• Minimal Logging

44. Parallel Data Loading

45. SIMD Improvements
Columnstore is SIMD-aware and uses it for
different purposes, such as:
- Huffman decoding
- Predicate Pushdown
- String Hash

46. Delta-Stores for NCCI
- Under memory pressure the size of the created
Delta-Stores for the Nonclustered Columnstore
Indexes will be doubled for each succeeding
operation from 1048576 rows to ~2 Million
Rows, ~4 Million Rows, ... 32 Million Rows

47. The solution: Compression Delay
Default delay is 0 min
alter index PK_NCCi_test_inline
on dbo.ncci_test_inline
set (COMPRESSION_DELAY = 60 Minutes);

48. SSIS 2016 Improvements
AutoAdjustBufferSize - True
Warning: It’s not enough, you will need to set up the
maximum number of rows per buffer, otherwise
your performance will be extremely slow!
DefaultBufferMaxRows = 1048576

49. Minimal Logging

50. Minimal Logging

51. Memory
Improvements

52. Dynamic Memory Grants
Trace Flag 9389 - Enables dynamic memory grant
for batch mode operators.
Memory Grant Memory Required Dynamic Memory Grant

53. Batch Mode
Improvements

54. Batch Mode Improvements
• Batch Mode support for 1 core execution plan operators
• Batch Mode support for the Sort operator
• Batch Mode support for the Multiple Distinct Count
operations
• Batch Mode support for the Left Anti-Semi Join operators

55. Batch Sorting Improvements
Trace Flag Description
9347 Disables batch mode sort operator
9349 Disables batch mode top sort operator
9358
Disable batch mode sort operations in a complex parallel query in SQL
Server 2016

56. Batch Mode
Demo

57. Batch Mode for Windowing Functions

58. Window
Batch Mode
Demo

59. Further Batch Improvements
• Simple Aggregate Predicate Pushdown
• Local Aggregation
• String Predicate Pushdown for Index Scan operator in
Batch Mode

60. Batch
Aggregation Pushdown Demo

61. Locally Aggregated Rows

62. String Predicate Pushdown

63. Change Tracking / CDC in SQL 2016
Columnstore Change Tracking Change Data Capture Temporal (2016+)
Clustered No No Yes
Nonclustered Yes Yes Yes
Memory-Optimized No No Yes

64. Maintenance Improvements
• Better ALTER INDEX … REORGANIZE
(removes deleted rows, less memory pressure)
• Self-Merging (this is when Row Group is being recompressed
eliminating obsolete row versions that are found in Deleted
Bitmap)
• Intergroup Merging (when the sum of active rows between
multiple Row Groups is less then 1048576, then their
information can be merged into a new single Row Group)

65. Maintenance Improvements
• Being a compressed Row Group (Delta-Stores & Tail
Row Groups are not qualified)
• A Row Group has to have at least 10% of its data being
marked as obsolete in Deleted Bitmap (for In-Memory
this threshold is set on 90%)
• – A Row Group should not being trimmed because of
the dictionary pressure.

66. Row Group Merging in SQL Server 2016
Columnstore Self-Merge Intergroup Merge Cleanup
Clustered Yes Yes Yes
Nonclustered Yes Yes*1 Yes
Memory-Optimized Yes*2 No Yes

67. New DMVs:
• sys.dm_column_store_object_pool
• sys.dm_db_column_store_row_group_physical_stats
• sys.dm_db_column_store_row_group_operational_stats
• sys.internal_partitions

68. Enhanced DMVs:
• sys.dm_db_index_operational_stats
• sys.dm_db_index_physical_stats

69. Extended Events

70. Extended Events Categories 2016
• Columnstore Index creation
• Batch Mode
• Tuple Mover
• Columnstore Internal Object Operations
• Columnstore Processing
• Columnstore Object Pool
• Errors

71. Configuration Improvement
Large Pages (Trace Flag 834)
should be supported in SQL Server 2016,
or at least the Connect Item for it is closed as fixed. 
Take them to DEV & QA Environments and let Microsoft
know the results

72. New Trace Flags
Trace Flag Description
9347 Disables batch mode sort operator
9349 Disables batch mode top sort operator
9358
Disable batch mode sort operations in a complex parallel query in SQL
Server 2016
9389 Enables dynamic memory grant for batch mode operators
10204 Disables merge/recompress during columnstore index reorganization

73. Indexed Views
SQL Server 2016 supports Indexed Views with updatable
Nonclustered Columnstore Indexes!

74. Resources:
My Columnstore Blogpost Series:
http://www.nikoport.com/columnstore
CISL – Open Source Columnstore Library:
https://github.com/NikoNeugebauer/CISL

75. Session Evaluations
ways to access
Go to passSummit.com Download the GuideBook App
and search: PASS Summit 2016
Follow the QR code link displayed
on session signage throughout the
conference venue and in the
program guide
Submit by 5pm
Friday November 6th to
WIN prizes
Your feedback is
important and valuable. 3

76. Thank You
Learn more from
Niko Neugebauer
n.neugebauer@oh22.net
or follow @NikoNeugebauer