Bussines case for Real-Time Operational Analytics Using In-Memory Technology. Columnstore overview. Tips and tricks for achieving great performance

1. SQL Server 2016
Real Time Operational Analytics
Liviu Ieran
ifliviu@live.com

2. The explosion of data sources...
…drives an explosion of data
…which drives businesses to learn more
and do more faster
2013-2020 CAGR = 41%
25B
4.0B1.3
B
2010 2013 2020
There’s an
opportunity to
drive smarter
decisions with data
Performance

3. 1
10
100
1000
10000
100000
1000000
1990
1991
1992
1993
1994
1994
1995
1996
1997
1998
1999
2000
2000
2001
2002
2004
2005
2007
2008
2009
2011
US$/GB
$ per GB of PC Class Memory
Meanwhile
RAM cost
continues to
drop
Moore’s Law on total CPU
processing power holds but
in parallel processing…
CPU clock rate stalled…
Because processors would
melt…
Hardware Trends
New CPU won’t
run a short
transaction
much faster
Performance

4. SQL In-Memory Technologies
Over 100x analytics query speed
and significant data compression
with In-Memory ColumnStore
Up to 30x faster transaction
processing with In-Memory OLTP
Faster AnalyticsFaster Transactions
IN-MEMORY OLTP IN-MEMORY DW
Performance

5. Traditional operational/analytics architecture
Key issues
Complex implementation
Requires two servers (capital
expenditures and operational
expenditures)
Data latency in analytics
High demand:
requires real-time analytics
IIS Server
BI analysts
Performance

6. Minimizing data latency for analytics
Challenges
Analytics queries are resource intensive and can
cause blocking
Minimizing impact on operational workloads
Sub-optimal execution of analytics on relational
schema
Benefits
No data latency
No ETL
No separate data warehouse
IIS Server
BI analysts
Performance

7. Real-Time Analytics – What it is NOT for
OLTP
OLTP
OLTP
• Operational Data Coming from
multiple sources
• Extreme Analytics
– Needs pre-aggregated cubes
– Star-Schema
• Challenge with OLTP schema
– Data is normalized
– Queries require multi-table joins
Performance

8. Memory Optimized Tables: Row and Hash Index Structure
90, 150 Susan Bogota
50, ∞ Jane Prague
Timestamps NameChain ptrs City
Hash index
on City
Hash index
on Name
100, 200 John Prague
200, ∞ John Beijing
f(John)
f(Jane)
f(Beijing)
f(Prague)
f(Bogota)
Performance

9. Columnstore Index: Why?
Improved compression:
• Data from same domain
compress better
• 10x compression
Reduced I/O:
• Fetch only columns needed
…
Data stored as rows Data stored as columns
Efficient operation on small
set of rows
Ideal for OLTP
C1 C2 C3 C5C4
Improved Performance:
• More data fits in memory
• Batch Mode execution
• upto100x
Ideal for DW Workload
Performance

10. Operational analytics: columnstore on in-memory tables
No explicit delta row group
Rows (tail) not in columnstore stay in In-Memory OLTP
table
No columnstore index overhead when operating on tail
Background task migrates rows from tail to columnstore in
chunks of 1 million rows not changed in last 1 hour
Columnstore data fully resident in memory
Persisted together with operational data
No application changes required
In-Memory OLTP table
Tail
Range index
Hash index
Performance

11. DEMO

12. © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on
the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.