AI&BigData Conference 2017 Sergiy Lunyakin "Azure SQL DWH: Tips and Tricks for developers"

1. Azure SQL DWH: Tips and Tricks
for developers
Sergiy Lunyakin

2. About me

3. Agenda

4. Scale up
vs Scale Out

5. One bucket (motherboard)
Contains all the water (resources)
Drinking through straws (logical procs)
Sometimes you only get one straw…
Scaling up
SMP = Scaling UP

6. Scaling out: The ultimate team game…
01
02
03
04
MPP
=
Scaling
OUT

7. Architecture

8. What is Azure SQL DW

9. Architecture of Azure SQL DW
Dist_DB_1
Dist_DB_2
Dist_DB_15
…
Dist_DB_16
Dist_DB_17
Dist_DB_30
…
…
…
…
Dist_DB_46
Dist_DB_47
Dist_DB_60
…

10. Control
ComputeStorage
Logical Overview

11. Sizing

12. Sizing factors

13. Introducing DWU
ALTER DATABASE ContosoRetailDW
MODIFY
(service_objective = 'DW1000')
;
DWU

14. Workloads

15. Analytical workloads
• Store large volumes of data. Supports a maximum
compressed size 240TB, that potentially is a
Petabyte uncompressed data.
• Consolidate disparate data into a single location
• Shape, model, transform and aggregate data
• Perform batch processing query across large
datasets
• Ad-hoc reporting across large data volume

16. Unsuitable workloads
OLTP workloads
• High frequency reads&writes
• Large numbers of singleton selects
• High volumes of single row inserts
Procedural ETL
• Row by row processing needs
• Incompatible formats (JSON,XML)

17. Compute Consumption

18. Azure Data Lake & SQL DW

19. Source for SSAS
Polybase
Azure SQL database
Web
Data Factory
Storage blob
SSAS

20. Distributions

21. Distributions
* Selecting the right distribution method is key for good performance

22. Round Robin Distribution
1
3
1
4
1
6
1
5
1
7
1
8
2
0
1
9
2
1
2
2
2
4
2
3
2
5
2
6
2
8
2
7
2
9
3
0
3
2
3
1
3
3
3
4
3
6
3
5
3
7
3
8
4
0
3
9
4
1
4
2
4
4
4
3
4
5
4
6
4
8
4
7
4
9
5
0
5
2
5
1
5
3
5
4
5
6
5
5
5
7
5
8
6
0
5
9
0
1
0
2
0
4
0
3
0
5
0
6
0
8
0
7
0
9
1
0
1
2
1
1

23. HASH Distribution
1
3
1
4
1
6
1
5
1
7
1
8
2
0
1
9
2
1
2
2
2
4
2
3
2
5
2
6
2
8
2
7
2
9
3
0
3
2
3
1
3
3
3
4
3
6
3
5
3
7
3
8
4
0
3
9
4
1
4
2
4
4
4
3
4
5
4
6
4
8
4
7
4
9
5
0
5
2
5
1
5
3
5
4
5
6
5
5
5
7
5
8
6
0
5
9
0
1
0
2
0
4
0
3
0
5
0
6
0
8
0
7
0
9
1
0
1
2
1
1
HASH ( )N01020301

24. Joining HASH tables

25. Joining HASH tables

26. Hash distribution key guidance
• Distribution key is not updateable!

27. Node 00
Node 01
Node 02
Node 03
Node 04
Node 05
Node 06
PRODUCTPRODUCTPRODUCTPRODUCTPRODUCTPRODUCTSALESPRODUCT
Replicated (vs. hash distributed)

28. Skewed Distribution
01
02
03
04

29. Distribution compatibility
HASH
ROUND ROBIN
REPLICATED

30. Control
ComputeStorage
Distributed queries

31. Distributed Query
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales]
;
SELECT SUM(*)
FROM dbo.[FactInternetSales]
;
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales]
;
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales]
;
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales]
;
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales]
;
Compute
Control

32. Data Loading

33. Options
Parallel
PolyBase
Azure Data Factory (PolyBase)
SSIS (Azure SQL DW Comp)
Single Gated Client
bcp / Insert Bulk
SQLBulkCopy
SSIS (data flow)
Azure Data Factory

34. Single Gated Client
DMS
DMS
DMS
DMS

35. Single Gated Client Parallelised
DMS
DMS
DMS
DMS

36. Parallel Loading with PolyBase
DMS
DMS
DMS
DMS

37. Sizing for the data load
DWU Readers Writers
Compressed text limits
concurrent access to
text files
Split data across files
OR
Use different file format

38. Dedicate user for data loading
Post Provisioning

39. Limitations

40. Limitations
• Primary/Foreign Keys
• Identity
• Computed Columns
• Triggers
• Cross-database joins
• Sequences
• Cursors
• MERGE
• ANSI joins on updates/deletes
https://feedback.azure.com/forums/307516-
sql-data-warehouse
More limitations:
https://azure.microsoft.com/en-
us/documentation/articles/sql-data-warehouse-
migrate-code/

41. Check compatibility
• Data warehouse migration utility
• Free tool
• Helps to identify unsupported features
• Helps to identify HASH distribution column
• Migrate scheama
• Migrate data (BCP tool)

42. Cross-database query
• Azure SQL DW doesn’t support cross-database query.
• Use ELT approach. Separate schemas.
• Use External tables as staging tables.

43. CTAS
• CTAS is super-charched version of SELECT...INTO
• Parallelized
• Better for
 Data import
 Data copy
 Workarounds
CREATE TABLE
[dbo].[FactInternetSales_new]
WITH
(
DISTRIBUTION = ROUND_ROBIN
, CLUSTERED COLUMNSTORE INDEX
)
AS
SELECT *
FROM
[dbo].[FactInternetSales];

44. Identity
• Handle it on source side
• IDENTITY property
 Explicit import
 Doesn’t support CTAS
• Custom Identity with ROW_NUMBER

45. ANSI JOINS Update/Del/Merge
• Update/Delete doesn’t support JOINS in FROM
• Use CTAS for preparing interim table with JOINS
• Use CTAS for Merge workaround
 Split Merge to operation steps and use UNION ALL
 Use interim table for big number of steps
 Use partitioning for big tables, don’t reload the
whole table

46. Compute columns
• Handle it in a source system
• Use CTAS during import
• Create a View
• Use explicit data type and nullability check in you
calculation expressions
 Wrong data during migration
 Schema error during partition switch

47. Cursor
• Use WHILE for lopping
 Prepare a list of elements as a table
 Loop through this list using While clause and
variable
 Do some action

48. Questions?