This document summarizes optimization tips for Amazon Redshift data warehousing including: - Designing tables with optimal distribution keys, sort keys, and column sizes. - Efficiently loading and vacuuming data. - Tuning write-heavy and long-running queries using WLM queues. - Maintaining fresh table statistics and vacuuming tables regularly. The presentation provides demonstrations and questions are taken at the end.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
GPS: Optimizing Tips: Amazon
Redshift for Cloud Data
WarehousingM i k e K a l b e r e r , S o l u t i o n s A r c h i t e c t
A n u p a m M i s h r a , P r i n c i p a l S o l u t i o n s A r c h i t e c t
G P S T E C 3 1 5
N o v e m b e r 2 8 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Service overview of Amazon Redshift
Typical node architecture
Optimization tips
Demonstration
Question & Answer (approx. 15 minutes)

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customers Using Amazon Redshift

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service Overview

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift
• Fast, simple, secure, cost-effective data warehousing
• Petabyte-scale data warehousing
• Query your Amazon S3 “data lake”
• No up-front costs
Amazon
Redshift

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift Node Architecture
Leader node
• Optimizes
• Compiles
• Coordinates query execution with the
compute nodes
Compute nodes
• Parallel execution of queries
• Local, columnar storage

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift Node Architecture
• Each slice has a portion of the overall
node’s memory and disk
• The number of slices per node is
determined by the node size of the cluster

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Load Data Efficiently
Load data using COPY command
• Leverages all nodes in the cluster to load data in parallel
Compress the files whenever possible
• COPY command will automatically perform compression analysis
• Manually specify compression with help from ANALYZE COMPRESSION command
Splitting your data into multiple files
• File count should be multiple of the number of slices in cluster

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift
• Up-front table design is critical
• DISTKEY and SORTKEY influence performance
• PRIMARY KEY, FOREIGN KEY, UNIQUE constraints will
help (not enforced)
• Compress for faster speed and lower cost
Amazon
Redshift

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sort Keys
WHERE to use sort keys
• Create on columns which are most commonly used in WHERE clauses
COMPOUND sort key
• Good for known query patterns
• Time-series data
INTERLEAVED sort key
• Gives equal weight to each column in the sort key
• Large tables (> billion rows)

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Distribution Keys
• With an uneven
distribution, your
queries will complete
as fast as your slowest
slice

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Distribution Keys
Skewed distribution
• Consider changing distribution key to a column that exhibits high cardinality and
uniform distribution
• Run TABLE_INSPECTOR.SQL against tables to analyze data skew
• Try EVEN distribution if there is no good distribution key in record set
• If you create a new table with the same data as the USERS table but set
the DISTSTYLE to EVEN, rows are always evenly distributed across slices
• Use DISTYLE ALL for smaller tables
• Distributes table data to every node

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Table Column Size
Large VARCHAR columns
• Complex queries may need to store data in temporary tables
• Temporary tables are not compressed (consume extra memory and disk)
• Use the smallest possible column size for the use case

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Disk-Based Queries
Unnecessary I/O if queries need to write to disk—a performance hit when compared
to queries executed in memory
Identify queries writing to disk
• SELECT distinct query FROM svl_query_summary WHERE is_diskbased='t‘
Queue assignment rules
• Allow additional memory allocation for selected queue
WLM dynamic memory allocation
• Increase allocated memory to specific sessions

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Commit Usage
Single commit queue
• Commits are expensive
• Cause queries to wait
• Identify unnecessary transactions
• Group dependent statements
• Check commit queue with STL_COMMIT_STATS

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Commit Usage
• Check commit queue with STL_COMMIT_STATS
• Table only visible to superusers

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WLM Queue
• Two queues created by default
• Superuser
• Only use this queue when you need to run queries that affect the
system or for troubleshooting purposes
• Default user
• The default queue is initially configured to run five queries
concurrently; you can change the concurrency, timeout, and memory-
allocation properties for the default queue

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WLM Queue
• Identify short- and long-running queries and prioritize them
• Define multiple queues to route queries
• Use WLM_APEX_HOURLY.SQL to tune on peak concurrency
• https://github.com/awslabs/amazon-redshift-
utils/blob/master/src/AdminScripts/wlm_apex_hourly.sql

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WLM Queue Routing

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Table Maintenance
Table statistics missing or out of date
• Run ANALYZE weekly for all columns
• Run ANALYZE when loading popular columns
Vacuum tables
• As often as possible (weekly)
Use AWS AnalyzeVacuumUtility tool
• https://github.com/awslabs/amazon-redshift-utils/tree/master/src/AnalyzeVacuumUtility

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Redshift Optimization Demo
• WLM queue optimization
• Updating stale table statistics
• Amazon Redshift Spectrum

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Question & Answer
Mike Kalberer—mjk@amazon.com
Anupam Mishra—anupamm@amazon.com
https://github.com/awslabs/amazon-redshift-
utils/

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
G P S T E C 3 1 5