Amazon Redshift offers a common query interface against  data stored in fast, local storage as well as data from high-capacity, inexpensive storage (S3).  This workshop will cover the basics of this tiered storage model and outline the design patterns you can leverage to get the most from large volumes of data.  You will build out your own Redshift cluster with multiple data sets to illustrate  the trade-offs between the storage systems.  By the time you leave, you’ll know how to distribute your data and design your DDL to deliver the best data warehouse for your business.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tiered Data Sets in Amazon Redshift
Matt Scaer
Principal DW Specialist SA
AWS
A N T 3 2 1
Karthik Odapally
Senior Solution Architect
AWS

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Introductions
Account Login and Cluster Spin-up
Refresher on Amazon Redshift
Workshop time
learning objectives
true-up against the website.

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why this session?
• Data typically grows at 10x every 5 years.
• Average lifetime for an Analytics Platform is 15yrs.
• Not just price and performance but also complexity.
Why you really need this session?

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Link https://bit.ly/2AkKaYX
[Estimated time to finish Setup: 20 min]

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Load
Unload
Backup
Restore
Massively parallel, shared nothing
columnar architecture
Leader node
• SQL endpoint
• Stores metadata
• Coordinates parallel SQL
processing
Compute nodes
• Local, columnar storage
• Executes queries in parallel
• Load, unload, backup, restore
Amazon Redshift Spectrum nodes
• Execute queries directly against
Amazon Simple Storage Service
(Amazon S3)
SQL clients/BI tools
128GB RAM
16TB disk
16 cores
JDBC/ODBC
128GB RAM
16TB disk
16 coresCompute
node
128GB RAM
16TB disk
16 coresCompute
node
128GB RAM
16TB disk
16 coresCompute
node
Leader
node
Amazon S3
...
1 2 3 4 N
Amazon
Redshift
Spectrum
Load
Query
Amazon Redshift architecture

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Redshift combines two usage patterns under a single, seamless service
Amazon Redshift (using direct-attached storage objects)
• Billed hourly for the number and type of compute nodes in the cluster
• An “all you can eat” model
Amazon Redshift Spectrum (table data resides on S3)
• Billed at $5 per TB of data scanned
• Both performance and cost-savings incent reducing the amount of data scanned through
• Compressing the data on S3
• Storing the data on S3 in a columnar format (for example, Parquet or ORC)
• Partitioning the data on S3
Amazon Redshift can be leveraged using the patterns either solely or in combination
Two complementary usage patterns

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Link

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #1: What happened in 2016?
1. Assemble your toolset:
1. Choosing a SQL editor (SQL Workbench, PGWeb, psql, query from Console, etc.)
2. (optional) Command Line Interface (CLI) for Amazon Redshift.
2. Load the Green company data for January 2016 into Redshift direct-
attached storage (DAS) with COPY.
3. Collect supporting/refuting evidence for the impact of the January,
2016 blizzard on taxi usage.
[Estimated time to finish this Scenario: 20 min]

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #1: Write you own SQL to pin-point the
blizzard (or use this)
0
10000
20000
30000
40000
50000
60000
70000
Number of Taxi Rides by Day for Green Taxi (January 2016)

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #2: Go back in time.
1. Query historical data residing on S3:
1. Create external DB for Redshift Spectrum.
2. Create the external table on Spectrum.
3. Write a script or SQL statement to add partitions.
4. Create and populate a small number of dimension tables on Redshift DAS.
2. Introspect the historical data, perhaps rolling-up the data in novel
ways to see trends over time, or other dimensions.
3. Enforce reasonable use of the cluster with Redshift Spectrum-specific
Query Monitoring Rules (QMR).
1. Test the QMR setup by writing an excessive-use query.
[Estimated time to finish this Scenario: 20 min]

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #2: [Advanced Topic] Debug a
Parquet/Redshift Spectrum datatype mismatch.
1. Create a new Redshift Spectrum table, changing the datatype of column ‘trip_distance’ from FLOAT8 to FLOAT4. Add a single partition for testing.
2. Counts still work, but what about other operations (SELECT MIN(trip_distance) FROM, SELECT * FROM, CTAS)?
3. Instead of considering Apache Drill or other tool to help resolve the issue, consider Redshift system view SVL_S3LOG (more hints at
https://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-troubleshooting.html#spectrum-troubleshooting-incompatible-data-format ).

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #3: Create a single version of truth.
1. Create a view that covers both the January, 2016 Green company DAS
table with the historical data residing on S3 to make a single table
exclusively for the Green data scientists:
1. Use CTAS to create a table with data from January, 2016 for the Green company. Compare
the runtime to populate this with the COPY runtime earlier.
2. Complete populating the January, 2016 table with an INSERT/SELECT statement for the
other taxi companies.
3. Create a new Spectrum table (or simply drop the January, 2016 partitions).
4. Create a view that’s allows for a seamless querying of the DAS and Spectrum data.
5. Examine the explain plan differences based on the filters used in a query.
2. Build a aggregate or roll-up table using the view to populate a Redshift
DAS table. ETL options here.
[Estimated time to finish this Scenario: 20 min]

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #4: Plan for the future.
1. Allow for trailing 5 quarters reporting by adding the Q4 2015 data to
Redshift DAS:
1. Anticipating the we’ll want to ”age-off” the oldest quarter on a 3 month basis, architect
your DAS table to make this easy to maintain and query.
2. Adjust your Redshift Spectrum table to exclude the Q4 2015 data.
2. Develop and execute a plan to move the Q4 2015 data to S3.
1. What are the discrete steps to be performed?
2. What extra-Redshift functionality must be leverage as of Monday, November 27, 2018?
3. Simulating the extra-Redshift steps with the existing Parquet data, age-off the Q4 2015
data from Redshift DAS and perform any needed steps to maintain a single version of the
truth.
[Estimated time to finish this Scenario: 20 min]

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario #4: Plan for the future.
1. Now, regardless of method, there’s a view covering the trailing 5 quarters in
Redshift DAS, and all of time on Redshift Spectrum, completely transparent to
users of the view.
2. What would be the steps to “age-off” the Q4 2015 data?

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Matt Scaer
mscaer@amazon.com
Karthik Odapally
karthio@amazon.com

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
[Estimated time to give us feedback: 1 min]