This document discusses modernizing a data warehouse with Amazon Redshift and Amazon Redshift Spectrum. It provides an overview of Amazon Redshift's massively parallel architecture and how Redshift Spectrum allows querying exabyte-scale data directly in Amazon S3. It also demonstrates the life of a query, showing how queries are optimized, executed across nodes, and retrieve results from both local Redshift tables and external data in S3 using standard SQL.

1. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Paul Macey
Solutions Architect – Big Data, Amazon Web Services
Modernise your Data Warehouse
with Amazon Redshift and Amazon Redshift
Spectrum

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why Modernise?
Performance Scalability Cost

3. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Introducing
Amazon Redshift

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Load
Unload
Query
Backup
Restore
Amazon Redshift Architecture
Massively parallel, shared nothing
columnar architecture
Leader node
• SQL endpoint
• Stores metadata
• Coordinates parallel SQL
processing
Compute nodes
• Local, columnar storage
• Executes queries in parallel
• Load, unload, backup, restore
Amazon Redshift Spectrum nodes
• Execute queries directly against
Amazon Simple Storage Service
(Amazon S3)
SQL Clients/BI Tools
128GB RAM
16TB disk
16 cores
JDBC/ODBC
128GB RAM
16TB disk
16 coresCompute
Node
128GB RAM
16TB disk
16 coresCompute
Node
128GB RAM
16TB disk
16 coresCompute
Node
Leader
Node
Amazon S3
...
1 2 3 4 N
Amazon
Redshift
Spectrum

5. Amazon Redshift + Spectrum
Performance at EB Scale
Fast Queries
Elastic and Highly Available
Elastic
On-demand, pay-per-query
Cost Effective
Multiple clusters access
same data
High Concurrency
Query data in-place using
open file formats
No ETL
Full Amazon Redshift SQL
Support
Standardised

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Query
SELECT COUNT(*)
FROM S3.EXT_TABLE
GROUP BY…
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
1

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Query is optimised and compiled at
the leader node. Determine what
gets run locally and what goes to
Amazon Redshift Spectrum
2

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Data Catalog
Apache Hive
Compatible Metastore
Query plan is sent to
all compute nodes
3

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Compute nodes obtain partition info
from Data Catalog; dynamically prune
partitions
4
Data Catalog
Apache Hive
Compatible Metastore

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Each compute node issues
multiple requests to the Amazon
Redshift Spectrum layer
5
Data Catalog
Apache Hive
Compatible Metastore

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Amazon Redshift Spectrum nodes
scan your S3 data
6
Data Catalog
Apache Hive
Compatible Metastore

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
7
Amazon Redshift
Spectrum projects,
filters, and
aggregates
Data Catalog
Apache Hive
Compatible Metastore

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Final aggregations and joins
with local Amazon Redshift
tables done in-cluster
8
Data Catalog
Apache Hive
Compatible Metastore

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life Of A Query
Amazon
Redshift
JDBC/ODBC
...
1 2 3 4 N
Amazon S3
Exabyte-scale object storage
Result is sent back to client9
Data Catalog
Apache Hive
Compatible Metastore

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Redshift Spectrum Is Fast
• Leverages Amazon Redshift’s advanced cost-based optimiser
• Pushes down projections, filters, aggregations and join reduction
• Dynamic partition pruning to minimise data processed
• Automatic parallelisation of query execution against S3 data
• Efficient join processing within the Amazon Redshift cluster

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Redshift Spectrum Is Cost-effective
• You pay for your Amazon Redshift cluster plus $5 per TB scanned from
S3
• Each query can leverage 1000s of Amazon Redshift Spectrum nodes
• You can reduce the TB scanned and improve query performance by:
• Partitioning data
• Using a columnar file format
• Compressing data

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Redshift Spectrum Uses Standard SQL
• Spectrum seamlessly integrates with your existing SQL & BI apps
• Support for complex joins, nested queries & window functions
• Support for data partitioned in S3 by any key
• Date, Time and any other custom keys
• e.g., Year, Month, Day, Hour

18. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo: Modern Data Architecture
Amazon Redshift + Spectrum

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo Architecture
Amazon Redshift
+ Spectrum
Amazon S3
• Database Tables & Views
• Traditional Star Schema
• Tickit Sample Database
• Data for the past 5 years
• S3 Bucket
• Multiple Folders (1 for each Tickit table)
• Multiple data files
• Data for the past 25 years

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why Modernise?
Performance Scalability Cost

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Next Steps
• Amazon Redshift Spectrum Getting
Started Guide
• Amazon Redshift Documentation
• And much, much more!
Tap your badge for additional resources:

22. Thank You