Download as PDF, PPTX
Uploaded byCobus Bernard
AWS SSA Webinar 20 - Getting Started with Data Warehouses on AWS
The document is a presentation on Amazon Redshift, focusing on its architecture, features, and data handling capabilities. It highlights the benefits of using a columnar data storage format for analytics, discusses data distribution methods, and outlines the process of querying data directly from Amazon S3. Additionally, the document covers the functionality of Amazon Redshift Spectrum for integrating data lakes with traditional data warehouses.
More Related Content
Similar to AWS SSA Webinar 20 - Getting Started with Data Warehouses on AWS
More from Cobus Bernard
AWS SSA Webinar 20 - Getting Started with Data Warehouses on AWS
- 1. Cobus Bernard Sr DeveloperAdvocate Amazon Web Services Getting Started with Data Warehouses on AWS @cobusbernard cobusbernard cobusbernard
- 2. Agenda Amazon Redshift Columnar Data Compression Queryexecution with Redshift Spectrum Demo Q&A
- 3. Load Unload Backup Restore Amazon Redshift architecture Massivelyparallel, shared nothing columnar architecture Leader node SQL endpoint Stores metadata Coordinates parallel SQL processing Compute nodes Local, columnar storage Executes queries in parallel Load, unload, backup, restore Amazon Redshift Spectrum nodes Execute queries directly against Amazon S3 SQL clients/BI tools JDBC/ODBC Compute node Compute node Compute node Leader node Amazon S3 ... 1 2 3 4 N Amazon Redshift Spectrum Load Query
- 4. Terminology and concepts:Node types Amazon Redshift analytics – RA3 Amazon Redshift managed storage – Solid state disks + Amazon S3 Dense compute – DC2 Solid state disks Dense storage – DS2 Magnetic disks Instance type Disk type Size Memory CPUs Slices RA3 4xlarge RMS Scales to 16 TB 96 GB 12 4 RA3 16xlarge RMS Scales to 64 TB 384 GB 48 16 DC2 large SSD 160 GB 16 GB 2 2 DC2 8xlarge SSD 2.56 TB 244 GB 32 16 DS2 xlarge Magnetic 2 TB 32 GB 4 2 DS2 8xlarge Magnetic 16 TB 244 GB 36 16
- 5. Terminology and concepts:Columnar Amazon Redshift uses a columnar architecture for storing data on disk Goal: reduce I/O for analytics queries Physically store data on disk by column rather than row Only read the column data that is required
- 6. Row-based storage • Needto read everything • Unnecessary I/O aid loc dt CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ); aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14 SELECT min(dt) FROM deep_dive; Columnar architecture: Example
- 7. Column-based storage • Onlyscan blocks for relevant column aid loc dt CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ); aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14 SELECT min(dt) FROM deep_dive; Columnar architecture: Example
- 8. Compression example Add 1of 13 different encodings to each column aid loc dt CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ); aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14
- 9. Compression example • Moreefficient compression is due to storing the same data type in the columnar architecture • Columns grow and shrink independently • Reduces storage requirements • Reduces I/O aid loc dt CREATE TABLE deep_dive ( aid INT ENCODE AZ64 ,loc CHAR(3) ENCODE BYTEDICT ,dt DATE ENCODE RUNLENGTH ); aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14
- 10. CREATE TABLE deep_dive( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) SORTKEY (dt, loc); Example: Sort key Add a sort key to one or more columns to physically sort the data on disk deep_dive aid loc dt 1 SFO 2017-10-20 2 JFK 2017-10-20 3 SFO 2017-04-01 4 JFK 2017-05-14 deep_dive (sorted) aid loc dt 3 SFO 2017-04-01 4 JFK 2017-05-14 2 JFK 2017-10-20 1 SFO 2017-10-20 deep_dive (sorted) aid loc dt 3 SFO 2017-04-01 4 JFK 2017-05-14 2 JFK 2017-10-20 deep_dive (sorted) aid loc dt 3 SFO 2017-04-01 4 JFK 2017-05-14 deep_dive (sorted) aid loc dt 3 SFO 2017-04-01
- 11. SELECT count(*) FROMdeep_dive WHERE dt = '06-09-2017'; MIN: 01-JUNE-2017 MAX: 06-JUNE-2017 MIN: 07-JUNE-2017 MAX: 12-JUNE-2017 MIN: 13-JUNE-2017 MAX: 21-JUNE-2017 MIN: 21-JUNE-2017 MAX: 30-JUNE-2017 Sorted by date MIN: 01-JUNE-2017 MAX: 20-JUNE-2017 MIN: 08-JUNE-2017 MAX: 30-JUNE-2017 MIN: 12-JUNE-2017 MAX: 20-JUNE-2017 MIN: 02-JUNE-2017 MAX: 25-JUNE-2017 Unsorted table Example: Zone maps and sorting
- 12. Data distribution Distribution styleis a table property that dictates how that table’s data is distributed throughout the cluster KEY: Value is hashed, same value goes to same location (slice) ALL: Full table data goes to the first slice of every node EVEN: Round robin AUTO: Combines EVEN and ALL Goals Distribute data evenly for parallel processing Minimize data movement during query processing KEY Node 1 Slice 1 Slice 2 Node 2 Slice 3 Slice 4 keyA keyB keyC keyD ALL Node 1 Slice 1 Slice 2 Node 2 Slice 3 Slice 4 EVEN Node 1 Slice 1 Slice 2 Node 2 Slice 3 Slice 4
- 13. Node 1 Data distributionexample Slice 0 Slice 1 Node 2 Slice 2 Slice 3 Table: deep_dive User columns System columns aid loc dt ins del row CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) (EVEN|KEY|ALL|AUTO);
- 14. Node 1 Slice 0Slice 1 Node 2 Slice 2 Slice 3 Data distribution, EVEN example INSERT INTO deep_dive VALUES (1, 'SFO', '2016-09-01'), (2, 'JFK', '2016-09-14'), (3, 'SFO', '2017-04-01'), (4, 'JFK', '2017-05-14'); Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 0 Rows: 0 Rows: 0 Rows: 0Rows: 1 Rows: 1 Rows: 1 Rows: 1 CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) DISTSTYLE EVEN;
- 15. Node 1 Slice 0Slice 1 Node 2 Slice 2 Slice 3 Data distribution, KEY example #1 INSERT INTO deep_dive VALUES (1, 'SFO', '2016-09-01'), (2, 'JFK', '2016-09-14'), (3, 'SFO', '2017-04-01'), (4, 'JFK', '2017-05-14'); Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 2 Rows: 0 Rows: 0Rows: 0Rows: 1 Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 2Rows: 0Rows: 1 CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) DISTSTYLE KEY DISTKEY (loc);
- 16. Node 1 Slice 0Slice 1 Node 2 Slice 2 Slice 3 Data distribution, KEY example #2 INSERT INTO deep_dive VALUES (1, 'SFO', '2016-09-01'), (2, 'JFK', '2016-09-14'), (3, 'SFO', '2017-04-01'), (4, 'JFK', '2017-05-14'); Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 0 Rows: 0 Rows: 0 Rows: 0Rows: 1 Rows: 1 Rows: 1 Rows: 1 CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) DISTSTYLE KEY DISTKEY (aid);
- 17. Node 1 Slice 0Slice 1 Node 2 Slice 2 Slice 3 Data distribution, ALL example INSERT INTO deep_dive VALUES (1, 'SFO', '2016-09-01'), (2, 'JFK', '2016-09-14'), (3, 'SFO', '2017-04-01'), (4, 'JFK', '2017-05-14'); Rows: 0 Rows: 0 Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 0Rows: 1Rows: 2Rows: 4Rows: 3 Table: deep_dive User Columns System Columns aid loc dt ins del row Rows: 0Rows: 1Rows: 2Rows: 4Rows: 3 CREATE TABLE deep_dive ( aid INT --audience_id ,loc CHAR(3) --location ,dt DATE --date ) DISTSTYLE ALL;
- 18. ... 1 2 34 N Amazon Redshift Spectrum Load Query Amazon Redshift architecture Massively parallel, shared nothing columnar architecture Leader node SQL endpoint Stores metadata Coordinates parallel SQL processing Compute nodes Local, columnar storage Executes queries in parallel Load, unload, backup, restore Amazon Redshift Spectrum nodes Execute queries directly against Amazon S3 SQL clients/BI tools JDBC/ODBC Compute node Compute node Compute node Leader node Amazon S3
- 19. Query SELECT COUNT(*) FROM S3.EXT_TABLE GROUPBY… Life Of A Query Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore 1
- 20. Life Of AQuery Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Query is optimised and compiled at the leader node. Determine what gets run locally and what goes to Amazon Redshift Spectrum 2
- 21. Life Of AQuery Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Query plan is sent to all compute nodes3
- 22. Life Of AQuery Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Compute nodes obtain partition info from Data Catalog; dynamically prune partitions4
- 23. Life Of AQuery Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Each compute node issues multiple requests to the Amazon Redshift Spectrum layer 5
- 24. Life Of AQuery Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Amazon Redshift Spectrum nodes scan your S3 data 6
- 25. Life Of AQuery Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore 7 Amazon Redshift Spectrum projects, filters, and aggregates
- 26. Life Of AQuery Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Final aggregations and joins with local Amazon Redshift tables done in-cluster 8
- 27. Life Of AQuery Amazon Redshift JDBC/ODBC ... 1 2 3 4 N Amazon S3 Exabyte-scale object storage Data Catalog Apache Hive Compatible Metastore Result is sent back to client9
- 28. Data warehouse (business data) Datalake (event data) Amazon Redshift Amazon Redshift enables you to have a lake house approach Customers moving to data lake architectures
- 29. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 30. Sources Targets Relational NoSQL AnalyticsData Warehouse* Amazon Aurora Amazon Aurora SAP ASE Amazon DynamoDB Amazon DocumentDB (with MongoDB compatibility) Db2 LUW SQL Azure SAP ASE MongoDB Cassandra Amazon Redshift Amazon S3 AWS SnowballAmazon S3 Amazon Elasticsearch Service Amazon Kinesis Data Streams Oracle SQL Server Netezza Greenplum Teradata Vertica * Supported via SCT data extractors Supported DMS source and targets
- 31.
- 32. Thank you! © 2020,Amazon Web Services, Inc. or its affiliates. All rights reserved. Cobus Bernard Sr Developer Advocate Amazon Web Services @cobusbernard cobusbernard cobusbernard