This document discusses options for large scale SQL analysis using Amazon Web Services. It describes the evolution of SQL and NoSQL databases, as well as options for SQL analytics on AWS including Amazon Redshift, Amazon Athena, and running SQL on Apache Spark via Amazon EMR. It provides information on tuning performance using different data formats, compression, and partitioning techniques. Finally, it shares the infrastructure evolution at Grab from using MySQL to Redshift to Presto on EMR and S3 for analytics and describes their current use of a data lake on S3 with analytics performed using Presto on EMR.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Russell Nash
Big Data Solutions Architect, Amazon Web Services
Level 300
SQL Strikes Back!
Options for Large Scale SQL Analysis

2. History of NoSQL
1970: NoSQL = We have no SQL
1980: NoSQL = We know SQL
2000: NoSQL = No SQL!
2005: NoSQL = Not only SQL
2017: NoSQL = No, SQL!
Mark Madsen

3. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
SQL Analytics
Amazon
Redshift
Amazon
Athena
SQL

4. Infrastructure
Data Layer
Process Layer
Framework
ApplicationsPIG

5. PIG
SQL
Infrastructure
Data Layer
Process Layer
Framework
Applications

6. PIG
SQL
Amazon
EMR

7. PIG
SQL
Amazon
EMR
Amazon
S3
EMRFS

9. Amazon
EMR
Amazon S3

10. S3/HDFS
Catalog
Users
CREATE EXTERNAL TABLE users (
Userid int,
Username string,
Firstname string,
Lastname string,
City string)
LOCATION "s3://data-bucket/users/";

11. S3/HDFS
Users
SELECT COUNT (*)
FROM users
WHERE city = ‘Sydney’ SQL
PIG
Catalog
Users

12. SQL

13. Amazon
EMR
SQL
Amazon S3

14. Spark
SQL
Spark
Streaming
Spark
ML
Spark R GraphX
Apache Spark Core

16. Amazon S3
PRESTO

17. Amazon
Athena
Query S3 data with SQL
Uses Presto
Serverless

18. Athena
S3
Catalogue
Table

19. MPP SQL Database
Optimised for Analytics
Fully relational
Amazon
Redshift

20. S3/HDFS
Amazon
Redshift
CREATE TABLE users (
Userid integer,
Username varchar(30),
Firstname varchar(30),
Lastname varchar(30),
City varchar(30));
COPY users FROM "s3://data-bucket/users/";

21. Amazon
Redshift
SELECT COUNT (*)
FROM users
WHERE city = “Sydney”

22. Comparison of SQL Processing engines
Amazon
Redshift
Amazon
Athena
Schema
Cost Model
On Read On Read
Per Hour Per Query
On Write
Per Hour
Data Store S3/HDFS S3 Local
SQL
On Read
Per Hour
S3/HDFS
Performance

23. Transformation
SQL Queries
For S3/HDFS
Fully Featured
SQL
Database
Use Case
Amazon
Redshift
Serverless
SQL Queries
for S3
SQL
Amazon
Athena
Comparison of SQL Processing engines

24. Tuning
Data Formats
Compression
Partitioning
Amazon Athena
SQL

25. CSV
JSON
AVRO
Parquet
ORC
ROW COLUMNAR

26. ID Age State
123 20 NSW
345 25 WA
678 40 VIC
999 21 WA
123 20 NSW 345 25 WA 678 40 VIC 999 21 WA
123 345 678 999 20 25 40 21 NSW WA VIC WA
ROW FORMAT
COLUMN FORMAT

27. gzip
Snappy
Zlib
LZO
bzip2
SPEED
SIZE
SPLITTABLE
DATA FORMAT

28. s3://mybucket/athena/inputdata/2016/data.csv
s3://mybucket/athena/inputdata/2015/data.csv
s3://mybucket/athena/inputdata/2014/data.csv
SELECT FIELDS from ACCESS_LOGS
WHERE YEAR = 2015
S3

29. CREATE EXTERNAL TABLE access_logs
(
ip_address String,
request_time Timestamp,
request_method String,
request_path String,
request_protocol String,
response_code String,
response_size String,
referrer_host String,
user_agent String
)
PARTITIONED BY (year STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY 't'
STORED AS TEXTFILE
LOCATION ‘s3://mybucket/athena/inputdata/’

30. s3://mybucket/athena/inputdata/2016/data.csv
s3://mybucket/athena/inputdata/2015/data.csv
s3://mybucket/athena/inputdata/2014/data.csv
SELECT FIELDS from ACCESS_LOGS
WHERE YEAR = 2015
S3

31. Demo

32. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Driving Southeast Asia
Forward with AWS
SQL Analytics @ Grab
MySQL => Redshift => Presto on EMR+S3

33. Infrastructure Evolution of Analytics@Grab
MySQL Replica
2013 / 2014
1 Database
~20 tables
< 50 reports
< 10 users
<1M records
1 Engineer
Redshift
2015/2016
20 Databases
100s tables
100s reports
< 500 users
> 100M records
3 Engineers
Presto+EMR+S3
Now
~20 Databases + streams
100s tables
> 500 reports
> 500 users
> 1B records
9 Engineers

34. Redshift for Analytics@Grab
Daily ETL After Midnight Redshift serves multiple use cases

35. Data Lake@Grab
Pyrois Orchestrator
Hourly ETL
Data Stored as Parquet
and Partitioned by Time
Helios Data Lake in S3

36. Analytics@Grab Today
Marketing Analytics
User Trust
Data Science

37. We’re just getting
started.

38. Thank you!