Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS Online Tech Talks

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rahul Bhartia, Principal Product Manager, S3
March 29th 2018
Transforming Data Lakes
with
Amazon S3 Select & Amazon Glacier Select

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect from the session
1.Introduction
2.Use-cases
3.Key features

Unmatched durability,
availability, and scalability
Best security, compliance, and audit capability Object-level control
at any scale
Business insight into your
data
Twice as many partner integrationsMost ways to bring
data in
Building a data lake with Amazon S3

With a wide variety of in-place tools…
Amazon Athena Amazon Redshift
Spectrum
Amazon EMR AWS Glue
Amazon S3

Today: Compute scales based…
on object size instead of the amount of data you want to process
DATA COMPUTE

Today: All of these tools…
retrieve a lot of data they don’t need and
do the heavy lifting

You have a choice of storage classes
Archive data Infrequently accessed data
Minutes to hours Milliseconds
0.4¢-GB/mo. 1.25¢-GB/mo.
Amazon S3
Standard
Amazon S3
Standard– Infrequent Access
Amazon Glacier
Active data
Milliseconds
From 2.1¢-GB/mo.

Today: You need to….
entire object from Amazon Glacier to Amazon S3
and then use it.
Amazon S3Amazon
Glacier

Introducing…
Amazon S3 Select and Amazon Glacier Select
Select subset of data from an object based on a SQL expression

Simple, faster, and cheaper!
Available as an API—no
infrastructure or
administration
Faster performance as
compared to doing it yourself
Pay as you go. The less you
retrieve the more you save.

Amazon S3 Select

Amazon S3 Select
Simple to use
Standard SQL expression
Familiar
Work and scales like GET
requests
Integrated
AWS SDK and Presto
(others coming soon)
Select contents from object instead of retrieving the object

Amazon S3 Select
Output
Format: delimited text (CSV, TSV),
JSON …
Clauses Data types Operators Functions
Select String Conditional String
From Integer, Float, Decimal Math Cast
Where Timestamp Logical Math
Boolean String (Like, ||) Aggregate
Input
Format: delimited text (CSV, TSV),
JSON …
Compression: GZIP …

Amazon S3 Select: Simple pattern matches
…get-object …object… | awk -F ’{ if($4=="x") print $1}’
...select-object …object… ‘SELECT o._1 WHERE o._4 == “x”…’

Amazon S3 Select: Serverless applications
Amazon
S3
AWS
Lambda
Amazon
SNS
S3
Select

After
200 seconds and 11.2 cents
# Download and process all keys
for key in src_keys:
response = s3_client.get_object(Bucket=src_bucket,
Key=key)
contents = response['Body'].read()
for line in contents.split('n')[:-1]:
line_count +=1
try:
data = line.split(',')
srcIp = data[0][:8]
….
Amazon S3 Select: Serverless MapReduce
Before
95 seconds and costs 2.8 cents
# Select IP Address and Keys
for key in src_keys:
response = s3_client.select_object_content
(Bucket=src_bucket, Key=key, expression =
SELECT SUBSTR(obj._1, 1, 8), obj._2 FROM s3object as
obj)
contents = response['Body'].read()
for line in contents:
line_count +=1
try:
….
2X Faster at 1/5 of the cost

Up to 400% Faster
Up to 80% Cheaper
Amazon S3 Select: Accelerating Big Data
Amazon S3
Before:
Amazon S3
S3 Select
After:

DEMO: Amazon S3 Select with Presto
Works with your existing Hive Metastore
Automatically converts predicates into S3 Select requests
Amazon S3
S3 Select

Before
Amazon S3 Select: Accelerating big data
After
After
5X Faster with 1/40 of the CPU

Amazon S3 Select: Will be supported by…
Amazon Athena Amazon EMR Amazon Redshift
Spectrum

Amazon S3 Select: In Preview
• Formats: CSV, JSON
• Compression: GZIP
• Encryption: None
• Encoding: UTF-8
• Integration: AWS SDK for Java and Python and Presto Connector
• Availability: Northern Virginia, Ohio, Oregon, Dublin, and Singapore
Apply at: https://pages.awscloud.com/amazon-s3-select-preview.html

Amazon Glacier Select

Simple SQL Expression
SELECT and WHERE
Familiar semantics
Work and scales like
RESTORE requests
Integrated
AWS SDK and CLI
Restore selective contents instead of restoring entire object

Input
Format: delimited text (CSV, TSV,
PSV, etc.)
Encryption: SSE-KMS, SSE-S3
Output
Format: delimited text (CSV, TSV,
PSV, etc.)
Clauses Data types Operators Functions
Select String Conditional String
From Integer, Float, Decimal Math Cast
Where Timestamp Logical
Boolean String (Like, ||)

Two ways to use Amazon Glacier Select
Using Glacier API
Data directly uploaded to Amazon Glacier
How to use Amazon Glacier Select?
Using S3 API
For data that is lifecycled to Amazon Glacier
from S3

How to use Amazon Glacier Select?
Object Tier
SQL query Output S3 location SNS topic
Current restore-object API arguments
New (optional) restore-object API arguments to use Glacier Select

Using Glacier Select
…restore-object …object… | …get-object … object …. | awk -F ’{ if($4==“id") print $1}’
...restore-object …object… ‘SELECT o._1 WHERE o._4 == “id”…’ | …get-object … object ….

Amazon Glacier Select: GA
• Formats: CSV, Any delimiter separated file
• Encryption: SSE- KMS, SSE-S3
• Encoding: UTF-8
• Integration: AWS SDK, CLI, Athena integration (expected 2018)
• Availability: All commercial regions where Amazon Glacier is launched

Summary

Amazon S3 Select and Amazon Glacier Select
Select subset of data from an object based on a SQL expression

Now: Your data lake on AWS
Simple Faster Cheaper
Amazon
Glacier
Amazon
S3
Amazon Redshift
Spectrum
Amazon Athena Amazon EMR
AWS
Lambda
ISVs and Custom
Applications
SELECT

Thank you

Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS Online Tech Talks

More Related Content

Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS Online Tech Talks