Customers using Amazon S3 at large scale benefit greatly from storage management features. Storage lifecycle policies help them reduce storage costs. Cross-region replication makes it easier to copy data between AWS regions for compliance or disaster recovery. Event notifications allow automatic initiation of processes on objects as they arrive, or capture information about objects and log it for security purposes. In this session, you'll learn about these features, and we'll also introduce several new storage management features like S3 Object Tagging, S3 Analytics Storage Class Analysis, S3 Inventory, S3 CloudWatch Metrics, and S3 CloudTrail integration that give users unmatched visibility into what data they are storing and how that data is being used. These new features make it simpler to analyze usage by users, apps, or organizations, to highlight anomalies, and to optimize business process workflows. They also help identify opportunities to reduce costs, improve performance, and archive infrequently used data. In addition, they can provide insight into who is accessing data stored in S3. Learning Objectives: • Understand the portfolio of storage management features available in S3 • Learn to manage cloud storage based upon what the data is (instead of where it's stored) using S3 Object Tagging • Understand the benefits of using S3 Inventory over the LIST API • Examine how S3 Analytics Storage Class Analysis helps to build lifecycle policies to optimize storage consumption • Learn about how new S3 integration with AWS CloudWatch and AWS CloudTrail help enhance performance and security

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sr. Product Manager, AWS
Deep Dive on S3 Storage
Management
Omair Gillani

2. Batches and Streams
AWS Direct
Connect
AWS Snowball,
Snowball Edge,
Snowmobile
3rd Party
Connectors
Transfer
Acceleration
AWS Storage
Gateway
Amazon Kinesis
Firehose
File BlockObject
Amazon GlacierAmazon S3
Amazon EC2
Instance Store
(ephemeral)
Amazon EFS Amazon EBS
(persistent)

3. AWS Direct Connect AWS Snowball ISV Connectors
Amazon Kinesis
Firehose
S3 Transfer
Acceleration
AWS Storage
Gateway
Data transfer into Amazon S3
AWS
Snowmobile
AWS Snowball Edge

4. Storage Gateway hybrid storage solutions
Enables using standard storage protocols to access AWS storage services
Customer Premises
Storage
Gateway
Amazon EBS
snapshots
Amazon
S3
Amazon Glacier
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
Enterprise
storage
Devices
Application
servers

5. What’s new? File gateway
File interface to objects in your Amazon S3 buckets
Virtual on-premises file server
Store and retrieve Amazon S3 objects through
standard file storage protocols (NFS v3 and v4.1)
Local cache for low-latency access
Direct access to objects with native S3 capabilities,
such as versioning, lifecycle management, and cross
region replication

6. What’s new? We’re reducing prices
Effective December 1, 2016
Storage prices reduced
Volume and virtual tape storage by up to 28%
Archived virtual tape storage by up to 43%
Flat per GB price replaces monthly gateway fee
$0.01 per GB written to AWS storage by your gateway*
Simplified virtual tape retrieval pricing
$0.01 per GB retrieved replaces peak-rate fee
* Up to $125/month per gateway

7. Cross-Region ReplicationLifecycle Policy S3 Object TagsEvent Notifications
Amazon S3 CloudWatch
Metrics
S3 Inventory Audit with AWS CloudTrail
S3 Data Events
S3 Analytics
Standard Standard - Infrequent Access Amazon Glacier
Storage Management for S3

8. Understand your storage usage
S3 InventoryAnalyze Logs with
Amazon EMR
S3 Analytics

9. S3 Inventory
Save time Daily or Weekly delivery Delivery to S3 bucketCSV File Output
Trigger business workflows and applications such as secondary index, garbage collection,
data auditing, and offline analytics
Half the price of LIST API at $0.0025 per million objects listed

10. S3 Inventory
More information about your objects than provided by LIST API such as replication
status, multipart upload flag, and delete marker
Name Value Type Description
Bucket String Bucket name. UTF-8 encoded.
Key String Object key name. UTF-8 encoded.
Version Id String Version Id of the object
Is Latest Boolean true if object is the latest version (current version) of a versioned object, otherwise false
Delete Marker Boolean true if object is a delete marker of a versioned object, otherwise false
Size Long Object size in bytes
Last Modified String Last modified timestamp. Format in ISO: YYYY-MM-DDTHH:mm:ss.SSSZ
ETag String eTag in HEX encoded format
StorageClass String
Valid values: STANDARD, REDUCED_REDUNDANCY, GLACIER, STANDARD_IA.
UTF-8 encoded.
Multipart Uploaded Boolean true if object is uploaded by using multipart, otherwise false
Replication Status String Valid values: REPLICA, COMPLETED, PENDING, FAILED. UTF-8 encoded.

12. S3 Inventory
Setup notification when S3 Inventory is complete
/Data
/<InventoryFile>.gz
/<InventoryFile>.gz
…
/<DayofReport>
/manifest.json
/manifest.checksum
…
AWS Lambda
Amazon SQS
Amazon SNS

13. Eventually consistent rolling snapshot
S3 Inventory
New objects may not be listed Recently deleted objects may still be included
O1
O2
O3
O1
O2
O3
O1
O2
O1
O2
O3
NEW
Validate before you act!
Use HEAD OBJECT or GET OBJECT

14. S3 Analytics – Storage Class Analysis
Analyze buckets,
prefixes or tags
$0.10 per million objects
analyzed per month
Daily Storage Class
Analysis & Lifecycle
candidates
Data-driven storage management for S3
Export Analysis data
to your S3 bucket

15. S3 Analytics – Storage Class Analysis

16. S3 Analytics – Storage Class Analysis

17. S3 Analytics – Storage Class Analysis
Export to use BI tool of your choice

18. S3 Analytics – Storage Class Analysis
Daily Export Dataset

19. Simple to configure S3 Analytics
S3 Management Console PUT Bucket Analytics
Multiple Policy
Documents
<AnalyticsConfiguration>
<Id>...</Id>
<Filter>
...
</Filter>
<StorageClassAnalysis>
<DataExport>
...
</DataExport>
</StorageClassAnalysis>
</AnalyticsConfiguration>

20. Monitor your storage
Monitor and Alert with
CloudWatch
Audit your storage with
CloudTrail Data Events
Server Access Logs

21. CloudWatch metrics for S3
Operational & performance monitoring
 Generate metrics for data of your choice
 Entire bucket, Prefixes, and Tags
 Up to 1,000 object groups
 1-minute CloudWatch metrics
 Alert and alarm on metrics

22. CloudWatch metrics for S3
Metric Name Metric value
AllRequests Count
PutRequests Count
PostRequests Count
GetRequests Count
ListRequests Count
DeleteRequests Count
HeadRequests Count
Metric Name Metric value
BytesDownloaded MB
BytesUploaded MB
4xxErrors Count
5xxErrors Count
FirstByteLatency ms
TotalRequestLatency ms
$0.50 per metric per month

23. S3 Data Events in CloudTrail
Perform security analysis, meet your IT auditing and compliance needs,
and take immediate action on object-level activity to immediately improve
security posture
Pricing: $1 per million data events recorded and storage charges apply
Log object level
operations
Changes to bucket
configurations
SNS notification for
log delivery

24. Manage your data
Cross-Region
Replication
Lifecycle Policies Event
Notifications
S3 Object Tags

25. Manage your data
S3 Object Tags
Easily manage and control access for Amazon S3 objects
 Classify your data
 Tag your objects with key-value pairs
 Write policies once based on the type of data
AnalyzeLifecycle PolicyAccess Control

26. Deep dive on tags
 Tags are key-value pairs
 Maximum 10 tags per object
 Maximum key length—127 Unicode characters
 Maximum value length—255 Unicode characters
 Tag keys and values are case-sensitive.
2 ways to put tags via API
 Put objects with tag parameter, or
 add tag API after object is created
Simple pricing
 $0.01 per 10,000 tags per month

27. Lifecycle policies based on tags
<LifecycleConfiguration>
<Rule>
<ID>sample-rule</ID>
<Filter>
<And>
<Prefix>documents/</Prefix>
<Tag>
<Key>Project</Key>
<Value>Delta</Value>
</Tag>
<Tag>
<Key>Data type</Key>
<Value>HPI</Value>
</Tag>
</And>
</Filter>
<Status>Enabled</Status>
<Transition>
<Days>365</Days>
<StorageClass>GLACIER</StorageClass>
</Transition>
<Expiration>
<Days>3650</Days>
</Expiration>
</Rule>
</LifecycleConfiguration>
 Transition or expire storage using tags
 Simplify S3 lifecycle policies
 Filter with prefix, tag, or both

28. Thank you!