Come learn about new and existing Amazon S3 features that can help you better protect your data, save on cost, and improve usability, security, and performance. We will cover a wide variety of Amazon S3 features and go into depth on several newer features with configuration and code snippets, so you can apply the learnings on your object storage workloads.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Omair Gillani, Sr. Product Manager
Hisham Baz, Solutions Architect
October 2015
Amazon S3: Deep Dive and
Best Practices
STG 401

2. New for 2015…
Cross-region
replication
Amazon S3 Standard-IA
AWS CloudTrail support
for Amazon S3
Amazon CloudWatch
metrics for Amazon S3
VPC endpoint
for Amazon S3
Amazon S3 bucket
limit increase
Event notifications
Read-after-write
consistency in all regions

3. Video sharing service
VidShare

4. Video sharing service
VidShare
Launch
V2 Optimize on cost
V3 Expand globally
V4 Enterprise enablement

5. Video sharing service
VidShare

6. - Thumbnail
- Update Index
- Update WebApp
Video sharing service
Event
MetadataThumbnail
logs
VidShare

7. Amazon S3 event notifications
Events
SNS topic
SQS
queue
Lambda
function
• Notification when objects are
created via PUT, POST, Copy, or
Multipart Upload, DELETE
• Filtering on prefixes and suffixes
for all types of notifications
Fast IntegratedSimple

8. Request specific notifications
Request notifications on specific
PUT APIs
Request notifications on specific
DELETE APIs
s3:ObjectCreated:*
s3:ObjectCreated:Put
s3:ObjectCreated:Post
s3:ObjectCreated:Copy
s3:ObjectCreated:CompleteMultipartUpload
s3:ObjectRemoved:*
s3:ObjectRemoved:Delete
s3:ObjectRemoved:DeleteMarkerCreated

9. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hisham Baz
Solutions Architect,
Amazon
DEMO
VidShare

10. Empty bucket in Oregon

11. Create uploadVideo handler

12. VidShare v1.0

13. Share a video

14. Put fires, metadata parsed, thumbnail generated

15. Amazon DynamoDB metadata index updated

16. Setup delete notification – AWS CLI

17. Setup delete notifications – config and execute

18. Delete files directly from S3 bucket

19. Delete notification fires, app updated

20. Thumb deleted and metadata index updated

21. Launch
V2 Optimize on cost
V3 Expand globally
V4 Enterprise enablement
Optimizing VidShare
VidShare
T T+3 days T+5 days T+ 15 days T + 25 days T + 30 days T + 60 days T + 90 days T + 150 days T + 250 days T + 365 days
Access
Frequency

22. Choice of storage classes on Amazon S3
Standard
Active data Archive dataInfrequently accessed data
Standard - Infrequent Access Amazon Glacier

23. 11 9’s of Durability
Standard-Infrequent Access storage
Infrequently accessed data
Designed for
99.9% availability
Durable Available
Same throughput as
Amazon S3 Standard storage
High performance
• Server-side encryption
• Use your encryption keys
• KMS managed encryption keys
Secure
• Lifecycle management
• Versioning
• Event notifications
• Metrics
Integrated
• No impact on user
experience
• Simple REST API
• Single bucket
Easy to use

24. - Transition Standard to Standard-IA
- Transition Standard-IA to Amazon Glacier
storage
- Expiration lifecycle policy
- Versioning support
Standard-Infrequent Access storage
Integrated with lifecycle
Integrated: Lifecycle management

25. Save money on VidShare
Lifecycle policy
Standard Storage -> Standard-IA
<LifecycleConfiguration>
<Rule>
<ID>sample-rule</ID>
<Prefix>documents/</Prefix>
<Status>Enabled</Status>
<Transition>
<Days>30</Days>
<StorageClass>STANDARD-IA</StorageClass>
</Transition>
<Transition>
<Days>365</Days>
<StorageClass>GLACIER</StorageClass>
</Transition>
</Rule>
</LifecycleConfiguration>

26. Save money on VidShare
Lifecycle Policy
Standard Storage -> Standard-IA
<LifecycleConfiguration>
<Rule>
<ID>sample-rule</ID>
<Prefix>documents/</Prefix>
<Status>Enabled</Status>
<Transition>
<Days>30</Days>
<StorageClass>STANDARD-IA</StorageClass>
</Transition>
<Transition>
<Days>365</Days>
<StorageClass>GLACIER</StorageClass>
</Transition>
</Rule>
</LifecycleConfiguration>
Standard-IA Storage -> Amazon Glacier

27. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hisham Baz
Solutions Architect,
Amazon
DEMO
VidShare

28. Setup lifecycle policy

29. Transition older videos to Standard-IA

30. Save money on VidShare
39%
* Assumes the highest public pricing tier

31. VidShare is global!
VidShare
Launch
V2 Optimize on cost
V3 Expand globally
V4 Enterprise enablement

32. Remote replicas managed
by separate AWS accounts
Secure
Distribute data to regional
customers
Lower Latency
Store hundreds of
miles apart
Compliance
Amazon S3 cross-region replication
Automated, fast, and reliable asynchronous replication of data across AWS regions

33. • Usual charges for
storage, requests, and
inter-region data transfer
for the replicated copy of
data
• Replicate into Standard-IA
or Amazon Glacier
Cost
HEAD operation on a source
object to determine replication
status
• Replicated objects will not be
re-replicated
• Use Amazon S3 COPY to
replicate existing objects
Replication status
DELETE without object
version ID
• Marker replicated
DELETE specific object
version ID
• Marker NOT replicated
Delete operation
Cross-region replication: Details
Object ACL updates are
replicated
• Objects with Amazon
managed encryption key
replicated
• KMS encryption not
replicated
Access control

34. Versioning with cross-region replication
A
B
Vid1- v2
Vid1- v1
Key: A/vid1 Key: B/vid1
Vid1- v2
Vid1- v1
Vid1- v4
Vid1- v3

35. Versioning with cross-region replication
A
B
Vid1- v2
Vid1- v1
Key: A/vid1 Key: B/vid1
Vid1- v2
Vid1- v1
Vid1- v4
Vid1- v3
Vid1- v3Vid1- v4

36. Replicate VidShare videos to Japan
PUT /?replication HTTP/1.1
Host: examplebucket.s3.amazonaws.com
x-amz-date: Wed, 11 Feb 2015 02:11:21 GMT
Content-MD5: q6yJDlIkcBaGGfb3QLY69A== Authorization:
authorization string
Content-Length: 406
<ReplicationConfiguration>
<Role>arn:aws:iam::35667example:role/CrossRegionReplicationRoleFo
rS3</Role>
<Rule>
<ID>rule1</ID>
<Prefix>vid/</Prefix>
<Status>Enabled</Status>
<Destination>
<Bucket>arn:aws:s3:::vidsharebucketjapan</Bucket>
</Destination>
</Rule>
</ReplicationConfiguration>
Setting up cross-region replication policy (same AWS account)
1. Enable versioning on both
buckets
2. Add Replication Configuration
3. Validate replication

37. Replicate VidShare videos to Japan
1. Add bucket policy on the destination bucket to allow the source bucket
owner permission for replication actions
2. Create an IAM role in AWS Account A.
3. Specify IAM role when adding replication configuration on the source
bucket
4. Enable versioning on both buckets
5. Add Replication Configuration on source bucket AWS Account A
Setting up cross-region replication policy (different AWS accounts)

38. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hisham Baz
Solutions Architect,
Amazon
DEMO
VidShare

39. VidShare v3.0 – Global expansion

40. Setup replication using AWS CLI

41. Setup replication – Execute via AWS CLI

42. Share video clip

43. Put fires, replication begins

44. File in bucket in Oregon

45. View replication status

46. View video replicated

47. Delete the video, and watch replication

48. Fully replicated delete

49. VidShare for enterprises!
Virtual
Private Cloud (Amazon VPC)
VidShare
Launch
V2 Optimize on cost
V3 Expand globally
V4 Enterprise enablement

50. Using Amazon S3 with VPC endpoints – Previously…
mybucket
Internet
PUT S3
PUTS3
NAT
Internet
Gateway
• Public IP on EC2 instances
and IGW
• Private IP on EC2
instances and NAT
Amazon S3 VPC endpoints
Access Amazon S3 from your Amazon VPC using VPC endpoints

51. VPC
Endpoint
PUT S3
VPC Policy
mybucket
Internet
Amazon S3 VPC endpoints
Access Amazon S3 from your Amazon VPC using VPC endpoints
Using Amazon S3 VPC endpoints

52. Improved throughput
from VPC resources to
Amazon S3
High availability
High performance
High availability
Reduce cost by
avoiding expensive
NAT, Internet
gateways
Lower cost
Simple to setup, no
need to manage NATs
and Internet gateways
Simple
Improved security, no
need to route traffic
through the internet
Secure
Amazon S3 VPC endpoints
Access Amazon S3 from your Amazon VPC using VPC endpoints

53. Using Amazon S3 VPC endpoints
Get VPC
Policy
VPC
Endpoint
VPC Policy
mybucket
Amazon S3 VPC endpoints
Access Amazon S3 from your Amazon VPC using VPC endpoints
PUT S3

54. VPC
Endpoint
VPC Policy
mybucket
Evaluate
VPC Policy
Amazon S3 VPC endpoints
Access Amazon S3 from your Amazon VPC using VPC endpoints
Using Amazon S3 VPC endpoints

55. VPC
Endpoint
VPC Policy
mybucket
Evaluate S3
BucketACL
Policy
Amazon S3 VPC endpoints
Access Amazon S3 from your Amazon VPC using VPC endpoints
Using Amazon S3 VPC endpoints

56. VPC
Endpoint
VPC Policy
mybucket
PUT S3
PUTS3
Amazon S3 VPC endpoints
Access Amazon S3 from your Amazon VPC using VPC endpoints
Using Amazon S3 VPC endpoints

57. Amazon S3 VPC endpoints
Using Amazon S3 VPC endpoints
• Control access to buckets from specific Amazon
VPC endpoints, or specific VPCs
• Control which VPCs or VPC endpoints have
access to your S3 buckets by using S3 bucket
policies

58. Amazon S3 VPC endpoints
Using Amazon S3
VPC endpoints
Amazon VPC policy to restricts access to a
specific bucket
{
"Statement": [
{
"Sid": "Access-to-specific-bucket-only",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:PutObject" ],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::my_secure_bucket",
"arn:aws:s3:::my_secure_bucket/*"]
}
]
}

59. Amazon S3 bucket policy to allow a specific
VPC endpoint access to my S3 bucket
{
"Version": "2012-10-17",
"Id": "Policy1415115909152",
"Statement": [
{
"Sid": "Access-to-specific-VPCE-only",
"Principal": "*",
"Action": "s3:*",
"Effect": "Deny",
"Resource": ["arn:aws:s3:::my_secure_bucket",
"arn:aws:s3:::my_secure_bucket/*"],
"Condition":
{
"StringNotEquals": {
"aws:sourceVpce": "vpce-1a2b3c4d" }
}
} ] }
Amazon S3 VPC endpoints
Using Amazon S3
VPC endpoints

60. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hisham Baz
Solutions Architect,
Amazon
DEMO
VidShare

61. Start in Amazon VPC console

62. Create a new VPC endpoint

63. Configure route tables

64. View routes

65. View bucket policy

66. Apply bucket policy

67. Play recorded video in Amazon WorkSpaces

68. Launch Amazon WorkSpaces

69. Watch video from Amazon WorkSpaces

70. Audit logs Amazon S3
Demonstrate compliance, improve security
Log Amazon S3 API
using AWS CloudTrail
Track bucket-level operations
• Creation and deletion of buckets
• Changes to access control, lifecycle policy, cross
region replications policy etc.
Integrated with Amazon CloudWatch
• Alarm if a specific API called
Configure once per AWS Account
• Track multiple services with AWS CloudTrail

71. Amazon S3 storage metrics
Understand your Amazon S3 buckets
Amazon CloudWatch metrics for Amazon S3
Bucket-level metrics include:
• Total bytes for Standard storage
• Total bytes for Standard-IA storage
• Total bytes for Reduced-Redundancy storage
• Total number of objects for a given S3 bucket
Alarm on S3 metrics
• Set thresholds for alarms
Daily metrics
• Metrics emitted daily, after midnight GMT

72. Thank you!

73. Remember to complete
your evaluations!