Your SlideShare is downloading. ×
Masterclass
Amazon S3 – beyond simple storage
Ian Massingham – Technical Evangelist
@IanMmmm
A technical deep dive beyond the basics
Help educate you on how to get the best from AWS technologies
Show you how things ...
It’s more than just a ‘simple’ storage platform
A sophisticated 21st century distributed system
A bedrock architectural co...
“Spotify needed a storage solution that
could scale very quickly without
incurring long lead times for upgrades.
This led ...
Minecraft Realms and AWS S3
Minecraft worlds and game state
are stored in Amazon S3. The
system takes advantage of S3
vers...
You put it in S3
AWS stores with 99.999999999% durability
You put it in S3
AWS stores with 99.999999999% durability
Highly scalable web
access to objects
Multiple redundant
copies ...
but it’s more than just a
simple storage service
Objects in S3
Trillions of Objects
(000,000,000,000s)
Servicing over 2 million
requests per Second
What is S3?
Highly scalable data storage
Access via APIsA web store, not a file system
Fast
Highly available & durable
Eco...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
Region
Availabil...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
Region
Availabil...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
Region
Availabil...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
Region
Availabil...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
Region
Availabil...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
Region
Availabil...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
Region
Availabil...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
Region
Availabil...
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
Conceptual diagram only
A web store, not a file system
Write once,
read many
(WORM)
Eventually
consistent
*except US-STANDARD region
New objects
U...
A regional service
Your data never leaves a region unless you move it
Storage classes
Controlling the way S3 holds your data
Standard
Designed to provide
99.999999999% durability
and 99.99% availability of
objects over a given year
Designed to sus...
Reduced
Redundancy
Storage
Reduces costs by storing data
at lower levels of redundancy
than the Standard storage
Designed ...
Reduced
Redundancy
Storage
Reduces costs by storing data
at lower levels of redundancy
than the Standard storage
Designed ...
Amazon S3 storage classes
Glacier
Objects you want to
keep in archive for a
long time
e.g. digital archive of old movies
&...
Namespaces
Object naming, buckets and keys
Amazon S3 namespace
Globally unique
bucket name + object name (key)
Amazon S3 namespace
Amazon S3
bucket bucket
object object objectobject
bucket
object object
Amazon S3 namespace
Amazon S3
jones-docshare media.mydomain.com
beach.jpg img1.jpg style.cssdrafts/rpt.doc
yourdomain.com
...
Amazon S3 namespace
Object key
Unique with a bucket
Amazon S3 namespace
Object key
Unique with a bucket
Max 1024 bytes UTF-8 Including ‘path’ prefixes
Amazon S3 namespace
Object key
Unique with a bucket
Max 1024 bytes UTF-8 Including ‘path’ prefixes
assets/js/jquery/plugin...
Throughput optimisation
S3 automatically partitions based upon key prefix:
2134857/gamedata/start.png
2134857/gamedata/res...
Throughput optimisation
S3 automatically partitions based upon key prefix:
2134857/gamedata/start.png
2134857/gamedata/res...
Throughput optimisation
S3 automatically partitions based upon key prefix:
2134857/gamedata/start.png
2134857/gamedata/res...
Throughput optimisation
S3 automatically partitions based upon key prefix:
7584312/gamedata/start.png
7584312/gamedata/res...
Throughput optimisation
S3 automatically partitions based upon key prefix:
7584312/gamedata/start.png
7584312/gamedata/res...
Encryption
Securing data at rest
Server side encryption
Automatic encryption
of data at rest
Durable
S3 key storage
Simple
Additional PUT
header
Strong
AES...
Server side encryption
Data bucket
High level design
Server side encryption
Data bucket Encrypted data
Per-object key
Encrypted object
High level design
Server side encryption
Data bucket Encrypted data
Encrypted per-
object key
Per-object key
Encrypted object
Master key
Hig...
Server side encryption
Data bucket Encrypted data
Encrypted per-
object key
Key management
(monthly rotation)
Per-object k...
docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
Access controls
You’re in control of who does what
Secure by default
You decide what to share
Apply policies to buckets and objects
Policies, ACLs & IAM
Use S3 policies, ACL...
Fine grained
Administer as part of
role based access
Apply policies to S3 at
role, user & group level
PutObject
arn:aws:s3...
Bucket PoliciesIAM VS
Fine grained
Administer as part of
role based access
Apply policies to S3 at
role, user & group leve...
Bucket Policies ACLsVSIAM VS
Fine grained
Administer as part of
role based access
Apply policies to S3 at
role, user & gro...
{"Statement":[{
"Effect":"Allow",
"Principal":{"AWS":["4649-6425", "5243-0045"]},
"Action":"*",
"Resource":"/mybucket/*",
...
{"Statement":[{
"Effect":"Allow",
"Principal":{"AWS":["4649-6425", "5243-0045"]},
"Action":"*",
"Resource":"/mybucket/*",
...
{"Statement":[{
"Effect":"Allow",
"Principal":{"AWS":["4649-6425", "5243-0045"]},
"Action":"*",
"Resource":"/mybucket/*",
...
{"Statement":[{
"Effect":"Allow",
"Principal":{"AWS":["4649-6425", "5243-0045"]},
"Action":"*",
"Resource":"/mybucket/*",
...
Transitions & Lifecycle
Management
Automated management of objects
Lifecycle management
Lifecycle management
Object deletion
Permanently delete objects from S3
Lifecycle management
Object deletion
Permanently delete objects from S3
Object archiving
Move objects to Glacier and out o...
Glacier
Long term durable archive
Long term Glacier archive
Durable
Designed for 99.999999999%
durability of archives
Cost effective
Write-once, read-never....
Logs
accessible from S3
time
Expiry
Logs
✗accessible from S3
Objects expire
and are
deleted
time
Expiry
Logs
Txns
✗accessible from S3
Objects expire
and are
deleted
time
accessible from S3
Object transition
to Glacier
invoked
...
Logs
Txns
✗accessible from S3
Objects expire
and are
deleted
time
accessible from S3
Object transition
to Glacier
invoked
...
Logs
Txns
✗accessible from S3
Objects expire
and are
deleted
time
accessible from S3
Object transition
to Glacier
invoked
...
3-5 hour retrieval time
We assume you won’t access often
using (client = new AmazonS3Client()){
var lifeCycleConfiguration = new LifecycleConfiguration()
{
Rules = new List<Lifecy...
using (client = new AmazonS3Client()){
var lifeCycleConfiguration = new LifecycleConfiguration()
{
Rules = new List<Lifecy...
using (client = new AmazonS3Client()){
var lifeCycleConfiguration = new LifecycleConfiguration()
{
Rules = new List<Lifecy...
POST /ObjectName?restore HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Authorization: signatureValue
Content-MD5: ...
POST /ObjectName?restore HTTP/1.1
Host: BucketName.s3.amazonaws.com
Date: date
Authorization: signatureValue
Content-MD5: ...
Website hosting
Static sites straight from S3
It’s a web service…
…so serve up web content
Setting default documents
Redirecting requests
{
"Version":"2008-10-17",
"Statement":[{
"Sid":"PublicReadGetObject",
"Effect":"Allow",
"Principal": {
"AWS": "*"
},
"Acti...
{bucket-name}.s3-website-{region}.amazonaws.com
e.g. mybucket.s3-website-eu-west-1.amazonaws.com
s3-{region}.amazonaws.com...
R53
Record set for:
aws-exampl.es
Website bucket name:
bucket
Record set for:
Website bucket name:
bucket
Error.
html
Index
.html
R53
www.aws-exampl.es aws-...
Website bucket name:
bucket
Record set for:
Website bucket name:
bucket
Error.
html
Index
.html
R53
www.aws-exampl.es aws-...
Website bucket name:
bucket
Record set for:
Website bucket name:
bucket
Error.
html
Index
.html
R53
www.aws-exampl.es aws-...
Website bucket name:
bucket
Record set for:
Website bucket name:
bucket
Error.
html
Index
.html
R53
www.aws-exampl.es aws-...
docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html
Time-Limited URLS
Control how long objects can be accessed for
Signed URLs
Provide time-limited access to
specific objects that expires after
a set period
Access Permissions
Use on obje...
>>>
>>> import boto
>>> conn = boto.connect_s3()
>>> conn.generate_url(300, 'GET', bucket='ianmas-aws.testbucket',
key='testfi...
Error response: link
expired
AccessDenied
Request has expired
70297390BE427DC7
2014-02-03T11:03:58Z
I0rI0OWUCnBttFSpEw6Mx4...
Object versioning
Preserving object histories
Persistent
Even deleted object
history is held
Bucket level
Automatically preserves
all copies of objects
Persistent
Even deleted object
history is held
Bucket level
Automatically preserves
all copies of objects
Persistent
Even deleted object
history is held
Bucket level
Automatically preserves
all copies of objects
>>>
>>> import boto
>>> conn = boto.connect_s3()
>>> bucket = conn.get_bucket(’mybucket')
>>> versions = bucket.list_versions(...
>>> key = bucket.get_key('myfile.txt',
version_id='8cjozv9Hmkzum8xj.8q8BZxR5CuXnzon’)
>>> key.get_contents_as_string()
'th...
>>> key = bucket.get_key('myfile.txt',
version_id='8cjozv9Hmkzum8xj.8q8BZxR5CuXnzon’)
>>> key.get_contents_as_string()
'th...
>>> key = bucket.get_key('myfile.txt',
version_id='8cjozv9Hmkzum8xj.8q8BZxR5CuXnzon’)
>>> key.get_contents_as_string()
'th...
Metadata
System and user generated
Name Description Editable?
Date Object creation date No
Content-Length Object size in bytes No
Content-MD5 Base64 encoded ...
User metadata
{your metadata key in here}
Key-value pairs stored with objects and returned with requests
>>> key.set_metadata(’my_tag', ’my metadata')
>>> key.get_metadata(’my_tag')
’my metadata'
CloudFront
Content delivery from global edge locations
CloudFront Edge Locations
Edge Locations
To deliver content to end users with lower latency
A global network of edge locat...
Global content distribution
Download
Enable static and dynamic assets
to be served from edge locations
Streaming
Serve RTM...
Edge access
controls
Optional CNAME
Cache control
S3 Bucket
Forwarding
S3 Bucket
Optional CNAME
Logging to S3
<html>
<script type='text/javascript'
src=’http://d2ew7gdzogp20x.cloudfront.net/jwplayer/jwplayer.js'></script>
<body>
<di...
Summary
Stop doing these:
Capacity planning
Management of storage
Worrying about backing up the backup
Fixing broken hardware
and start doing these
Application backends
Incorporate S3 SDKs into
your applications
Bootstrapping
Store scripts and driv...
aws.amazon.com
Code samples from this Masterclass: github.com/ianmas-aws/s3-masterclass
Ian Massingham – Technical Evangelist
@IanMmmm
@AWS_UKI for local AWS events & news
@AWScloud for Global AWS News and Anno...
AWS Training & Certification
Certification
aws.amazon.com/certification
Demonstrate your skills,
knowledge, and expertise
...
We typically see customers start by trying our
services
Get started now at : aws.amazon.com/getting-started
Design your application for the AWS Cloud
More details on the AWS Architecture Center at : aws.amazon.com/architecture
Masterclass Webinar - Amazon Simple Storage Service S3
Masterclass Webinar - Amazon Simple Storage Service S3
Masterclass Webinar - Amazon Simple Storage Service S3
Masterclass Webinar - Amazon Simple Storage Service S3
Masterclass Webinar - Amazon Simple Storage Service S3
Masterclass Webinar - Amazon Simple Storage Service S3
Masterclass Webinar - Amazon Simple Storage Service S3
Upcoming SlideShare
Loading in...5
×

Masterclass Webinar - Amazon Simple Storage Service S3

862

Published on

In the first Webinar of the 2014 Masterclass Series AWS Technical Evangelist Ian Massingham dives deep into the Amazon Simple Storage Service, S3. He starts by providing an overview of the high level architecture of S3 and the fundamental characteristics of the service before moving on to take a tour through the various features of S3 including storage classes, namespaces, encryption, access controls, transitions and lifecycle management. He also covers related AWS services such as Glacier and the AWS content distribution network, CloudFront, as well as explaining how you can use Amazon S3 to serve static web content.

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
862
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • Speaker Notes:

    We have several programs available to help you deepen your knowledge and proficiency with AWS. We encourage you to check out the following resources:

    Get hands-on experience testing products and gaining practical experience working with AWS technologies by taking an AWS Self-Paced Lab at run.qwiklab.com. Available anywhere, anytime, you have freedom to take self-paced labs on-demand and learn at your own pace. AWS self-paced labs were designed by AWS subject matter experts and provide an opportunity to use the AWS console in a variety of pre-designed scenarios and common use cases, giving you hands-on practice in a live AWS environment to help you gain confidence working with AWS. You have flexibility to choose from topics and products about which you want to learn more.

    Take an instructor-led AWS Training course. We have a variety of role-based courses to meet the requirements of your job role and business need, whether you’re a Solutions Architect, Developer, SysOps Administrator, or just interested in learning AWS fundamentals.

    AWS Certification validates your skills, knowledge an expertise in working with AWS services. Earning certification enables you to gain visibility and credibility for your skills.
  • 750 hours for EC2 per month, for Linux and Windows, 5Gb for S3, and 30Gb for EBS
  • If you already have a project in mind then you may just want to start by designing the application.
  • Transcript of "Masterclass Webinar - Amazon Simple Storage Service S3"

    1. 1. Masterclass Amazon S3 – beyond simple storage Ian Massingham – Technical Evangelist @IanMmmm
    2. 2. A technical deep dive beyond the basics Help educate you on how to get the best from AWS technologies Show you how things work and how to get things done Broaden your knowledge in ~45 minutes Masterclass
    3. 3. It’s more than just a ‘simple’ storage platform A sophisticated 21st century distributed system A bedrock architectural component for many applications Amazon S3
    4. 4. “Spotify needed a storage solution that could scale very quickly without incurring long lead times for upgrades. This led us to cloud storage, and in that market, Amazon Simple Storage Service (Amazon S3) is the most mature large-scale product. Amazon S3 gives us confidence in our ability to expand storage quickly while also providing high data durability.” Emil Fredriksson, Operations Director Find out more at : aws.amazon.com/solutions/case-studies/spotify/
    5. 5. Minecraft Realms and AWS S3 Minecraft worlds and game state are stored in Amazon S3. The system takes advantage of S3 versioning, giving the owner / administrator of a realm the ability to roll back to a previous state. The team implemented efficient transfers to S3 by using S3's multipart upload capabilty. Find out more at : aws.typepad.com/aws/2014/01/hosting-minecraft-realms-on-aws.html
    6. 6. You put it in S3 AWS stores with 99.999999999% durability
    7. 7. You put it in S3 AWS stores with 99.999999999% durability Highly scalable web access to objects Multiple redundant copies in a region
    8. 8. but it’s more than just a simple storage service
    9. 9. Objects in S3 Trillions of Objects (000,000,000,000s) Servicing over 2 million requests per Second
    10. 10. What is S3? Highly scalable data storage Access via APIsA web store, not a file system Fast Highly available & durable Economical
    11. 11. A web store, not a file system Write once, read many (WORM) Eventually consistent
    12. 12. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only Region Availability Zone Indexing Storage Load balancers Web servers Availability Zone Indexing Storage Load balancers Web servers
    13. 13. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only Region Availability Zone Indexing Storage Load balancers Web servers Availability Zone Indexing Storage Load balancers Web servers
    14. 14. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only Region Availability Zone Indexing Storage Load balancers Web servers Availability Zone Indexing Storage Load balancers Web servers
    15. 15. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only Region Availability Zone Indexing Storage Load balancers Web servers Availability Zone Indexing Storage Load balancers Web servers
    16. 16. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only Region Availability Zone Indexing Storage Load balancers Web servers Availability Zone Indexing Storage Load balancers Web servers
    17. 17. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only Region Availability Zone Indexing Storage Load balancers Web servers Availability Zone Indexing Storage Load balancers Web servers
    18. 18. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only Region Availability Zone Indexing Storage Load balancers Web servers Availability Zone Indexing Storage Load balancers Web servers
    19. 19. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only Region Availability Zone Indexing Storage Load balancers Web servers Availability Zone Indexing Storage Load balancers Web servers
    20. 20. A web store, not a file system Write once, read many (WORM) Eventually consistent Conceptual diagram only
    21. 21. A web store, not a file system Write once, read many (WORM) Eventually consistent *except US-STANDARD region New objects Updates Deletes Synchronously stores your data across multiple facilities before returning SUCCESS Read-after-write consistency* Write then read: could report key does not exist Write then list: might not include key in list Overwrite then read: old data could be returned Delete then read: could still get old data Delete then list: deleted key could be included in list
    22. 22. A regional service Your data never leaves a region unless you move it
    23. 23. Storage classes Controlling the way S3 holds your data
    24. 24. Standard Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year Designed to sustain the concurrent loss of data in two facilities Amazon S3 storage classes
    25. 25. Reduced Redundancy Storage Reduces costs by storing data at lower levels of redundancy than the Standard storage Designed to provide 99.99% durability and 99.99% availability of objects over a given year Standard Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year Designed to sustain the concurrent loss of data in two facilities Amazon S3 storage classes
    26. 26. Reduced Redundancy Storage Reduces costs by storing data at lower levels of redundancy than the Standard storage Designed to provide 99.99% durability and 99.99% availability of objects over a given year Standard Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year Designed to sustain the concurrent loss of data in two facilities Glacier Suitable for archiving data, where data access is infrequent and a retrieval time of several hours is acceptable Uses the very low-cost Amazon Glacier storage service, but managed through Amazon S3 Amazon S3 storage classes
    27. 27. Amazon S3 storage classes Glacier Objects you want to keep in archive for a long time e.g. digital archive of old movies & broadcasts Standard Objects you want to have high durability e.g. master copy of movie media Standard Reduced Redundancy Objects you can afford to lose or can recreate e.g. different encodings of movie media
    28. 28. Namespaces Object naming, buckets and keys
    29. 29. Amazon S3 namespace Globally unique bucket name + object name (key)
    30. 30. Amazon S3 namespace Amazon S3 bucket bucket object object objectobject bucket object object
    31. 31. Amazon S3 namespace Amazon S3 jones-docshare media.mydomain.com beach.jpg img1.jpg style.cssdrafts/rpt.doc yourdomain.com swf/mediaplayer.swf img/banner1.jpg
    32. 32. Amazon S3 namespace Object key Unique with a bucket
    33. 33. Amazon S3 namespace Object key Unique with a bucket Max 1024 bytes UTF-8 Including ‘path’ prefixes
    34. 34. Amazon S3 namespace Object key Unique with a bucket Max 1024 bytes UTF-8 Including ‘path’ prefixes assets/js/jquery/plugins/jtables.js this is an object key
    35. 35. Throughput optimisation S3 automatically partitions based upon key prefix: 2134857/gamedata/start.png 2134857/gamedata/resource.rsrc 2134857/gamedata/results.txt 2134858/gamedata/start.png 2134858/gamedata/resource.rsrc 2134858/gamedata/results.txt 2134859/gamedata/start.png 2134859/gamedata/resource.rsrc 2134859/gamedata/results.txt Object keys:Bucket: mynewgame
    36. 36. Throughput optimisation S3 automatically partitions based upon key prefix: 2134857/gamedata/start.png 2134857/gamedata/resource.rsrc 2134857/gamedata/results.txt 2134858/gamedata/start.png 2134858/gamedata/resource.rsrc 2134858/gamedata/results.txt 2134859/gamedata/start.png 2134859/gamedata/resource.rsrc 2134859/gamedata/results.txt Object keys:Bucket: mynewgame Incrementing game id
    37. 37. Throughput optimisation S3 automatically partitions based upon key prefix: 2134857/gamedata/start.png 2134857/gamedata/resource.rsrc 2134857/gamedata/results.txt 2134858/gamedata/start.png 2134858/gamedata/resource.rsrc 2134858/gamedata/results.txt 2134859/gamedata/start.png 2134859/gamedata/resource.rsrc 2134859/gamedata/results.txt Object keys:Bucket: mynewgame mynewgame/2 Partition:
    38. 38. Throughput optimisation S3 automatically partitions based upon key prefix: 7584312/gamedata/start.png 7584312/gamedata/resource.rsrc 7584312/gamedata/results.txt 8584312/gamedata/start.png 8584312/gamedata/resource.rsrc 8584312/gamedata/results.txt 9584312/gamedata/start.png 9584312/gamedata/resource.rsrc 9584312/gamedata/results.txt Object keys:Bucket: mynewgame Reversed game ID
    39. 39. Throughput optimisation S3 automatically partitions based upon key prefix: 7584312/gamedata/start.png 7584312/gamedata/resource.rsrc 7584312/gamedata/results.txt 8584312/gamedata/start.png 8584312/gamedata/resource.rsrc 8584312/gamedata/results.txt 9584312/gamedata/start.png 9584312/gamedata/resource.rsrc 9584312/gamedata/results.txt Object keys:Bucket: mynewgame mynewgame/7 mynewgame/8 mynewgame/9 Partitions:
    40. 40. Encryption Securing data at rest
    41. 41. Server side encryption Automatic encryption of data at rest Durable S3 key storage Simple Additional PUT header Strong AES-256 Self managed No need to manage a key store Secure 3-way simultaneous access
    42. 42. Server side encryption Data bucket High level design
    43. 43. Server side encryption Data bucket Encrypted data Per-object key Encrypted object High level design
    44. 44. Server side encryption Data bucket Encrypted data Encrypted per- object key Per-object key Encrypted object Master key High level design
    45. 45. Server side encryption Data bucket Encrypted data Encrypted per- object key Key management (monthly rotation) Per-object key Encrypted object Master key High level design
    46. 46. docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
    47. 47. Access controls You’re in control of who does what
    48. 48. Secure by default You decide what to share Apply policies to buckets and objects Policies, ACLs & IAM Use S3 policies, ACLs or IAM to define rules
    49. 49. Fine grained Administer as part of role based access Apply policies to S3 at role, user & group level PutObject arn:aws:s3:::mybucket/* Bob Jane IAM
    50. 50. Bucket PoliciesIAM VS Fine grained Administer as part of role based access Apply policies to S3 at role, user & group level Fine grained Apply policies at the bucket level in S3 Incorporate user restrictions without using IAM PutObject arn:aws:s3:::mybucket/* Bob Jane Bob, Jane PutObject arn:aws:s3:::mybucket/* mybucket
    51. 51. Bucket Policies ACLsVSIAM VS Fine grained Administer as part of role based access Apply policies to S3 at role, user & group level Fine grained Apply policies at the bucket level in S3 Incorporate user restrictions without using IAM Coarse grained Apply access control rules at the bucket and/or object level in S3 PutObject arn:aws:s3:::mybucket/* Bob Jane Bob, Jane PutObject arn:aws:s3:::mybucket/* mybucket Everyone, Bob, Jane Read mybucket myobject
    52. 52. {"Statement":[{ "Effect":"Allow", "Principal":{"AWS":["4649-6425", "5243-0045"]}, "Action":"*", "Resource":"/mybucket/*", "Condition":{ "IpAddress":{"AWS:SourceIp":"176.13.0.0/12"} }}]} Access controls – bucket policy
    53. 53. {"Statement":[{ "Effect":"Allow", "Principal":{"AWS":["4649-6425", "5243-0045"]}, "Action":"*", "Resource":"/mybucket/*", "Condition":{ "IpAddress":{"AWS:SourceIp":"176.13.0.0/12"} }}]} Access controls – bucket policy Accounts to allow
    54. 54. {"Statement":[{ "Effect":"Allow", "Principal":{"AWS":["4649-6425", "5243-0045"]}, "Action":"*", "Resource":"/mybucket/*", "Condition":{ "IpAddress":{"AWS:SourceIp":"176.13.0.0/12"} }}]} Access controls – bucket policy Resource
    55. 55. {"Statement":[{ "Effect":"Allow", "Principal":{"AWS":["4649-6425", "5243-0045"]}, "Action":"*", "Resource":"/mybucket/*", "Condition":{ "IpAddress":{"AWS:SourceIp":"176.13.0.0/12"} }}]} Access controls – bucket policy Source address to allow
    56. 56. Transitions & Lifecycle Management Automated management of objects
    57. 57. Lifecycle management
    58. 58. Lifecycle management Object deletion Permanently delete objects from S3
    59. 59. Lifecycle management Object deletion Permanently delete objects from S3 Object archiving Move objects to Glacier and out of S3 storage
    60. 60. Glacier Long term durable archive
    61. 61. Long term Glacier archive Durable Designed for 99.999999999% durability of archives Cost effective Write-once, read-never. Cost effective for long term storage. Pay for accessing data
    62. 62. Logs accessible from S3 time Expiry
    63. 63. Logs ✗accessible from S3 Objects expire and are deleted time Expiry
    64. 64. Logs Txns ✗accessible from S3 Objects expire and are deleted time accessible from S3 Object transition to Glacier invoked ExpiryTransition
    65. 65. Logs Txns ✗accessible from S3 Objects expire and are deleted time accessible from S3 Object transition to Glacier invoked Restoration of object requested for x hrs ExpiryTransition
    66. 66. Logs Txns ✗accessible from S3 Objects expire and are deleted time accessible from S3 Object transition to Glacier invoked Restoration of object requested for x hrs 3-5hrs Object held in S3 RRS for x hrs ExpiryTransition
    67. 67. 3-5 hour retrieval time We assume you won’t access often
    68. 68. using (client = new AmazonS3Client()){ var lifeCycleConfiguration = new LifecycleConfiguration() { Rules = new List<LifecycleRule> { new LifecycleRule { Id = "Archive and delete rule", Prefix = "projectdocs/", Status = LifecycleRuleStatus.Enabled, Transition = new LifecycleTransition() { Days = 365, StorageClass = S3StorageClass.Glacier }, Expiration = new LifecycleRuleExpiration() { Days = 3650 } } } };
    69. 69. using (client = new AmazonS3Client()){ var lifeCycleConfiguration = new LifecycleConfiguration() { Rules = new List<LifecycleRule> { new LifecycleRule { Id = "Archive and delete rule", Prefix = "projectdocs/", Status = LifecycleRuleStatus.Enabled, Transition = new LifecycleTransition() { Days = 365, StorageClass = S3StorageClass.Glacier }, Expiration = new LifecycleRuleExpiration() { Days = 3650 } } } }; Transition to Glacier after 1 year
    70. 70. using (client = new AmazonS3Client()){ var lifeCycleConfiguration = new LifecycleConfiguration() { Rules = new List<LifecycleRule> { new LifecycleRule { Id = "Archive and delete rule", Prefix = "projectdocs/", Status = LifecycleRuleStatus.Enabled, Transition = new LifecycleTransition() { Days = 365, StorageClass = S3StorageClass.Glacier }, Expiration = new LifecycleRuleExpiration() { Days = 3650 } } } }; Delete object after 10 years
    71. 71. POST /ObjectName?restore HTTP/1.1 Host: BucketName.s3.amazonaws.com Date: date Authorization: signatureValue Content-MD5: MD5 <RestoreRequest xmlns="http://s3.amazonaws.com/doc/2006-3-01"> <Days>NumberOfDays</Days> </RestoreRequest>
    72. 72. POST /ObjectName?restore HTTP/1.1 Host: BucketName.s3.amazonaws.com Date: date Authorization: signatureValue Content-MD5: MD5 <RestoreRequest xmlns="http://s3.amazonaws.com/doc/2006-3-01"> <Days>NumberOfDays</Days> </RestoreRequest> 202 Accepted 200 OK 409 Conflict Restoration already in progress Object already restored, number of days updated Restore request accepted Response codes:
    73. 73. Website hosting Static sites straight from S3
    74. 74. It’s a web service… …so serve up web content
    75. 75. Setting default documents Redirecting requests
    76. 76. { "Version":"2008-10-17", "Statement":[{ "Sid":"PublicReadGetObject", "Effect":"Allow", "Principal": { "AWS": "*" }, "Action":["s3:GetObject"], "Resource":["arn:aws:s3:::example-bucket/*" ] } ] }
    77. 77. {bucket-name}.s3-website-{region}.amazonaws.com e.g. mybucket.s3-website-eu-west-1.amazonaws.com s3-{region}.amazonaws.com/{bucket-name}/{object-key} e.g. s3-eu-west-1.amazonaws.com/mybucket/img.png {bucket-name}.s3-{region}.amazonaws.com/{object-key} e.g. mybucket.s3-eu-west-1.amazonaws.com/img.png Website addressing Normal addressing
    78. 78. R53 Record set for: aws-exampl.es
    79. 79. Website bucket name: bucket Record set for: Website bucket name: bucket Error. html Index .html R53 www.aws-exampl.es aws-exampl.es aws-exampl.es
    80. 80. Website bucket name: bucket Record set for: Website bucket name: bucket Error. html Index .html R53 www.aws-exampl.es aws-exampl.es aws-exampl.es Website redirect to: aws-exampl.es
    81. 81. Website bucket name: bucket Record set for: Website bucket name: bucket Error. html Index .html R53 www.aws-exampl.es aws-exampl.es aws-exampl.es Website redirect to: aws-exampl.es A Record ‘Alias’ to S3 website: aws-exampl.es @ s3-website-eu-west-1.amazonaws.com docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html
    82. 82. Website bucket name: bucket Record set for: Website bucket name: bucket Error. html Index .html R53 www.aws-exampl.es aws-exampl.es aws-exampl.es Website redirect to: aws-exampl.es A Record ‘Alias’ to S3 website: aws-exampl.es @ s3-website-eu-west-1.amazonaws.com CNAME for www. to: www.aws-exampl.es.s3-website-eu-west- 1.amazonaws.com docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html
    83. 83. docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html
    84. 84. Time-Limited URLS Control how long objects can be accessed for
    85. 85. Signed URLs Provide time-limited access to specific objects that expires after a set period Access Permissions Use on objects in non-public buckets to prevent access once the signed URL has expired https://ianmas-aws.testbucket.s3.amazonaws.com/testfile.txt ?Signature=JHCa39GV1fKRKkEnAWzI88lH7f8%3D &Expires=1391425438 &AWSAccessKeyId=AKIAIRBKBJ3ZAYAXFC2Q
    86. 86. >>>
    87. 87. >>> import boto >>> conn = boto.connect_s3() >>> conn.generate_url(300, 'GET', bucket='ianmas-aws.testbucket', key='testfile.txt') 'https://ianmas- aws.testbucket.s3.amazonaws.com/testfile.txt?Signature=QUWA%2BGOohFeJ 5pdEzxtdaIFIA6w%3D&Expires=1391425142&AWSAccessKeyId=AKIAIRBKBJ3ZAYAX FC2Q’ >>> conn.generate_url(300, 'GET', bucket='ianmas-aws.testbucket', key='testfile.txt', force_http=True) 'http://ianmas- aws.testbucket.s3.amazonaws.com/testfile.txt?Signature=tALx9KeeSisDSC 0N7KlM%2BIDFZXI%3D&Expires=1391425562&AWSAccessKeyId=AKIAIRBKBJ3ZAYAX FC2Q' 1st parameter is link lifetime in seconds Force a non- SSL link
    88. 88. Error response: link expired AccessDenied Request has expired 70297390BE427DC7 2014-02-03T11:03:58Z I0rI0OWUCnBttFSpEw6Mx4u8uRHgtOSw9k2euDW37skFCU7HH0ulSkUGGaUbn2vg 2014-02-03T11:08:22Z
    89. 89. Object versioning Preserving object histories
    90. 90. Persistent Even deleted object history is held Bucket level Automatically preserves all copies of objects
    91. 91. Persistent Even deleted object history is held Bucket level Automatically preserves all copies of objects
    92. 92. Persistent Even deleted object history is held Bucket level Automatically preserves all copies of objects
    93. 93. >>>
    94. 94. >>> import boto >>> conn = boto.connect_s3() >>> bucket = conn.get_bucket(’mybucket') >>> versions = bucket.list_versions() >>> for version in versions: ... print version.name + version.version_id ... myfile.txt jU9eVv800OlP4PQx6zskMEyPIoExne57 myfile.txt xOJzMvMmGv0Bx2v4QpIypbkkH2XE2yyq myfile.txt 8cjozv9Hmkzum8xj.8q8BZxR5CuXnzon Object version IDs
    95. 95. >>> key = bucket.get_key('myfile.txt', version_id='8cjozv9Hmkzum8xj.8q8BZxR5CuXnzon’) >>> key.get_contents_as_string() 'this is version 1 of my file’ Grabbing the contents of a version
    96. 96. >>> key = bucket.get_key('myfile.txt', version_id='8cjozv9Hmkzum8xj.8q8BZxR5CuXnzon’) >>> key.get_contents_as_string() 'this is version 1 of my file’ >>> key = bucket.get_key('myfile.txt', version_id='xOJzMvMmGv0Bx2v4QpIypbkkH2XE2yyq’) >>> key.get_contents_as_string() 'this is version 2 of my file’
    97. 97. >>> key = bucket.get_key('myfile.txt', version_id='8cjozv9Hmkzum8xj.8q8BZxR5CuXnzon’) >>> key.get_contents_as_string() 'this is version 1 of my file’ >>> key = bucket.get_key('myfile.txt', version_id='xOJzMvMmGv0Bx2v4QpIypbkkH2XE2yyq’) >>> key.get_contents_as_string() 'this is version 2 of my file’ >>> key.generate_url(600) 'https://mybucket.s3.amazonaws.com/myfile.txt?Signature=ABCD& Expires=1358857379&AWSAccessKeyId=AB& versionId=xOJzMvMmGv0Bx2v4QpIypbkkH2XE2yyq' Generating a ’10 minute time bombed’ url for an older version
    98. 98. Metadata System and user generated
    99. 99. Name Description Editable? Date Object creation date No Content-Length Object size in bytes No Content-MD5 Base64 encoded 128bit MD5 digest No x-amz-server-side-encryption Server side encryption enabled for object Yes x-amz-version-id Object version No x-amz-delete-marker Indicates a version enabled object is deleted No x-amz-storage-class Storage class for the object Yes x-amz-website-redirect-location Redirects request for the object to another object or external URL Yes System metadata
    100. 100. User metadata {your metadata key in here} Key-value pairs stored with objects and returned with requests
    101. 101. >>> key.set_metadata(’my_tag', ’my metadata') >>> key.get_metadata(’my_tag') ’my metadata'
    102. 102. CloudFront Content delivery from global edge locations
    103. 103. CloudFront Edge Locations Edge Locations To deliver content to end users with lower latency A global network of edge locations Supports global DNS infrastructure (Route53) and Cloud Front CDN Dallas(2) St.Louis Miami JacksonvilleLos Angeles (2) Palo Alto Seattle Ashburn(3) Newark New York (3) Dublin London(2) Amsterdam (2) Stockholm Frankfurt(2) Paris(2) Singapore(2) Hong Kong (2) Tokyo (2) Sao Paulo South Bend San Jose Osaka Milan Sydney Madrid Seoul Mumbai Chennai
    104. 104. Global content distribution Download Enable static and dynamic assets to be served from edge locations Streaming Serve RTMP directly from media files in buckets
    105. 105. Edge access controls Optional CNAME Cache control S3 Bucket Forwarding
    106. 106. S3 Bucket Optional CNAME Logging to S3
    107. 107. <html> <script type='text/javascript' src=’http://d2ew7gdzogp20x.cloudfront.net/jwplayer/jwplayer.js'></script> <body> <div id='player'></div> <script type='text/javascript'> jwplayer('player').setup({ file: "rtmp://s1eat02wfxn38u.cloudfront.net/cfx/st/montage-medium.mp4", width: "480", height: "270", }); </script> </body> </html> Streaming distribution Download distribution
    108. 108. Summary
    109. 109. Stop doing these: Capacity planning Management of storage Worrying about backing up the backup Fixing broken hardware
    110. 110. and start doing these Application backends Incorporate S3 SDKs into your applications Bootstrapping Store scripts and drive EC2 instances on startup Application logs Store logs and analyse with EMR Web content Serve content and distribute globally Documentation Store documents with versioning and security models Backups & archive Storage gateway, 3rd party tools
    111. 111. aws.amazon.com Code samples from this Masterclass: github.com/ianmas-aws/s3-masterclass
    112. 112. Ian Massingham – Technical Evangelist @IanMmmm @AWS_UKI for local AWS events & news @AWScloud for Global AWS News and Announcements ©Amazon.com, Inc. and its affiliates. All rights reserved.
    113. 113. AWS Training & Certification Certification aws.amazon.com/certification Demonstrate your skills, knowledge, and expertise with the AWS platform Self-Paced Labs aws.amazon.com/training/ self-paced-labs Try products, gain new skills, and get hands-on practice working with AWS technologies aws.amazon.com/training Training Skill up and gain confidence to design, develop, deploy and manage your applications on AWS
    114. 114. We typically see customers start by trying our services Get started now at : aws.amazon.com/getting-started
    115. 115. Design your application for the AWS Cloud More details on the AWS Architecture Center at : aws.amazon.com/architecture

    ×