Web Applications on AWS: This session introduces AWS services that you can leverage to build a scalable web application architecture on AWS to handle large-scale flows.

1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
數位媒體雲端儲存技術分享與案例
Harry Lin, AWS Solutions Architect
linharry@amazon.com

2. Agenda
Media Storage & Archive
Media Ingest
Case Study

3. Where is the problem?

4. AWS Direct Connect
Elastic Load!
Balancing!
AWS Import/
Export
Amazon
S3
AWS Storage
Gateway
Amazon
Glacier!
Amazon
EBS
Amazon!
CloudFront!
Amazon
CloudSearch!
Amazon
SNS!
Amazon
SQS!
Amazon
SWF!
Amazon
Elastic
Transcoder!
Amazon
EC2! Amazon
EMR!
Amazon!
DynamoDB!
Amazon
VPC!
Amazon
RDS!
Amazon
Redshift
Amazon!
ElastiCache
AMI!
Amazon!
Route 53!
Ingest Store DeliverProcess
Amazon
EC2!
AWS Services Stack in a Media Workflow

5. MEDIA STORAGE & ARCHIVE

6. A Concept – The Content Lake
Inspired from Data Lake (Coined by James Dixon in 2010)
• A single store of all of digital contents that you create and
acquire in any form or factor
• Don’t assume any resolutions/formats
• It is up to the consumer (application consuming the content) to use the
appropriate infrastructure for processing

7. Amazon S3: The Content Lake
• Designed for 99.999999999% durability
• Authentication & authorisation mechanisms
ensure that data is kept secure
• Cross-region replication
• Multiple encryption options
• Object lifecycle management
• Standard, infrequent access (IA) & Reduced
redundancy storage (RRS)

8. S3 Resources: Buckets and Objects

9. S3 Cross-Region Replication
Automated, fast, and reliable asynchronous replication of data across AWS regions
Source
(Virginia)
Destination
(Oregon)
• Only replicates new PUTs. Once
S3 is configured, all new uploads
into a source bucket will be
replicated
• Entire bucket or prefix based
• 1:1 replication between any 2
regions
• Versioning required
Use cases:
• Compliance—store data hundreds of miles apart
• Lower latency—distribute data to regional customers
• Security—create remote replicas managed by separate AWS accounts

10. S3 Event Notifications
Delivers notifications to Amazon SNS, Amazon SQS, or AWS
Lambda when events occur in S3
S3
Events
SNS topic
SQS queue
Lambda function
Notifications
Notifications
Notifications
Foo() {
…
}

11. Amazon S3 Security Controls
• Bucket-level and
object-level permissions
• Owner-only access (by default)
• Signed URLs/query string
authentication
• AWS IAM policies
• Versioning (MFA delete)
• Detailed access logging
✔Access logs

12. Amazon S3 Server-Side Encryption
• Encryption& Decryption
• Key management
(Encrypted by Amazon S3 master
key; stored separately from your
data)
• 256-bit AES encryption
• User-provided keys
• Integration with AWS KMS
Content to be uploaded
(encryption enabled in the
HTTP header)
Envelop Key
Encrypted stored key
101010101010
101010101010
101010101010
101010101010
101010101010
101010101010
101010101010
101010101010
Encrypted stored data
Master Amazon S3 key
Encryption
Encryption Key
generator
Amazon S3

13. 1 PB raw storage
800 TB usable storage
600 TB allocated storage
400 TB application data
S3 Capacity Pricing—Pay Only For What You Use!
Amazon S3
No need for capacity
planning or provisioning

14. Archiving Options - Amazon Glacier
• Designed for 99.999999999% durability
• AES 256 encryption at rest
• Highly scalable and reliable
• Authentication & authorisation mechanisms
ensure data is kept secure
• Retrievals:
• 5% of monthly average storage (pro-rated daily)
free, after this 0.011 per gigabyte
• 3–5 hour asynchronous retrieval
aws.amazon.com/glacier

15. $0.0114
per GB-month

16. $140
per TB/year

17. 99.999999999%
durability

18. Storage costs
vs.
Retrieval costs

19. Glacier Benefits
• Reduce cost for long-term archiving
• Leverage unlimited storage capacity
• Replace tape museums
• Improve durability

20. S3 Lifecycle Policies →
Key prefix “logs/”
Transition objects to Glacier 30 days after creation
Delete 365 days after creation date
<LifecycleConfiguration>
<Rule>
<ID>archive-in-30-days</ID>
<Prefix>logs/</Prefix>
<Status>Enabled</Status>
<Transition>
<Days>30</Days>
<StorageClass>GLACIER</StorageClass>
</Transition>
<Expiration>
<Days>365</Days>
</Expiration>
</Rule>
</LifecycleConfiguration

21. Selecting the Right Object Storage Class
S3
S3-IA
Glacier
L
i
f
e
c
y
c
l
e
Available
S3: 99.99%
S3-IA: 99.9%
Performant
Low Latency
High Throughput
Secure
SSE, client
encryption, IAM
integration
Event
Notifications
SQS, SNS, and
Lambda
Versioning
Keep multiple
copies
automatically
Cross Region
Replication
Common
Namespace
Define storage
class per object
Durable
99.999999999%
Scalable
Elastic capacity
No preset limits
“Hot” Data
Active and/or
Temporary Data
“Warm” Data
Infrequently
Accessed Data
“Cold” Data
Archive and
Compliance Data

22. Selecting the Right Object Storage Class
S3
S3-IA
Glacier
L
i
f
e
c
y
c
l
e
Available
S3: 99.99%
S3-IA: 99.9%
Performant
Low Latency
High Throughput
≥ 30 Days
≥ 90 Days
Durable
99.999999999%
Scalable
Elastic capacity
No preset limits
$0.007/GB per month
$0.0125/GB per month
“Hot” Data
Active and/or
Temporary Data
“Warm” Data
Infrequently
Accessed Data
“Cold” Data
Archive and
Compliance Data
≥ 0 Days~$0.03/GB per month
3 – 5 Hrs
$0.01/GB retrieval
$0.01/GB retrieval < 5%

23. Amazon Elastic Block Store (EBS)
• High I/O block storage for Amazon EC2
• Predictably scale to 1000s of IOPS per
Amazon EC2 instance
• Automatic replication within the Availability
Zone
• Point-in-time snapshots
• Amazon S3 durability (11-9s)
• Point-in-time snapshots across regions
aws.amazon.com/ebs/

24. EBS – The SAN in the cloud

25. EBS – Persists beyond an instance

26. EBS – Attach multiple volumes to an instance

27. EBS – Snapshots to S3 are your backups

28. EBS use cases
• Persistent block storage for Amazon EC2
• Transactional workloads
• File system for an instance—NTFS, ExtFS, and so on

29. Overview of Amazon EBS Volume Types
Solid State Drives (SSD) Hard Disk Drives (HDD)
------------------------------------------------------------------Current Generation------------------------------------------------------------------- -----Previous Gen-----
Volume Type
Provisioned IOPS SSD
(io1)
General Purpose SSD
(gp2)
Throughput Optimized
HDD (st1)
Cold HDD
(sc1)
Magnetic
Use Cases
I/O-intensive NoSQL
and relational databases
Boot volumes, low-
latency interactive
apps, dev & test
Big data, data
warehouses, log
processing
Colder data requiring
fewer scans per day
Infrequent data
Access
Volume Size 4 GB - 16 TB 1 GB - 16 TB 500 GB - 16 TB 500 GB - 16 TB Up to 1TB
Max IOPS*/
Volume
20,000 10,000 500 250 40 - 200
Max Throughput/
Volume
320 MB/s 160 MB/s 500 MB/s 250 MB/s 40 - 90MBps
Max Throughput /
Instance
800 MB/s 800 MB/s 800 MB/s 800 MB/s 800MBps
Price**
$0.125/GB-month
$0.065/provisioned
IOPS
$0.10/GB-month $0.045/GB-month $0.025/GB-month
$.05/GB - Month
$.05/million I/O
New!
EBS Volumes available in all regions
* io1/gp2 based on 16K I/O size, st1/sc1 based on 1 MB I/O size
** Pricing is for US East-1 as of 4/19/2016, for the latest see Amazon EBS Pricing

30. MEDIA INGEST

31. • Reduces costs for bandwidth-heavy
workloads
• Private connectivity to AWS
• Physical connection – 1 Gbps or
10 Gbps port
• Logical connections (802.1q
VLANs)
• Public: To AWS cloud (Amazon S3
endpoints etc.)
• Private: To in-VPC resources
• Consistent network performance
• Compatible with all AWS services
Ingest Options – AWS Direct Connect
aws.amazon.com/directconnect

32. • Rapidly move data into and out of AWS
• Portable storage device shipment to
AWS
• eSATA
• USB 2.0 and 3.0
• 2.5 and 3.5 inch internal SATA hard
drives
• Supports
• Amazon Elastic Block Store (EBS)
• Amazon Simple Storage Service (S3)
• Amazon Glacier
Ingest Options – AWS Import/Export Disk
aws.amazon.com/importexport/disk

33. 1. Multipart upload
2. Request rate optimization
3. TCP window scaling
4. TCP selective acknowledgement
Ingest Options – S3 Multipart Uploads

34. Amazon S3 Transfer Acceleration
Embedded WAN acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Move over long geographic distances
Up to 6 times faster
No firewall mods, no client software
54 global edge locations
Change your endpoint, not your code

35. S3 Transfer Acceleration Pricing
Starting at $0.04/GB transferred (+ usual
bandwidth charges). Up to $0.08/GB in
some regions
Pay only for what you use
Accelerated performance or no charge

36. CASE STUDY

37. • Netflix is one of the leading streaming
video services for movies and TV with
millions of users globally
• Needed a reliable, scalable system
that would not be prone to data
corruption
• With AWS, Netflix can quickly deploy
thousands of servers and terabytes of
storage within minutes—and spin
servers down when they’re not needed
Netflix Uses AWS to Increase Reliability and Efficiency

38. Use of AWS:
Corporate website, 40 producer websites, streaming
for iPhone and iPad applications
Use Amazon EC2, Amazon S3, Amazon RDS, and
stream 1.5 PB/m on Amazon CloudFront
Business Benefits:
Launched and scaled multiple media services
fast
Improved server set up time from months to 15
minutes
Improved Reliability to 99.99%
“I can’t imagine having been able to do very many of the things that we’ve done, in the
timeframe that we’ve done them, without the benefit of a platform like this.” – Jon
Brendsel, VP, Products
PBS Distributes its Online and Mobile Media
on AWS, Launching new Media Services Fast

39. Summary: Content Lake

40. Thank You!
AWS is in TW for You