• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
STG303 Building Scalable Applications on Amazon S3 - AWS re: Invent 2012
 

STG303 Building Scalable Applications on Amazon S3 - AWS re: Invent 2012

on

  • 2,080 views

Want to build an application that requires minimal up-front investment, and will seamlessly scale from hundreds to millions of users? Amazon S3 is a powerful building block that can enable you to ...

Want to build an application that requires minimal up-front investment, and will seamlessly scale from hundreds to millions of users? Amazon S3 is a powerful building block that can enable you to focus your time on the value and functionality of your application, rather than the challenges of scaling it. In this session we'll cover techniques to best take advantage of the platform. We'll discuss structuring your key naming convention to maximize consistency of performance, as well as ways to optimize your upload and download throughput. We'll learn how to eliminate proxies between your application and Amazon S3, and use the platform for your logging needs. Finally, we'll cover simple techniques for efficiently managing the billions of objects your highly scaled application may accumulate.

Statistics

Views

Total Views
2,080
Views on SlideShare
2,074
Embed Views
6

Actions

Likes
5
Downloads
0
Comments
0

1 Embed 6

https://twitter.com 6

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    STG303 Building Scalable Applications on Amazon S3 - AWS re: Invent 2012 STG303 Building Scalable Applications on Amazon S3 - AWS re: Invent 2012 Presentation Transcript

    • BucketsObjectsKeys
    • 1. Hashing Keys2. Lifecycle Policies3. Multipart Upload4. Object Aggregation5. Content Distribution6. Server Access Logging7. Bucket Policies8. Pre-signed URLs9. DNS Time To Live10.TCP window scaling
    • mybucket/2012-11-29-15-06-03/cust1234234/photo1.jpgmybucket/2012-11-29-15-06-03/cust8234233/photo2.jpgmybucket/2012-11-29-15-06-03/cust1234234/photo2.jpgmybucket/2012-11-29-15-06-03/cust7433234/photo1.jpgmybucket/2012-11-29-15-06-04/cust8234233/photo5.jpgmybucket/2012-11-29-15-06-04/cust2123234/photo1.jpgmybucket/2012-11-29-15-06-04/cust8234233/photo8.jpgmybucket/2012-11-29-15-06-04/cust3241234/photo7.jpg
    • mybucket/2012-11-29-15-06-03/cust1234234/photo1.jpgmybucket/2012-11-29-15-06-03/cust8234233/photo2.jpgmybucket/2012-11-29-15-06-03/cust1234234/photo2.jpgmybucket/2012-11-29-15-06-03/cust7433234/photo1.jpgmybucket/2012-11-29-15-06-04/cust8234233/photo5.jpgmybucket/2012-11-29-15-06-04/cust2123234/photo1.jpgmybucket/2012-11-29-15-06-04/cust8234233/photo8.jpgmybucket/2012-11-29-15-06-04/cust3241234/photo7.jpg
    • mybucket/2012-11-29-15-06-03/cust1234234/photo1.jpg
    • Host-D Host-AHost-C Host-B
    • Partition-A Partition-B Partition-C Partition-D
    • Partition-A Partition-B Partition-C Partition-D
    • Partition-A Partition-B Partition-C Partition-D
    • Partition-A Partition-B Partition-C Partition-D Partition-E
    • mybucket/232A-2012-11-29-15-06-03/cust1234234/photo1.jpgmybucket/7B54-2012-11-29-15-06-03/cust8234233/photo2.jpgmybucket/921C-2012-11-29-15-06-03/cust1234234/photo2.jpgmybucket/BA65-2012-11-29-15-06-03/cust7433234/photo1.jpgmybucket/8761-2012-11-29-15-06-04/cust8234233/photo5.jpgmybucket/2E4F-2012-11-29-15-06-04/cust2123234/photo1.jpgmybucket/9810-2012-11-29-15-06-04/cust8234233/photo8.jpgmybucket/7E34-2012-11-29-15-06-04/cust3241234/photo7.jpg
    • Partition-A Partition-B Partition-C Partition-D
    • Partition-A Partition-B Partition-C Partition-C’ Partition-D
    • MyOtherBucket/1290052.obj MyOtherBucket/2500921.objMyOtherBucket/1290053.obj MyOtherBucket/3500921.objMyOtherBucket/1290054.obj MyOtherBucket/4500921.objMyOtherBucket/1290055.obj MyOtherBucket/5500921.objMyOtherBucket/1290056.obj MyOtherBucket/6500921.objMyOtherBucket/1290057.obj MyOtherBucket/7500921.objMyOtherBucket/1290058.obj MyOtherBucket/8500921.objMyOtherBucket/1290059.obj MyOtherBucket/9500921.obj
    • 30 days log objects expire rule
    • log/
    • 30
    • MyHugeDataBlob.obj Scaling Large Uploads
    • MyHugeDataBlob.obj Scaling Large Uploads
    • MyHugeDataBlob.obj Scaling Large Uploads X
    • MyHugeDataBlob.obj Multipart Uploads Parallel uploads
    • MyHugeDataBlob.obj Multipart Uploads Recovery from network issues
    • MyLogFile.txt Multipart Uploads Unknown Object Length
    • Upload a Single Object
    • Upload using Multipart Upload
    • Retrieve via Ranged Gets
    • S3
    • S3Cloudfront
    • 68 msRound trip
    • Incoming Outgoing
    • IAM Users Restricted Partner Private
    • Scaling Customer Uploads
    • Scaling Customer Uploads
    • Hand out credentials -- NO
    • Pre-signed URLS
    • S3 has many IP addressIP address come and goDNS resolution is cachedS3 service SLA not IP address SLA
    • Java DNS Caching Behaviorhttp://docs.oracle.com/javase/1.5.0/docs/api/java/net/InetAddress.html
    • T11.544Mbit/s188KiB/s 100 ms BDP = 18.8 KiB
    • 1Gbit/s LAN119.2 MiB/s 1 ms BDP = 122 KiB
    • 1Gbit/s LAN119.2 MiB/s 100 ms BDP = 11.9 MiB
    • http://tools.ietf.org/html/rfc1323
    • 1Gbit/s Fiber119.2 MiB/s 100 ms 64Kib Window  640KiB/s throughput
    • 1Gbit/s Fiber119.2 MiB/s 100 ms 64Kib Window  640KiB/s throughput
    • 1Gbit/s Fiber119.2 MiB/s 100 ms 64Kib Window  640KiB/s throughput
    • 1Gbit/s Fiber119.2 MiB/s 100 ms 64Kib Window  640KiB/s throughput
    • 1Gbit/s Fiber119.2 MiB/s 100 ms 64Kib Window  640KiB/s throughput
    • WSCALE Size in Bytes Size 0 65,536 64 KiB 1 131,072 128KiB 2 262,144 256 KiB 3 524,288 512 KiB 4 1,048,576 1 MiB 5 2,097,152 2 MiB 6 4,194,304 4 MiB 7 8,388,608 8 MiB 8 16,777,216 16 MiB 9 33,554,432 32 MiB 10 67,108,864 64 MiB 11 134,217,728 128 MiB 12 268,435,456 256 MiB 13 536,870,912 512 MiB 14 1,073,741,824 1 GiB
    • 100Mb/s11.9 MiB/s 335ms 4Mib Window
    • 2000 1400 1000 0
    • 2000 1400 1000 0 ACK 1000
    • 2000 1400 1000 0 1500-2000 1100-1400 0 -1000 http://tools.ietf.org/html/rfc2018
    • 1. Hashing Keys2. Lifecycle Policies3. Multipart Upload4. Object Aggregation5. Content Distribution6. End User Logging7. Bucket Policies8. Pre-signed URLs9. DNS Time To Live10.TCP window scaling