Want to build an application that requires minimal up-front investment, and will seamlessly scale from hundreds to millions of users? Amazon S3 is a powerful building block that can enable you to focus your time on the value and functionality of your application, rather than the challenges of scaling it. In this session we'll cover techniques to best take advantage of the platform. We'll discuss structuring your key naming convention to maximize consistency of performance, as well as ways to optimize your upload and download throughput. We'll learn how to eliminate proxies between your application and Amazon S3, and use the platform for your logging needs. Finally, we'll cover simple techniques for efficiently managing the billions of objects your highly scaled application may accumulate.

3. Buckets
Objects
Keys

8. 1. Hashing Keys
2. Lifecycle Policies
3. Multipart Upload
4. Object Aggregation
5. Content Distribution
6. Server Access Logging
7. Bucket Policies
8. Pre-signed URLs
9. DNS Time To Live
10.TCP window scaling

10. mybucket/2012-11-29-15-06-03/cust1234234/photo1.jpg
mybucket/2012-11-29-15-06-03/cust8234233/photo2.jpg
mybucket/2012-11-29-15-06-03/cust1234234/photo2.jpg
mybucket/2012-11-29-15-06-03/cust7433234/photo1.jpg
mybucket/2012-11-29-15-06-04/cust8234233/photo5.jpg
mybucket/2012-11-29-15-06-04/cust2123234/photo1.jpg
mybucket/2012-11-29-15-06-04/cust8234233/photo8.jpg
mybucket/2012-11-29-15-06-04/cust3241234/photo7.jpg

11. mybucket/2012-11-29-15-06-03/cust1234234/photo1.jpg
mybucket/2012-11-29-15-06-03/cust8234233/photo2.jpg
mybucket/2012-11-29-15-06-03/cust1234234/photo2.jpg
mybucket/2012-11-29-15-06-03/cust7433234/photo1.jpg
mybucket/2012-11-29-15-06-04/cust8234233/photo5.jpg
mybucket/2012-11-29-15-06-04/cust2123234/photo1.jpg
mybucket/2012-11-29-15-06-04/cust8234233/photo8.jpg
mybucket/2012-11-29-15-06-04/cust3241234/photo7.jpg

12. mybucket/2012-11-29-15-06-03/cust1234234/photo1.jpg

14. Host-D
Host-A
Host-C
Host-B

15. Partition-A Partition-B Partition-C Partition-D

16. Partition-A Partition-B Partition-C Partition-D

17. Partition-A Partition-B Partition-C Partition-D

18. Partition-A Partition-B Partition-C Partition-D Partition-E

19. mybucket/232A-2012-11-29-15-06-03/cust1234234/photo1.jpg
mybucket/7B54-2012-11-29-15-06-03/cust8234233/photo2.jpg
mybucket/921C-2012-11-29-15-06-03/cust1234234/photo2.jpg
mybucket/BA65-2012-11-29-15-06-03/cust7433234/photo1.jpg
mybucket/8761-2012-11-29-15-06-04/cust8234233/photo5.jpg
mybucket/2E4F-2012-11-29-15-06-04/cust2123234/photo1.jpg
mybucket/9810-2012-11-29-15-06-04/cust8234233/photo8.jpg
mybucket/7E34-2012-11-29-15-06-04/cust3241234/photo7.jpg

20. Partition-A Partition-B Partition-C Partition-D

21. Partition-A Partition-B Partition-C Partition-C’ Partition-D

22. MyOtherBucket/1290052.obj MyOtherBucket/2500921.obj
MyOtherBucket/1290053.obj MyOtherBucket/3500921.obj
MyOtherBucket/1290054.obj MyOtherBucket/4500921.obj
MyOtherBucket/1290055.obj MyOtherBucket/5500921.obj
MyOtherBucket/1290056.obj MyOtherBucket/6500921.obj
MyOtherBucket/1290057.obj MyOtherBucket/7500921.obj
MyOtherBucket/1290058.obj MyOtherBucket/8500921.obj
MyOtherBucket/1290059.obj MyOtherBucket/9500921.obj

26. 30 days log objects expire rule

27. log/

28. 30

31. MyHugeDataBlob.obj Scaling Large Uploads

32. MyHugeDataBlob.obj Scaling Large Uploads

33. MyHugeDataBlob.obj Scaling Large Uploads
X

34. MyHugeDataBlob.obj Multipart Uploads
Parallel uploads

35. MyHugeDataBlob.obj Multipart Uploads
Recovery from network issues

36. MyLogFile.txt Multipart Uploads
Unknown Object Length

39. Upload a Single Object

40. Upload using Multipart Upload

41. Retrieve via Ranged Gets

44. S3

45. S3
Cloudfront

47. 68 ms
Round trip

52. Incoming
Outgoing

53. IAM Users Restricted
Partner
Private

55. Scaling Customer Uploads

56. Scaling Customer Uploads

57. Hand out credentials -- NO

58. Pre-signed URLS

60. S3 has many IP address
IP address come and go
DNS resolution is cached
S3 service SLA not IP address SLA

61. Java DNS Caching Behavior
http://docs.oracle.com/javase/1.5.0/docs/api/java/net/InetAddress.html

63. T1
1.544Mbit/s
188KiB/s
100 ms
BDP = 18.8 KiB

64. 1Gbit/s LAN
119.2 MiB/s
1 ms
BDP = 122 KiB

65. 1Gbit/s LAN
119.2 MiB/s
100 ms
BDP = 11.9 MiB

66. http://tools.ietf.org/html/rfc1323

67. 1Gbit/s Fiber
119.2 MiB/s
100 ms
64Kib Window  640KiB/s throughput

68. 1Gbit/s Fiber
119.2 MiB/s
100 ms
64Kib Window  640KiB/s throughput

69. 1Gbit/s Fiber
119.2 MiB/s
100 ms
64Kib Window  640KiB/s throughput

70. 1Gbit/s Fiber
119.2 MiB/s
100 ms
64Kib Window  640KiB/s throughput

71. 1Gbit/s Fiber
119.2 MiB/s
100 ms
64Kib Window  640KiB/s throughput

72. WSCALE Size in Bytes Size
0 65,536 64 KiB
1 131,072 128KiB
2 262,144 256 KiB
3 524,288 512 KiB
4 1,048,576 1 MiB
5 2,097,152 2 MiB
6 4,194,304 4 MiB
7 8,388,608 8 MiB
8 16,777,216 16 MiB
9 33,554,432 32 MiB
10 67,108,864 64 MiB
11 134,217,728 128 MiB
12 268,435,456 256 MiB
13 536,870,912 512 MiB
14 1,073,741,824 1 GiB

73. 100Mb/s
11.9 MiB/s
335ms
4Mib Window

74. 2000 1400 1000 0

75. 2000 1400 1000 0
ACK 1000

76. 2000 1400 1000 0
1500-2000 1100-1400 0 -1000
http://tools.ietf.org/html/rfc2018

77. 1. Hashing Keys
2. Lifecycle Policies
3. Multipart Upload
4. Object Aggregation
5. Content Distribution
6. End User Logging
7. Bucket Policies
8. Pre-signed URLs
9. DNS Time To Live
10.TCP window scaling