Learn how Amazon.com continuously improves the availability and performance of its website with AWS. Gavin Jewell, Director of Amazon's Consumer Cloud Enablement group, will go in depth on how Amazon CloudFront helps them accelerate their website globally, and how it gives flexibility to apply various security measures at the edge. He will also explain how they are using services such as AWS Shield, AWS WAF, and Route 53. Lastly, we will explore Amazon.com’s continuous and incremental re-architecture program that ensures their infrastructure is constantly updated to use AWS natively.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Case Study:
T h e I n t e r n a l s o f A m a z o n . C o m ' s A r c h i t e c t u r e T h a t A l l o w s i t t o S e c u r e l y
S c a l e f o r M i l l i o n s o f T r a n s a c t i o n s p e r S e c o n d w i t h L o w L a t e n c y , a n d H i g h
A v a i l a b i l i t y
G a v i n J e w e l l
N o v e m b e r 3 0 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security Availability Latency Cost
Guiding Principles Architecting Amazon.com

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Business at Internet Scale
Dynamic Content
15 PB/Mo
Requests per Day
40B ??
Hosted in
13 Countries

4. Architecture Evolution—Journey to AWS
P r e v i o u s
C i r c a 2 0 0 6
C u r r e n t
H y b r i d
F u t u r e
N a t i v e A W S
Open Source Kits
Relational Databases
On-Premises Hosting
Physical NetworkingIn-house Frameworks
Regional Datacenters
Amazon
CloudFront
AWS
WAF
Amazon
EC2
Amazon
DynamoDB
Amazon
S3
AWS
Lambda
AWS
Shield
Amazon
Route 53
Amazon
ECS

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“Amazon.com is a novel application on top
of AWS primitives.”
§ Amazon CloudFront can reduce your latency for your global websites and services
§ Amazon Route 53 can be used to support your multiple origins
§ HTTPS re-directs at CloudFront can save you money
§ Use AWS WAF to efficiently drop unwanted traffic
§ Access the same experts at DDoS that we do with Shield Advanced

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Legacy On Premises
Compute
Server Server Server
Amazon.com Hybrid Architecture
On-Premises
Storage
AWS Compute AWS Storage
Amazon
DynamoDB
Amazon
S3
AWS
Lambda
Amazon
EC2
Amazon
ECS
Database Local DiskDesktop
Mobile
DNS
Amazon
Route 53
Security & Defense
Other CDN
CDN
Amazon
CloudFront
AWS Shield AWS WAF
Origin
Amazon VPC
Amazon EC2
Reverse Proxy

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
R o u t e 5 3
How did Amazon integrate
multiple origins and improve
availability?

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route 53 Features That Amazon Depends On
§ Self-service DNS management with APIs
§ Weighted routing policies allows Amazon
to split and control traffic
§ Flexible configuration for managing
CNAMES and complex routes

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route 53—Amazon ARecord/CNAMEs
§ ARecord and CNAMEs are used
throughout our design
§ Route 53 features manage the
routing policies
§ CloudFront is configured to access
the origin via certificate
nslookup shows the A Record/CNAME hierarchy

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route 53—A Record and CNAME config
ORIGINORIGIN
www.cdn.amazon.com.
(CNAME)
[NAME].[otherCDN].net.
(CNAME)
[NAME].cloudfront.net.
(CNAME)
[NAME].[origin].net.
(CNAME)
www.amazon.com
(A Record)
User

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route 53—DNS Request Weighting
§ Creating a record set for the origin allows you to send
requests over the CDN in percentages
§ Configuration will “weight” requests to anything you
can configure a CNAME for
§ Creating a “weighted” routing policy for the origin
allows you to send traffic over the CDN or remove the
CDN without changing DNS

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route 53—DNS Request Weighting
Support for different record set types;
Amazon uses CNAMEs
Custom settings for Time To Live (TTL) so
changes are predictable
CNAME to CDN Provider or Origin
“Weighted” routing policy allows
percentage granular control
Percentage of requests routed to the
CNAME in this “Record Set”

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How did Amazon
improve latency without
changing code? C l o u d F r o n t

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
§ POP locations reduce time to connect and round-trip time
§ Routing rules route between datacenters
§ Connection reuse
§ TLS/SSL termination at the edge
§ HTTP2 support (big latency wins!)
CloudFront Features Amazon Depends On

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon.com Page Latency
Direct to Origin
(No CloudFront)
CloudFront

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudFront—Amazon.com Config
§ Path pattern adds routing logic for multiple fleets of servers
§ All HTTP requests are redirected to HTTPS at the edge
§ Certificates identify communication between CloudFront and origin
When request pattern
matches, requests are
routed to “origin” value
To ensure all user’s
requests are encrypted,
HTTP requests are
redirected to HTTPS
/dp/* Custom Detail Page Routing to ASINs

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How did Amazon become the MOST
TRUSTED brand in eCommerce?
W A F S h i e l d

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Protecting your site and your customers
§ DDoS is an Availability risk
§ Robot mitigation is a business choice
§ Customer data protection is a false choice;
it’s an extinction-level event for your customer trust

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Response to Pingback Attacks
§ WordPress is an open source blog platform
§ It is widely used across the internet
§ Pingback is a supported feature many bloggers use for better SEO and visibility
A pingback is an XML-RPC request sent from Site A to Site B, when an author of Site
A writes a post that links to Site B. When Site B receives the notification signal, it
automatically goes back to Site A, checking for the existence of a live incoming link

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Does a Pingback Attack Work?
Attacker distributes spoofed
XMLRPC pingback requests
with victims’ addresses
Legitimate WordPress (WP) sites
follow instructions and generate
many pingback requests
Target website is sent lots of
HTTP requests from multiple
legitimate WP websites
Attacker’s
Server
Abused
WP
Website
Abused
WP
Website
Abused
WP
Website
Target
Website

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Legacy On Prem
Compute
Server Server Server
Amazon.com Hybrid Architecture
On Prem Storage
AWS Compute AWS Storage
Amazon
DynamoDB
Amazon
S3
AWS
Lambda
Amazon
EC2
Amazon
ECS
Database Local DiskDesktop
Mobile
DNS
Amazon
Route 53
Security & Defense
Other CDN
CDN
Amazon
CloudFront
AWS Shield AWS WAF
Origin
Amazon VPC
Amazon EC2
Reverse Proxy
Pingback
Filter
x

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How to Identify a WP Pingback Request
A pingback request will use a user-agent string similar to:
WordPress/4.3.3; http://168.63.218.68; verifying pingback
from 185.130.5.209
WAF Rule

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS WAF—Did it Work?
Burst of pingback
traffic being filtered
by AWS WAF

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Shield Advanced
§ Amazon relies on their 24x7 DDoS response support using
advanced tools
§ They have many years of experience running the biggest events
of Amazon (Cyber Monday, Black Friday, Prime Day)
§ Amazon partners with Shield Advance each year to review and
update their preparation
§ Benefit from their experience and continual improvement
“It’s support from experts who’ve honed their skills defending Amazon.com.”

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon can use Lambda edge to politely respond to requests based on HTTP headers and provide
alternate content and response codes, protecting users and the website
Lambda@Edge with CloudFront

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Configure Lambda at the Edge
Create or edit a behavior that
will be triggered by the
“Viewer Request” event

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudFront Field Level Encryption
§ An important part of our PCI compliance
§ A simple design encrypts data before most code runs
§ Protect credit cards and other sensitive data
§ CloudFront encrypts user-designated fields
§ Field Level Encryption is in preview today and will be
launching in Dec. 2017

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudFront Field-Level Encryption
• Single Page Checkout
• Widgets for:
• Add a credit card/bank account number
• Add an address
• Shows what product(s) are being sold
• Computes taxes, shipping, promotions, total cost
• Apply for a new credit card
• Confirm and checkout
$1.40
$21.75

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudFront Field Level Encryption
§ Multiple widgets from separate teams in Amazon on a shared
website environment
§ Only the credit card widget needs the credit card number
§ With Field Level Encryption, the rest of the widgets can’t see the
un-encrypted data

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
This all sounds complex to
manage; how does Amazon
do it safely at scale? A W S S D K

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Managing AWS Config at Scale
§ Amazon has a lot of AWS stuff…
§ Each site requires dozens of AWS objects
and configurations
§ Hundreds of sites and domains
§ Consistency is critical

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Do We Manage It All Consistently?
§ Write layers onto the AWS SDK to apply
changes and maintain consistency
§ Provide regular feedback to AWS to include
our features in future versions of SDK
§ Built governance systems to enforce safe
use of AWS
AWS SDK Code access to configuration APIs
S3 Hosts node.js code invoked by Lambda
Lambda Execution environment
DynamoDB Store config status
CloudTrail Audit logs of all activity
Built Using

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Code Sample: Command Line Interface
user$ ./AmazonNodeCLI.js --updateCDNWeight www.cdn.amazon.com
--withWeights '{"cf":90,”zz":10,”Origin":0}’
Record set we
are updating
Weights set on
the routing policy

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Code Sample: Command Line Interface
AWS SDK
Reference
Route 53 object
and changes
Validation to ensure
100% of requests are
accounted for
Returns values just
updated to CLI for
manual confirmation

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Wins by Using AWS
§ Latency improvements and stability on CloudFront as opposed to the origin
§ Reduction in the cost of DDoS insurance policies with WAF
§ Consistent use of best-practices via the AWS SDK
§ Field Level Encryption of customer’s data before it reaches the origin
§ Same Shield Advanced experts help the Amazon consumer business
§ Robot mitigation with Lambda@Edge

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!