Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Drinking Our Own Champagne:
How Woot, an Amazon subsidiary, uses AWS technologies
Vivek Sagi, CTO, Woot - vsagi@woot.com

...
Introduction to Woot
Woot Trivia (Question 1)
Q: What is Woot?
A. A rabid monkey only found in the Amazon jungle
B. A daily deal site, flash si...
Woot Trivia (Question 2)
Q: Name the event on the Woot website where we sell a
random collection of leftover inventory and...
Woot Trivia (Question 3)
Q: When was Woot acquired by Amazon?
A. 2004
B. 2008
C. 2010

D. None of the above because Woot i...
Ye Olde Woot

11/15/2013

Amazon Confidential

6
Modern Day Woot
Growth of
new
categories
Introduction
of Woot!
Plus flash
sales

11/15/2013
11/15/2013

Amazon Confidentia...
It’s All About the Benjamins
Woot’s technology stack
Woot (Pre-Acquisition)

S3
EC2

• Traditional co-location (since 2003)
– 8 servers
– Monolithic application

• Amazon S3 f...
Woot (Post-Acquisition)

S3
EC2

• Migrated to Amazon VPC (2011)
–
–
–
–
–

2 months to prep VPC
VPN to colo (for secure d...
Application Rewrite in 2012
• Business Goals
– Support multiple deal types (Daily, Flash, VIP)
– Changes must be visible q...
Woot Current Technologies
• Core Technologies
–
–
–
–

C#, ASP.Net, MVC4, WCF,
MongoDB (All Content), SQL Server (Transact...
Woot’s AWS footprint
AWS Footprint
S3

EC2

ELB

RDS

SQS

CloudSearch

Route 53

CloudFront

DynamoDB

SES

VPC

IAM
Network Infrastructure
• Multi-node VPN
• 3 accounts
• 3 environments

Staging

Production

Development
Monitoring
• System Center + AWS Management Pack
– .Net aware application monitoring
– Ties to external monitors
– Correla...
Monitoring
• System Center + AWS Management Pack
• OpenNMS
– Service discovery
– SNMP integration
– Multi-AZ perspective
Monitoring
• System Center + AWS Management Pack
• OpenNMS
• StatsD + Graphite + Skyline
– Great dashboards – Measure ever...
Monitoring
•
•
•
•

System Center + AWS Management Pack
OpenNMS
StatsD + Graphite + Skyline
Amazon CloudWatch
– Most direc...
Monitoring
•
•
•
•
•

System Center + AWS Management Pack
OpenNMS
StatsD + Graphite + Skyline
Amazon CloudWatch
Secret Sau...
Woot SOA Overview

SG – Retail
Public

SG – Retail
Private

SG - Catalog
Public

SG - Catalog
Private

SG – Content
Public...
SOA Benefits for Woot
•
•
•
•
•

Scalability
Organizational flexibility
Security
Faster time to market
Improved reliabilit...
How do we choose a technology?
Woot Throwdown Tenets
•
•
•
•

Don’t make assumptions; all options on the table
AWS is not the default choice
Simulate rea...
Throw Down 1: Load Balancers
Contenders

Pros

Cons

Varnish

Flexibility

No SSL

NginX

Lightweight and fast

No HTTP1/1...
Throw Down 2: Cart Persistence Storage
Woot “Cart Service”
– Simple

– Host a bag of crap

– Fast

– Scalable to follow gr...
Cart Persistent Storage
Document Databases

Contenders
Client API
High availability
Performance
Ramp-up time
Tools
Cost
Op...
Cart Persistent Storage
Document Databases

Criteria
Client API
High availability
Performance
Ramp-up time
Tools
Cost
Oper...
Cart Persistent Storage
Document Databases

Relational Databases

DynamoDB
Client API
High availability
Performance
Ramp-u...
Cart Throw Down Results
- MongoDB and DynamoDB were our finalists
- DynamoDB outperformed MongoDB in our

performance test...
Cart Persistent Storage: Cost Analysis
Cost ($/Month)
3500
3000
2500
2000
1500
1000
500
0
DynamoDB

MongoDB

Oracle

RDS

...
Cart Persistent Storage: Performance
Operations / second
10000
2060

1348

1000
100

64

107
42

70

10
1

Add Items

Get
...
DynamoDB – Rough Edges
• Only basic index support (primary, secondary)
– Use searching technologies such as Amazon CloudSe...
Cart in Production
•
•
•
•
•

20,000,000,000 requests served
Sub 10ms response times
0 errors reported since launch
No mai...
Recovering from a cloud outage
Triage of an outage: July 2, 2012
• Massive Storm + Maintenance Problem =
No Power in an Availability Zone
• Availability ...
Lessons Learnt
• Data integrity means more Availability Zones
– Mirror/replicate to other Availability Zones
– Lots of hos...
Lessons Learnt
• Data integrity means more Availability Zones
• Queue everything
–
–
–
–

Decouple systems
Save state info...
Lessons Learnt
• Data integrity means more Availability Zones
• Queue everything
• Hosts are disposable
–
–
–
–

Don’t sto...
Lessons Learnt
•
•
•
•

Data integrity means more Availability Zones
Queue everything
Hosts are disposable
Unique solution...
Final thoughts on the cloud
Security
• Same problems as anywhere else
– Centralize logging
– Centralize accounts

• Application-level logging
– Gain v...
Best Practices
• CloudSearch Best Practices
– Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
Best Practices
• CloudSearch Best Practices
– Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
...
Best Practices
• CloudSearch Best Practices
– Don’t make too many “cs-describe-domain” calls.
– Perform updates in batches...
Best Practices
• CloudSearch Best Practices
– Don’t make too many “cs-describe-domain” calls
– Perform updates in batches
...
Woot’s AWS Experience
• Tools continue to improve
- AWS Dashboard
- Expanded CloudWatch
- AWS Trusted Advisor - Premium Su...
Please give us your feedback on this
presentation

ARC212
As a thank you, we will select prize
winners daily for completed...
Upcoming SlideShare
Loading in …5
×

Drinking our own Champagne: How Woot, an Amazon subsidiary, uses AWS (ARC212) | AWS re:Invent 2013

5,778 views

Published on

Woot, an Amazon subsidiary, specializes in offering great new product deals every day. Woot's deeply discounted deals; and signature events like the 'Woot Off 'and 'Bag of Crap' sales launch at specific times throughout the day, and the resulting spiky traffic patterns are highly correlated to revenue.
In this session, we offer an unvarnished perspective into how Woot uses services such as Amazon DynamoDB, EC2, ELB, CloudSearch, CloudFront, and SES. Learn how to architect for security and PCI for a retail website running on AWS. Dig into the technical details of a data-store comparison between DynamoDB, Mongo, Oracle, and SQLServer, to find the right solution for unique workloads. Join us as we share our musings and real-lessons learned from using a cocktail of AWS services. We encourage you to attend even if none of this makes sense or is interesting. Don't miss the opportunity to hang out with Mortimer the Woot monkey and his crew and to walk away with one of our legendary flying monkeys.

Published in: Technology
  • Be the first to comment

Drinking our own Champagne: How Woot, an Amazon subsidiary, uses AWS (ARC212) | AWS re:Invent 2013

  1. 1. Drinking Our Own Champagne: How Woot, an Amazon subsidiary, uses AWS technologies Vivek Sagi, CTO, Woot - vsagi@woot.com Dan Pinkard, Systems Manager, Woot – dpinkard@woot.com November 14, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. Introduction to Woot
  3. 3. Woot Trivia (Question 1) Q: What is Woot? A. A rabid monkey only found in the Amazon jungle B. A daily deal site, flash site and vibrant deal community C. The name of a dance performed by hungry SDEs D. A retail website selling toys 11/15/2013 Amazon Confidential 3
  4. 4. Woot Trivia (Question 2) Q: Name the event on the Woot website where we sell a random collection of leftover inventory and “surprises”? A. Bunch of Carrots No Canadians were harmed in the making of this presentation B. Bitter Old Canadians C. Baldness on Curly D. Bag of Crap 11/15/2013 Amazon Confidential 4
  5. 5. Woot Trivia (Question 3) Q: When was Woot acquired by Amazon? A. 2004 B. 2008 C. 2010 D. None of the above because Woot is only a figment of a six year old’s imagination 11/15/2013 Amazon Confidential 5
  6. 6. Ye Olde Woot 11/15/2013 Amazon Confidential 6
  7. 7. Modern Day Woot Growth of new categories Introduction of Woot! Plus flash sales 11/15/2013 11/15/2013 Amazon Confidential Amazon Confidential 7 7
  8. 8. It’s All About the Benjamins
  9. 9. Woot’s technology stack
  10. 10. Woot (Pre-Acquisition) S3 EC2 • Traditional co-location (since 2003) – 8 servers – Monolithic application • Amazon S3 for images from 2006 • Woot Deals launch in 2009 – Amazon EC2 hosted (public) – SimpleDB MongoDB + SOLR – Amazon SQS to relay to Woot Retail ELB SQS
  11. 11. Woot (Post-Acquisition) S3 EC2 • Migrated to Amazon VPC (2011) – – – – – 2 months to prep VPC VPN to colo (for secure data migration) P2V for SQL servers Migrate MongoDB replicas 18 hours to move production services ELB SQS Route 53 CloudFront SES VPC RDS • Went from 8 servers to 20
  12. 12. Application Rewrite in 2012 • Business Goals – Support multiple deal types (Daily, Flash, VIP) – Changes must be visible quickly • Technology Goals – – – – Services-oriented architecture Improve site reliability Optimize for cloud deployments Language and platform neutral
  13. 13. Woot Current Technologies • Core Technologies – – – – C#, ASP.Net, MVC4, WCF, MongoDB (All Content), SQL Server (Transactional), DynamoDB (Shopping Cart) Memcached RabbitMQ • Reporting & Operations Technologies – – – – – Node.JS, Perl, Python, Ruby, PowerShell Durango, Rails Oracle, MySQL Redis Bind9, Postfix
  14. 14. Woot’s AWS footprint
  15. 15. AWS Footprint S3 EC2 ELB RDS SQS CloudSearch Route 53 CloudFront DynamoDB SES VPC IAM
  16. 16. Network Infrastructure • Multi-node VPN • 3 accounts • 3 environments Staging Production Development
  17. 17. Monitoring • System Center + AWS Management Pack – .Net aware application monitoring – Ties to external monitors – Correlates security and system events
  18. 18. Monitoring • System Center + AWS Management Pack • OpenNMS – Service discovery – SNMP integration – Multi-AZ perspective
  19. 19. Monitoring • System Center + AWS Management Pack • OpenNMS • StatsD + Graphite + Skyline – Great dashboards – Measure everything – High resolution – Flexible graphing options
  20. 20. Monitoring • • • • System Center + AWS Management Pack OpenNMS StatsD + Graphite + Skyline Amazon CloudWatch – Most direct source of truth – Systems performance and usage metrics
  21. 21. Monitoring • • • • • System Center + AWS Management Pack OpenNMS StatsD + Graphite + Skyline Amazon CloudWatch Secret Sauce(s) – Graphite as graphing platform – Integrate Amazon CloudWatch – Track AMPQ stats
  22. 22. Woot SOA Overview SG – Retail Public SG – Retail Private SG - Catalog Public SG - Catalog Private SG – Content Public SG - Content Private SG – Users Public SG – Users Private CloudFront Public
  23. 23. SOA Benefits for Woot • • • • • Scalability Organizational flexibility Security Faster time to market Improved reliability
  24. 24. How do we choose a technology?
  25. 25. Woot Throwdown Tenets • • • • Don’t make assumptions; all options on the table AWS is not the default choice Simulate real-world behavior Compare as many facets as possible - Driver support - Resiliency - Performance - Support - Scalability - Cost
  26. 26. Throw Down 1: Load Balancers Contenders Pros Cons Varnish Flexibility No SSL NginX Lightweight and fast No HTTP1/1 to origin Apache HTTP1/1 in/out Hard to tune Elastic Load Balancing Cloud Aware Limited logging
  27. 27. Throw Down 2: Cart Persistence Storage Woot “Cart Service” – Simple – Host a bag of crap – Fast – Scalable to follow growth – Resilient – Lots of small data facts – Secure patterns
  28. 28. Cart Persistent Storage Document Databases Contenders Client API High availability Performance Ramp-up time Tools Cost Operational management Debugging Ad-hoc queries (analytics) Support Documentation Total Relational Databases DynamoDB MongoDB Oracle SQL Server 4 5 2 5 5 5 2 5 5 4 -- -- 3 5 1 2 4 4 4 5 5 1 3 3 5 4 3 3 4 5 5 5 3 4 5 5 5 5 4 4 5 5 5 5 48 47 -- --
  29. 29. Cart Persistent Storage Document Databases Criteria Client API High availability Performance Ramp-up time Tools Cost Operational management Debugging Ad-hoc queries (analytics) Support Documentation Total Relational Databases DynamoDB MongoDB Oracle SQL Server 4 5 2 5 5 5 2 5 5 4 -- -- 3 5 1 2 4 4 4 5 5 1 3 3 5 4 3 3 4 5 5 5 3 4 5 5 5 5 4 4 5 5 5 5 48 47 -- --
  30. 30. Cart Persistent Storage Document Databases Relational Databases DynamoDB Client API High availability Performance Ramp-up time Tools Cost Operational management Debugging Ad-hoc queries (analytics) Support Documentation Total MongoDB Oracle SQL Server 4 5 2 5 5 5 2 5 5 4 -- -- 3 5 1 2 4 4 4 5 5 1 3 3 5 4 3 3 4 5 5 5 3 4 5 5 5 5 4 4 5 5 5 5 48 47 -- --
  31. 31. Cart Throw Down Results - MongoDB and DynamoDB were our finalists - DynamoDB outperformed MongoDB in our performance tests - We could launch and scale DynamoDB with a button push - We could host a bag of crap
  32. 32. Cart Persistent Storage: Cost Analysis Cost ($/Month) 3500 3000 2500 2000 1500 1000 500 0 DynamoDB MongoDB Oracle RDS SQL Server
  33. 33. Cart Persistent Storage: Performance Operations / second 10000 2060 1348 1000 100 64 107 42 70 10 1 Add Items Get DynamoDB MongoDB Update Items
  34. 34. DynamoDB – Rough Edges • Only basic index support (primary, secondary) – Use searching technologies such as Amazon CloudSearch to index data • Ad-hoc query support not so simple – Export data we want to query into a data warehouse
  35. 35. Cart in Production • • • • • 20,000,000,000 requests served Sub 10ms response times 0 errors reported since launch No maintenance No SLA misses
  36. 36. Recovering from a cloud outage
  37. 37. Triage of an outage: July 2, 2012 • Massive Storm + Maintenance Problem = No Power in an Availability Zone • Availability Zone hosted Woot’s primary SQL and AD role masters • Amazon EBS failures prevented recovery • AD DNS collapsed
  38. 38. Lessons Learnt • Data integrity means more Availability Zones – Mirror/replicate to other Availability Zones – Lots of hosts for MongoDB – Dedicated hosts for irregular patterns • Backups • Long-running queries • Specialized indexes
  39. 39. Lessons Learnt • Data integrity means more Availability Zones • Queue everything – – – – Decouple systems Save state information Inspection of service operation Recoverable
  40. 40. Lessons Learnt • Data integrity means more Availability Zones • Queue everything • Hosts are disposable – – – – Don’t store unique data Push-button launches Centralize configuration Dynamic role assignment
  41. 41. Lessons Learnt • • • • Data integrity means more Availability Zones Queue everything Hosts are disposable Unique solutions – Different failure modes than traditional data center – Combine off-the shelf items for resiliency
  42. 42. Final thoughts on the cloud
  43. 43. Security • Same problems as anywhere else – Centralize logging – Centralize accounts • Application-level logging – Gain visibility – Follow state • Use application tiers – SOA separation
  44. 44. Best Practices • CloudSearch Best Practices – Don’t make too many “cs-describe-domain” calls – Perform updates in batches
  45. 45. Best Practices • CloudSearch Best Practices – Don’t make too many “cs-describe-domain” calls – Perform updates in batches • EC2+EBS Best Practices – Deploy EBS-optimized hosts for required bandwidth – Provisioned IOPs to ensure lower latency for disks
  46. 46. Best Practices • CloudSearch Best Practices – Don’t make too many “cs-describe-domain” calls. – Perform updates in batches • EC2+EBS Best Practices – Deploy EBS Optimized hosts for required bandwidth – Provisioned IOPs to ensure lower latency for disks
  47. 47. Best Practices • CloudSearch Best Practices – Don’t make too many “cs-describe-domain” calls – Perform updates in batches • EC2+EBS Best Practices – EBS-optimized hosts for more bandwidth – Provisioned IOPs to ensure lower latency • Scaling Best Practices – No reward for using too much or too little
  48. 48. Woot’s AWS Experience • Tools continue to improve - AWS Dashboard - Expanded CloudWatch - AWS Trusted Advisor - Premium Support • AWS advocates for users • Capacity really is elastic • The more we see, the better it looks
  49. 49. Please give us your feedback on this presentation ARC212 As a thank you, we will select prize winners daily for completed surveys!

×