AWS customer Southeastern Pennsylvania Transportation Authority (SEPTA) worked with a partner to build a new website on AWS to sell 350,000 train tickets for the Pope’s visit to Philadelphia in September 2015. The site launched Monday morning on July 20 at 9:00 a.m. and was quickly overwhelmed by visitors. By 11:00 a.m. that day, ticket sales were postponed indefinitely with only 28 tickets having been sold. After the failure, AWS was engaged to help develop a solution. This session reviews what caused the problems and also explores the creative solutions that were used to make the second attempt at ticket sales a success.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mike Kuentz, Senior Solutions Architect, AWS
Mike Zaleski, Director Emerging and Specialty Technology, SEPTA
Tim Raybould, CEO Ticketleap
June 21, 2016
Helping SEPTA with the Pope’s Visit
to Philadelphia

2. Agenda
• Review of events leading up to Papal visit
• Not all technical problems are best solved with technical
solutions
• Details of AWS Infrastructure Event Management (IEM)
• How leveraging partners can help customers
• The importance of load testing

3. Background
 Pope Francis plans visit to Philadelphia Friday,
September 25, 2015
 Estimates put over 1 million people coming to the
city for his visit
 Strict security perimeter was in place in downtown
Philadelphia
 To manage the traffic
SEPTA had to limit train
tickets to only 350,000

4. Pilgrims

5. Transit map vs. event map

6. Ticket sales website requirements
• Sell tickets on first come, first serve basis, no lottery
• No “take a number” queuing systems allowed
• No third-party e-commerce (eBay, PayPal, etc.)
• Complex rules regarding number of tickets allowed for
various stations
• Requirements were based on the premise that tickets
will sell over the course of a week—not in under an hour

7. Ticket sales website 1.0

8. Reality by the numbers
• Considerable media frenzy leading up to site opening at 9 a.m.
• 54,000 visits in the first minute
• 900,000 visits in the first 10 minutes
• ~1,700 transactions per second
• By 11 a.m., ticket sales were suspended, because the architecture
could not meet the load

9. The Internet responds

10. Review of first architecture
• Areas of concern
• Redis servers running on t2.small
• Running out of MySQL connections
• As number of connections increases, memory increases as well
• No read replicas
• Drupal writing log info into the same DB
• Tightly coupled design that had to send an email through
Amazon SES before transaction completed
• Amazon SES limit had not been raised
• Load testing didn’t match demand

11. InfrastructureCost
Time
Periodic Large
Capital Expenditures
Opportunity
Cost
Predicted
Demand
Traditional
Data Center
Model
Actual
Demand
Cloud
Computing
Shortage:
Unable to serve
customers
Scale on demand

12. How to scale?
• Scaled MySQL vertically and horizontally
• Scaled Redis vertically and horizontally
• Removed bottlenecks in logic
• Removed bottlenecks in code

13. Load testing
900 transactions per second
(We saw 1,700 transactions per second!)

14. Requirements are forced to change
 Move to a lottery based system

15. Lottery based system
 Researched third parties (Ticketmaster, Eventbrite, etc.)—
they all refused given the original requirements
 Engage AWS IEM and Ticketleap
 Open access for 24 hours
 De-duplicate entries and fraud checks
 Lottery prevents/limits scalpers
Get rid of all the servers!

17. • AWS customer
• Lightweight online ticketing platform used by thousands
of event organizers across the US and Canada
• Case study available
• https://aws.amazon.com/solutions/case-studies/ticketleap/

18. Review of second architecture
Amazon SQS Amazon SESAmazon
S3
AWS
Lambda
Amazon
CloudFront

19. S3 statistics
Amazon S3

20. Request rate and performance considerations
• If your workload in an Amazon S3 bucket routinely
exceeds 100 PUT/LIST/DELETE requests per second
• Talk to us!
• Avoid sequential patterns in key names
• examplebucket/2013-26-05-15-00-00/cust1234234/photo1.jpg
• examplebucket/2013-26-05-15-00-00/cust3857422/photo2.jpg
• ...
• examplebucket/2013-26-05-15-00-01/cust1248473/photo4.jpg
• examplebucket/2013-26-05-15-00-01/cust1248473/photo5.jpg

21. Review of second architecture
Amazon SQS Amazon SESAmazon
S3
AWS
Lambda
Amazon
CloudFront

22. Results
• Load testing
• c4.8xlarge instance and The Grinder
• POST 200,000 entries in 85 seconds (~2300 TPS)

23. AWS Infrastructure Event Management
• Initiation
• Planning and execution
• Review and closure
• https://aws.amazon.com/premiumsupport/iem/

24. IEM success stories

26. Thank you!