A Front-Row Seat to Ticketmaster’s Use of MongoDB


Published on

Ticketmaster is the world leader in selling tickets. After more than a decade of developing applications extensively on Oracle and MySQL, Ticketmaster made the move to MongoDB. The reasons for the move are generally in line with those of other companies – increased flexibility and performance, and decreased costs and time-to-market. In this session we’ll discuss how the conversion to MongoDB went at Ticketmaster and we’ll take a deeper dive into some of the successes and set-backs that we faced. We’ll give an overview of the MongoDB topology at Ticketmaster, discuss exactly what data we’re writing to MongoDB and comment on the MongoDB support model that we’re using. We’ll also touch on the transition from relational DBA to NoSQL DBA at Ticketmaster.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A Front-Row Seat to Ticketmaster’s Use of MongoDB

  1. 1. A Front-Row Seat to Ticketmaster’s use of MongoDB! Ed Presz – VP, Database Services Linda Xu – Director Database Development/Architecture Tuesday, June 24 1:30-2:30PM
  2. 2. Live Nation Entertainment The world's leading live entertainment and ecommerce company, comprised of four market leaders: Ticketmaster.com, Live Nation Concerts, Front Line Management Group and Live Nation Network Ticketmaster.com One of the world's top five ecommerce sites Database with 350 million customers/fans worldwide 27 million monthly unique visitors to www.ticketmaster.com and www.livenation.com Overview – Live Nation/Ticketmaster Ticketmaster and Live Nation merged in 2010 www.livenation.com > 15,000 tickets/minute sold during large on-sales Live Nation Concerts produces over 22,000 shows annually for more than 2,300 artists Ticketmaster was official ticket company for 2012 London Olympics 2
  3. 3. MongoDB at Ticketmaster 3 WriteConcern Replica Sets JSON/BSON getLastError collections documents election oplog
  4. 4. The Move to MongoDB Ticketmaster Database Services team 26 DBAs located globally Strong Oracle/MySQL/MS SQL Server background Consists of Operational DBAs, Architects and Database Engineers Database performance/scale is a huge part of what we do Transition to MongoDB Flexible, development friendly Ease of deployment/speed-to-market Cost savings Re-training relational DB experts (new challenge) Performance trade-offs
  5. 5. MongoDB at Ticketmaster February, 2012 Ticketmaster starts evaluating MongoDB June, 2012 Begin development on MongoDB May, 2013 Enterprise Subscription support for MongoDB July, 2013July, 2013 Go live with MongoDB 2.2.4 for Ticketmaster Resale March, 2014 Upgrade to MongoDB 2.4 (currently using 2.4.9) April, 2014 Reviewing MongoDB 2.6 Concurrency improvements in MongoDB 2.8, including document-level locking.
  6. 6. MongoDB at Ticketmaster 6"Humongous“ – huge , monstrous
  7. 7. Ticketmaster MongoDB use-cases MongoDB is used as a compliment to relational databases (MySQL and Oracle) Not used for highly transactional applications and/or sensitive customer/financial data Currently for TM+ (resale) Order details/history (secondary market)Order details/history (secondary market) Shopping cart and messages Configuration management Release tools Concerts-related services in the future Venue view (View from section) Event Content Service B2B (session recovery) Client Reports
  8. 8. Operational challenge 1 – MongoDB Security NoSQL security concerns Earlier versions of MongoDB (prior to 2.0), auth mode was not available (run in trusted environment). Big advancements in security with later versions of MongoDBversions of MongoDB Ticketmaster has enabled authentication By default authentication is disabled Edit your MongoDB configuration file to enable auth (auth = true) or specify a shared key file for replica sets MongoDB supports simple role-based authentication at the database level Only two permissions are available: read-only and read/write. Read/write is the default. Regular user setup is local to each database
  9. 9. Requirements: Implement MongoDB replica set that is highly redundant, fault tolerant, and highly available in 2 datacenters. Need to provide a solution that allows datacenter #1 to be brought off-line for routine maintenance. Operational challenge 2 – Geographically Distributed Replica Sets MongoDB Nodes: Red Hat Enterprise Linux Server release 5.8 (64-bit) XenSever VM Intel Xeon dual quad-core processors w/32 GB RAM
  10. 10. Operational challenge 2 – Geographically Distributed Replica Sets Step #1: Add Arbiter Step #2: Remove Arbiter Step #4: Shutdown Shift in primary New Primary Step #3: Shutdown
  11. 11. Moved Arbiter to third datacenter Operational challenge 2 – Geographically Distributed Replica Sets
  12. 12. Operational challenge 3 – Performance on NFS Performance on NFS 2 of 3 MongoDB outages with data rollback in production were cased by NFS mount issues Journal write performance is common bottleneck on NFS Journal file has direct disk mapping every 100ms (default) or more frequently MongoDB is sensitive to IO contentionMongoDB is sensitive to IO contention Compete for resource on back-end filer with other services on Live Nation system Use NFS to take snapshot/backups. Application is running 24*7, we can’t quiet writes to take consistent backup
  13. 13. Operational challenge 3 - Performance on NFS MongoDB writing to NETAPP filer.
  14. 14. Operational challenge 3 - Performance on NFS Solution Moved journal and data files to block device/local Storage Build extra secondary replica on each data center and run data and journal files on filer for (for snapshot/backup purposes) Notes We tried moving journal only to block devices with limitedWe tried moving journal only to block devices with limited performance gains. Block device has limited quota size. Extending storage is not easy. We added monitoring script to check space usage < 250GB Non-standard setup using symlinks to point to data files and journal files Extra nodes have priority set to 1 to have less chance of being promoted to primary
  15. 15. Operational challenge 3 - Performance on NFS MongoDB writing to block device/local storage
  16. 16. Operational challenge 3 - Performance on NFS Benefits • Internal testing showed 20x speedup over NFS v3 when we moved to local drive/block devices • Besides performance gains, also less page faults, less queuing and % locks • Development also confirmed improved performance
  17. 17. Other Operational challenges Understanding the write behavior ( WriteConcern vs performance) Write concern describes the guarantee on a write operation (i.e. the strength of the write) The weaker the WriteConcern, the faster the write operations Backups (mongodump vs NETAPP filer snapshots vs MMS) Delay replicate slavesDelay replicate slaves Adding indexes in MongoDB Create index for replica set on one member at a time Take nodes out of replica set to add indexes (bring up Mongo instance with different port#) Resizing the oplog Shrinking the oplog was tricky (documentation geared towards making oplog bigger) We built database with default (5% of available space) with default size for oplog. 130GB oplog at go-live.
  18. 18. MongoDB Support Enterprise Support 42 JIRA tickets open for MongoDB support with company = “Ticketmaster” Overall, good turnaround Sample JIRA tickets…. Setting up Arbiter documentation - still initializingSetting up Arbiter documentation - still initializing Resizing oplog does not seem to shrink the size of the local data files Very high CPU/iowait and long read/write locks on primary Master yesterday evening MongoDB query review
  19. 19. MongoDB Monitoring (MMS 1.4)
  20. 20. Let’s have some fun! • O2 Arena in London, U.K. • Phones 4u Arena in Manchester, U.K. • Barclays Center in Brooklyn, NY • Bon Jovi • Michael Jackson The Immortal World Tour • P!nk Top Tours of 2013
  21. 21. A Front-Row Seat to Ticketmaster’s use of MongoDB! Ed Presz – VP, Database Services, ed.presz@ticketmaster.com Linda Xu – Director Database Development and Architecture, linda.xu@ticketmaster.com Question and Answer!