Ticketmaster has been using Splunk for around 5 years to gain visibility into their operations and customer usage. They ingest over 1.6TB of log data per day from their various systems like web servers, databases, and applications. Splunk is used for monitoring systems and applications, troubleshooting issues, and providing analytics on customer search patterns and event demand. It has helped Ticketmaster improve customer service, protect against ticket brokers, and increase revenue by better understanding concert demand.

1. Copyright © 2014 Splunk Inc.
Splunk @
TicketMaster

2. 2
Shakeel Sorathia
VP Systems Engineering

3. 3
Ticketmaster Overview
• We are part of Live Nation Entertainment
• You may have heard of us…
• We sell a few tickets
• 7 data centers worldwide
• 20,000+ OS images (VMs and bare metal)
• Transactions > $16B worldwide
• Onsales > $1M/minute
• 255M+ user accounts

4. 4
About Me
• Been with Ticketmaster for almost 10
years
• Responsible for the infrastructure and
site operations
• My name is pretty well known (but for
different reasons)
• Splunk, because ninjas are too busy!

5. 5
How We Got Started
Adopted Splunk for ~5 years
– Before Splunk: Lots of ‘grep’ping
– Lack of understanding of what was going on with web properties
– Very time consuming to try and troubleshoot issues
Over the last 2 years, numerous new software launches demanding
– Visibility into usage, performance, availability for engineering and
development
– View of blocks vs open reservations, comparison of popularity of events,
planning and predicting for high-volume events

6. 6
Splunk at Ticketmaster Today
• Keep applications and operations running – 900 users organization-wide
Monitoring ticket process for failures
Monitor Splunk NOC dashboards for capacity problems, availability issues,
forensics
Transaction tracing, counts, durations, failed transactions
• Provide analytics to product managers/business owners
What is the response to new events? High enough to create more similar
events at different venues?
Are we experiencing too much block activity – potential illegal resale?
What are people searching for the most?

7. 7
Splunk at Ticketmaster
1.6TB/day
16 indexers
~2000 forwarders
80 indexes with many data types:
.Net, Apache, JBOSS, weblogic, Java, Perl,
python, C, C++ application logs
100s of applications across 17
different ticketing systems
Apps used:
Exchange, AD, NetApp E-series,
SoS
Offload search load to Splunk search heads
Auto load-balanced forwarding to Splunk indexers
Send data from thousands of servers using any combination of Splunk forwarders

8. 8
Developer Guidelines
• Ticketmaster does 120 deployments/month
• Splunk used to correlate production issues with new
release deployments
• Developers given logging guidelines initially:
- Key value pairs in logs helps faster on boarding, greater
visibility when the code is in production
- Shorten variable names for ease of use (moving towards this)
- Selling to developers is critical for fast problem solving

9. 9
Splunk Powering Our Operations
Problem
Customer service call from ticket buyers not receiving their tickets
via mail results in hours of grepping email logs to verify claims
Reduce Time to
Resolution
By putting mail logs in Splunk and creating an app for customer
service, we could fix email issues quickly!
Problem
Correlated monitoring of the application stack was not possible;
monitoring was siloed across environments, it took minutes (an
eternity) to discover errors
End-to-End Insights
“The EOS Splunk dashboard represented a quantum leap in the information
available to us during ‘on sale’ with real-time monitoring and real-time
analysis of what is happening on the website and our hosts systems”
an internal customer

10. 10
Splunk Powering Our Operations And The
Business
Problem
Ticket broker automation was blocking access to open inventory
for ticket fans
Secure Inventory
By using Splunk we protected inventory by responding quickly to
evolving broker automation tactics
Problem
Inability to track concert demand in real time resulted in risks
associated with adding new shows: either undersold new shows or
didn't add shows and left money on the table
Increase Revenue
By having up-to-the-minute information ticket demand we can
respond quickly by adding new shows based on actual demand
metrics, resulting in more revenue!

11. 11
Screenshots
Screenshot here

12. 12
Best Practice Recommendations
• Think through log formatting as you through your
deployments – better logs accelerate success
• Define who owns which data – will help clarify what the use
is
• Plan to scale – its viral, everyone jumps on it, plan for
success
• Infrastructure
• Semantic logging (use short variable names!)

13. 13
What’s Next
• Greater use of Splunk across the enterprise
• Making Splunk accessible via mobile devices
• Splunk access to customer support for self-service
resolution
• Democratize data: more self-service access, more
correlation use cases

14. Thank You