SlideShare a Scribd company logo
1 of 16
From 6 Weeks to 6 Minutes… 
…Almost! 
Shakeel Sorathia
Ticketmaster 
• We are part of Live Nation Entertainment 
• You may have heard of us… 
– We sell a few tickets 
– 7 Data Centers worldwide 
– 20,000+ OS images (VM’s and bare metal) 
2 
– Transactions > $16B 
Worldwide 
– Onsales > $1M/minute 
– 255M+ user accounts
Here’s our story 
• Challenge 
Here’s what was going on backstage… 
• Solution 
On with the show… 
• Impact 
The reviews are in… 
3
Early Tactical Solutions 
Challenge 
4 
2001 
Management 
through M4 
2003 
Implemented 
virtualization 
in 
development 
environments 
2004 
Developed 
Spine (rubix) 
for CMS 
2005 
Scaled out 
150% for 
Yankee's 
onsale within 
48 hours
And then… 
Challenge 
5 
2008 
• Our split from IACI saddled us with $750M in debt 
• The economy forced a cost-driven mindset 
• Our initial proposal for internal cloud approved due to 
cost savings
We reached a turning point… 
Challenge 
6 
2010 
• Merged with Live Nation 
• Faced a key decision to turtle or grow; we chose 
growth! 
• The re-architecture of the platform began 
• This led to the development of a real DevOps strategy
We face a lot of uncertainty and 
potential pitfalls in an enterprise… 
There were a lot of questions to answer… 
7 
Fan 
Experience 
Employee 
Experience 
Dollars & 
Sense
Fan Experience 
• Will we be able to provide an 
equal or better product? 
• Will this result in faster time-to-market 
for our requested features? 
• Will our fans have to deal with 
problems while we figure things out? 
8
Employee Experience 
• How will people in the company 
adapt? 
• What type of training will they 
need? 
• How do we overcome fear in 
the minds of our people? 
• How do we share the vision for the future? 
9
Dollars and Sense 
• How much will this cost us? 
• What about learning pains, 
what will that cost? 
• Will our sales be better if we 
do this? 
• How will we know if this is working? 
10
Where we started 
Challenge 
• 17 different ticketing systems 
• 30 year old technology 
• Hard split between Development and 
Operations teams 
• Product release cycles were bi-monthly to 
quarterly 
11
Here’s what we did… 
1. Developed a 
tier 1 TOC 
Solution 
12 
2. Developed an 
SRE team 
3. Aligned Systems 
Engineering to 
delivery teams 
4. Empowered 
developers to push 
code to production
What has this achieved? 
Impact 
• 51% of all services/products are deployed 
by development teams 
• Development teams now have an on call 
• 109 Deployments in Sept 
• 490 Alerts resolved by TOC 
13
What has this achieved? 
Impact 
• Ticketmaster Resale is up 30% for first half 
of 2014 
• Increase in Customer Service Rating 
– In general, how would you rate the stability of 
Ticketmaster products/services? 
14 
YOY Increase in Excellent Rating 
0% 
5% 
24% 
30.00% 
20.00% 
10.00% 
0.00%
And it continues… 
• We strive for continuous refinement 
• We are pushing more services to the edge 
– Customer service tools 
– Client Services tools for event building 
• We continue to train people in new 
methodologies 
– Lean PMO 
– Agile (Scrum/Kanban) 
Impact 
15
Things we don’t know 
Impact 
• How many people should be aligned in 
Systems Engineering? 
• What should some of the responsibilities be 
for the different roles? 
• How does infosec fit into all of this? 
• What other metrics should we be looking at? 
16

More Related Content

Viewers also liked

Viewers also liked (6)

The Spotify Tribe
The Spotify TribeThe Spotify Tribe
The Spotify Tribe
 
DevOps, Microservices and containers - a high level overview
DevOps, Microservices and containers - a high level overviewDevOps, Microservices and containers - a high level overview
DevOps, Microservices and containers - a high level overview
 
Segregation of Duties and Continuous Delivery
Segregation of Duties and Continuous DeliverySegregation of Duties and Continuous Delivery
Segregation of Duties and Continuous Delivery
 
Microservices at Spotify
Microservices at SpotifyMicroservices at Spotify
Microservices at Spotify
 
Enabling Fast IT using Containers, Microservices and DevOps Model
Enabling Fast IT using Containers, Microservices and DevOps ModelEnabling Fast IT using Containers, Microservices and DevOps Model
Enabling Fast IT using Containers, Microservices and DevOps Model
 
How Spotify Builds Products (Organization. Architecture, Autonomy, Accountabi...
How Spotify Builds Products (Organization. Architecture, Autonomy, Accountabi...How Spotify Builds Products (Organization. Architecture, Autonomy, Accountabi...
How Spotify Builds Products (Organization. Architecture, Autonomy, Accountabi...
 

Similar to DOES14 - Shakeel Sorathia - Ticketmaster

Iwsm2014 dev ops measurements (amir arooni)
Iwsm2014   dev ops measurements (amir arooni)Iwsm2014   dev ops measurements (amir arooni)
Iwsm2014 dev ops measurements (amir arooni)
Nesma
 
Long view euc calgary april 26 2013 v0.3
Long view euc calgary april 26 2013 v0.3Long view euc calgary april 26 2013 v0.3
Long view euc calgary april 26 2013 v0.3
lvsmarketing
 

Similar to DOES14 - Shakeel Sorathia - Ticketmaster (20)

Softchoice: Why UC Projects Fail (and what to do about it)
Softchoice: Why UC Projects Fail (and what to do about it)Softchoice: Why UC Projects Fail (and what to do about it)
Softchoice: Why UC Projects Fail (and what to do about it)
 
IT In The Park 2016
IT In The Park 2016IT In The Park 2016
IT In The Park 2016
 
Growing Up The Right Way: An Example of How to Scale When You're Aren't a Sta...
Growing Up The Right Way: An Example of How to Scale When You're Aren't a Sta...Growing Up The Right Way: An Example of How to Scale When You're Aren't a Sta...
Growing Up The Right Way: An Example of How to Scale When You're Aren't a Sta...
 
Hands-on Agile: The Agility Assessment Framework Workshop
Hands-on Agile: The Agility Assessment Framework WorkshopHands-on Agile: The Agility Assessment Framework Workshop
Hands-on Agile: The Agility Assessment Framework Workshop
 
How Healthy is Your Marketo Instance?
How Healthy is Your Marketo Instance?How Healthy is Your Marketo Instance?
How Healthy is Your Marketo Instance?
 
Iwsm2014 dev ops measurements (amir arooni)
Iwsm2014   dev ops measurements (amir arooni)Iwsm2014   dev ops measurements (amir arooni)
Iwsm2014 dev ops measurements (amir arooni)
 
Overselling Agile Certifications and Frameworks : Presented by Sridharan Vembu
Overselling Agile Certifications and Frameworks : Presented by Sridharan VembuOverselling Agile Certifications and Frameworks : Presented by Sridharan Vembu
Overselling Agile Certifications and Frameworks : Presented by Sridharan Vembu
 
Engineering mindset fort corporate management
Engineering mindset fort corporate managementEngineering mindset fort corporate management
Engineering mindset fort corporate management
 
Notes from the field - End User Computing
Notes from the field - End User ComputingNotes from the field - End User Computing
Notes from the field - End User Computing
 
LTV prediction - How to save millions with ML
LTV prediction - How to save millions with MLLTV prediction - How to save millions with ML
LTV prediction - How to save millions with ML
 
Future-proof your office 365 digital workplace
Future-proof your office 365 digital workplaceFuture-proof your office 365 digital workplace
Future-proof your office 365 digital workplace
 
Adam Ochs - Office 365 Roadmap
Adam Ochs - Office 365 RoadmapAdam Ochs - Office 365 Roadmap
Adam Ochs - Office 365 Roadmap
 
Roadmap
RoadmapRoadmap
Roadmap
 
Teams - There's no place like home
Teams - There's no place like homeTeams - There's no place like home
Teams - There's no place like home
 
Long view euc calgary april 26 2013 v0.3
Long view euc calgary april 26 2013 v0.3Long view euc calgary april 26 2013 v0.3
Long view euc calgary april 26 2013 v0.3
 
Performance Support: Enabling the Most Valuable Resource Within Your Organiza...
Performance Support: Enabling the Most Valuable Resource Within Your Organiza...Performance Support: Enabling the Most Valuable Resource Within Your Organiza...
Performance Support: Enabling the Most Valuable Resource Within Your Organiza...
 
How deploy a KM platform in an international context
How deploy a KM platform in an international contextHow deploy a KM platform in an international context
How deploy a KM platform in an international context
 
Confused CMS Presentation - Internet World London 2011 #iwexpo. Delivered on...
Confused CMS Presentation - Internet World London 2011 #iwexpo.  Delivered on...Confused CMS Presentation - Internet World London 2011 #iwexpo.  Delivered on...
Confused CMS Presentation - Internet World London 2011 #iwexpo. Delivered on...
 
Agile Mumbai 2019 Conference | Understanding the ask for Agility from a Digit...
Agile Mumbai 2019 Conference | Understanding the ask for Agility from a Digit...Agile Mumbai 2019 Conference | Understanding the ask for Agility from a Digit...
Agile Mumbai 2019 Conference | Understanding the ask for Agility from a Digit...
 
Setting the Customer's Journey: Walk a Mile In Your Customer's Shoes
Setting the Customer's Journey: Walk a Mile In Your Customer's ShoesSetting the Customer's Journey: Walk a Mile In Your Customer's Shoes
Setting the Customer's Journey: Walk a Mile In Your Customer's Shoes
 

More from DevOps Enterprise Summmit

DOES14 - Justin Arbuckle - CHEF - Hunting the DevOps Whale
DOES14 - Justin Arbuckle - CHEF - Hunting the DevOps Whale DOES14 - Justin Arbuckle - CHEF - Hunting the DevOps Whale
DOES14 - Justin Arbuckle - CHEF - Hunting the DevOps Whale
DevOps Enterprise Summmit
 

More from DevOps Enterprise Summmit (6)

DOES14 - Justin Arbuckle - CHEF - Hunting the DevOps Whale
DOES14 - Justin Arbuckle - CHEF - Hunting the DevOps Whale DOES14 - Justin Arbuckle - CHEF - Hunting the DevOps Whale
DOES14 - Justin Arbuckle - CHEF - Hunting the DevOps Whale
 
DOES14 - Courtney Kissler - Nordstrom - Transforming to a Culture of Continuo...
DOES14 - Courtney Kissler - Nordstrom - Transforming to a Culture of Continuo...DOES14 - Courtney Kissler - Nordstrom - Transforming to a Culture of Continuo...
DOES14 - Courtney Kissler - Nordstrom - Transforming to a Culture of Continuo...
 
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
DOES14 - Gary Gruver - Macy's - Transforming Traditional Enterprise Software ...
 
DOES14 - Gene Kim and Steve Brodie - Tuesday Opening Remarks
DOES14 - Gene Kim and Steve Brodie - Tuesday Opening Remarks DOES14 - Gene Kim and Steve Brodie - Tuesday Opening Remarks
DOES14 - Gene Kim and Steve Brodie - Tuesday Opening Remarks
 
DOES14 - David Ashman, Blackboard Learn - Keep Your Head in the Clouds Tuesda...
DOES14 - David Ashman, Blackboard Learn - Keep Your Head in the Clouds Tuesda...DOES14 - David Ashman, Blackboard Learn - Keep Your Head in the Clouds Tuesda...
DOES14 - David Ashman, Blackboard Learn - Keep Your Head in the Clouds Tuesda...
 
DOES14: Scott Prugh, CSG - DevOps and Lean in Legacy Environments
DOES14: Scott Prugh, CSG - DevOps and Lean in Legacy EnvironmentsDOES14: Scott Prugh, CSG - DevOps and Lean in Legacy Environments
DOES14: Scott Prugh, CSG - DevOps and Lean in Legacy Environments
 

Recently uploaded

JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
Max Lee
 

Recently uploaded (20)

Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
Sourcing Success - How to Find a Clothing Manufacturer
Sourcing Success - How to Find a Clothing ManufacturerSourcing Success - How to Find a Clothing Manufacturer
Sourcing Success - How to Find a Clothing Manufacturer
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
What is an API Development- Definition, Types, Specifications, Documentation.pdf
What is an API Development- Definition, Types, Specifications, Documentation.pdfWhat is an API Development- Definition, Types, Specifications, Documentation.pdf
What is an API Development- Definition, Types, Specifications, Documentation.pdf
 
Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
SQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionSQL Injection Introduction and Prevention
SQL Injection Introduction and Prevention
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 

DOES14 - Shakeel Sorathia - Ticketmaster

  • 1. From 6 Weeks to 6 Minutes… …Almost! Shakeel Sorathia
  • 2. Ticketmaster • We are part of Live Nation Entertainment • You may have heard of us… – We sell a few tickets – 7 Data Centers worldwide – 20,000+ OS images (VM’s and bare metal) 2 – Transactions > $16B Worldwide – Onsales > $1M/minute – 255M+ user accounts
  • 3. Here’s our story • Challenge Here’s what was going on backstage… • Solution On with the show… • Impact The reviews are in… 3
  • 4. Early Tactical Solutions Challenge 4 2001 Management through M4 2003 Implemented virtualization in development environments 2004 Developed Spine (rubix) for CMS 2005 Scaled out 150% for Yankee's onsale within 48 hours
  • 5. And then… Challenge 5 2008 • Our split from IACI saddled us with $750M in debt • The economy forced a cost-driven mindset • Our initial proposal for internal cloud approved due to cost savings
  • 6. We reached a turning point… Challenge 6 2010 • Merged with Live Nation • Faced a key decision to turtle or grow; we chose growth! • The re-architecture of the platform began • This led to the development of a real DevOps strategy
  • 7. We face a lot of uncertainty and potential pitfalls in an enterprise… There were a lot of questions to answer… 7 Fan Experience Employee Experience Dollars & Sense
  • 8. Fan Experience • Will we be able to provide an equal or better product? • Will this result in faster time-to-market for our requested features? • Will our fans have to deal with problems while we figure things out? 8
  • 9. Employee Experience • How will people in the company adapt? • What type of training will they need? • How do we overcome fear in the minds of our people? • How do we share the vision for the future? 9
  • 10. Dollars and Sense • How much will this cost us? • What about learning pains, what will that cost? • Will our sales be better if we do this? • How will we know if this is working? 10
  • 11. Where we started Challenge • 17 different ticketing systems • 30 year old technology • Hard split between Development and Operations teams • Product release cycles were bi-monthly to quarterly 11
  • 12. Here’s what we did… 1. Developed a tier 1 TOC Solution 12 2. Developed an SRE team 3. Aligned Systems Engineering to delivery teams 4. Empowered developers to push code to production
  • 13. What has this achieved? Impact • 51% of all services/products are deployed by development teams • Development teams now have an on call • 109 Deployments in Sept • 490 Alerts resolved by TOC 13
  • 14. What has this achieved? Impact • Ticketmaster Resale is up 30% for first half of 2014 • Increase in Customer Service Rating – In general, how would you rate the stability of Ticketmaster products/services? 14 YOY Increase in Excellent Rating 0% 5% 24% 30.00% 20.00% 10.00% 0.00%
  • 15. And it continues… • We strive for continuous refinement • We are pushing more services to the edge – Customer service tools – Client Services tools for event building • We continue to train people in new methodologies – Lean PMO – Agile (Scrum/Kanban) Impact 15
  • 16. Things we don’t know Impact • How many people should be aligned in Systems Engineering? • What should some of the responsibilities be for the different roles? • How does infosec fit into all of this? • What other metrics should we be looking at? 16

Editor's Notes

  1. Speak my name slowly.
  2. This is what we were doing to set us up early on. Number of tactical solutions that were put into place because of many of the issues that we had. Yankee fans are a little bit on the crazy side. 80 games, 70k tickets per game. Took 8 hours to sell through about 1.5 mil tickets. Define rubix a bit better.
  3. Talk about the lack of innovation between 2008-2010. No money for innovation, all the money went to the cloud project. Cloud ROI was going to save money compared to running bare metal servers.
  4. Why did we face the decisions (2008 economy, new leadership wasn’t sure what to do) Springsteen lawsuit just settled which made resale a difficult line of business for us. Talk through the lawsuit (redirected to ticketsnow (our resale site), had to refund over $1M) Devops started with jetson (the new platform)
  5. 17 ticketing systems, M&A activity, great on the A, not so much on the M. Development would build the product and then throw it over the wall to operations
  6. How did we chose the place to start. When did we start doing this? 1.5 – 2 years ago.
  7. Number of releases 2 years ago. Number of releases today. Time to do deployments. Old products are still taking a long time to release. New products are released far quicker One service is doing 15 deployments per month 490 * 5 minutes is 40 hours (1 man week)
  8. Continuous refine Pushing more services to the edge Continue to train people in new methodologies