EVOLUTION OF
TICKETMASTER’S
JOURNEY TO DEVOPS
WINOPS LDN SEPTEMBER 2017
#WinOps
@connonm
@tmTech
WHO AM I & WHY AM I HERE?
Connon MacRae
VP, Technical Operations
(International)
#WinOps
@connonm
@tmTech
HISTORY IS USEFUL
CONTEXT IS EVERYTHING
DON’T BE AFRAID
MICROSOFT VS LINUX
MICROSOFT LINUX
NO MORE CYGWIN ON WINDOWS#WinOps
@connonm
@tmTech
#WinOps
@connonm
@tmTech
WHO IS TICKETMASTER?
LIVE NATION ENTERTAINMENT – 42 MARKETS
Live Nation & Ticketmaster
Live Nation ONLY
Ticketmaster ONLY
#WinOps
@connonm
@tmTech
#WinOps
@connonm
@tmTech
THE HOST
8 HOURS
5-9 HOURS
1975 – TM started with Albert Leffler, Peter Gadwaw, Gordon Gunn . . . . . .
Also Bill Gates and Paul Allen develop a BASIC program for the Altair 8800,
Microsoft becomes a registered trademark.
BetaMax is introduced.
Motorola obtains patent for mobile phone!
1976 – TM Starts . . . . IBM Launch the first laser printer.
1977 – ELO is first ticketed event at Johnson Gymnasium, University of New Mexico
1978 – Norway is the first international client with Oslo Nye Theatre
1981 – Ticketmaster UK Opens above the Albery Theatre
1983 – Ticketmaster UK starts to work with Arsenal
1988 – Ticketmaster Australia operations begin
THE THING YOU REALLY NEED TO KNOW
#WinOps
@connonm
@tmTech
I WANT THE
BEST SEAT(S)
ONSALE . . . 9AM . . .
#WinOps
@connonm
@tmTech
GROWING FAST
8-9 HOURS
5-9 HOURS
5-6 HOURS
10 HOURS
1-4 HOURS
5-9 HOURS
1991 – Ticket Tron in North America acquired
1996 – UEFA Euros with Ticketmaster United Kingdom
1996 – First Ticketmaster internet sale!
1997 – Archtics (Sports & Arts) integrated with and later ‘DSA’ acquired.
1997 – Synchro Systems (later Ticketmaster Sport)
1997 – Ticketmaster Ireland as a joint venture
1998 – Mexcio starts operations
1998 – ticketmaster.co.uk goes online, Windows NT 4
1999 – Joint venture in Australia
1999 - Rugby World Cup in Wales
2000 – TicketWeb, Vista and Admission Canada
2001 – First deal with Clear Channel – later to become Live Nation
2001 – Norway Billetservice acquired
2002 – Manchester Commonwealth Games
2002 – Netherlands & Denmark acquired
2004 – Athens Olympics
2004 – Sweden and Finland acquired
2005 – New Zealand established and Kartenhaus in Germany acquired
2006 – Acquires TickTackTickets in Spain and Biletix in Turkey
2007 – TM China joint venture
2008 – Beijing Olympics
2008 – TM acquires TicketsNow & GetMeIn,
2008 - Paperless tickets debut in North America
2008 – Paperless debuts in UK with 18k people attending at the O2 Arena
2008 – Ticketmaster Spun off from IAC
2008 - FrontLine Entertainment management acquired
2009 – TM UK breaks records with Michael Jackson, 50 dates at the O2.
2010 – LN and TM Merge to become Live Nation Entertainment
2010 - Ticnet in France acquired
2011 – Expansion in Spain with ServiCaixa
2012 – Ticketing for London2012, over 10.9 million tickets sold and processed.
Unprecedented at the time.
2013 - Launches in the middle east with UAE
2014 – Glasgow Commonwealth Games.
2014 - TM acquires Seatwave across Europe.
2014 - Middle east expands to Qatar
2014 – Relaunch DE, ES, acquires PL and Sherpa in Belgium.
2015 – Universe and Frontgate join the company
2015 – England Rugby World Cup
2016 – TM acquires Tickethour
2017 – TM acquires TicketPro in Czech Republic and launches Ticketmaster Israel
14 CORE
TICKETING
PRODUCTS
#WinOps
@connonm
@tmTech
THE INTERNET - 1998
WWW
NT4
SQL
THE ‘HOSTS’ - VAXEN 1999
8%?
20%
GULP
#WinOps
@connonm
@tmTech
REBUILD v1 – 2000
WWW
LINUX
ORACLE
THE ‘HOSTS’ - VAXEN
MySQL
#WinOps
@connonm
@tmTech
THEN THESE . . . . .
SOX -
2002
PCI v1 -
2004
#WinOps
@connonm
@tmTech
REMEMBER THESE?
#WinOps
@connonm
@tmTech
WE BUILT THIS . . .
DEV QA
CAP
STAGE
5-8 HOURS
PRODUCTION
. . . WE WERE PROUD OF IT!
LESS THAN . . .
OPTIMAL?
#WinOps
@connonm
@tmTech
Group A
WINDOW
OPS
Service
Delivery
WINDOWS
SYSTEMS
DBA TEAM
LINUX
OPS
LINUX
SYSTEMS
Group B Group C
PLATFORM
Product Support
CLOUD
SYSTEMS
TOOLING
NETWORKINGDATACENTRE
INTERNATIONAL TECHNOLOGY TEAMS IN 2016
#WinOps
@connonm
@tmTech
SECURITY
CORP IT
ARCHITECTURE
NA TEAMS
Product
Group 1
Product
Group 2
Product
Group 1
Product
Group 2
Product
Group 1
Product
Group 2
Product
Group 3
Product
Group 3
THE MORE YOU TIGHTEN YOUR GRIP,
THE MORE STAR SYSTEMS WILL SLIP
THROUGH YOUR FINGERS
#WinOps
@connonm
@tmTech
WHERE HAVE WE BEEN?
Engineering Ops
Syseng
Database
Engineering Ops
Syseng
Database
Engineering Ops
Syseng
Database
#WinOps
@connonm
@tmTech
WHERE HAVE WE BEEN?
#WinOps
@connonm
@tmTech
Engineering
Ops
Syseng Database
Platform
PUSH ME, PULL YOU
#WinOps
@connonm
@tmTech
PUSH ME, PULL YOU
#WinOps
@connonm
@tmTech
Ops & Reliablity
Service
Management
Ops & Reliablity Ops & Reliablity
PLATFORM
ENGINEERING
Product Support
Runtime
Services
& Data
Tooling
Infrastructure
(including DC)
Network
Engineering
Technical Service Delivery
SCALING THE SILOS, LOWERING THE WALL
PLATFORM
INFRASTRUCTURE
#WinOps
@connonm
@tmTech
Group A Group B Group C
Product
Group 1
Product
Group 2
Product
Group 1
Product
Group 2
Product
Group 1
Product
Group 2
Product
Group 3
Product
Group 3
Group B
Product
Group 1
Product
Group 2
Ops & Reliablity
HOW CAN WE EVOLVE?
#WinOps
@connonm
@tmTech
CAT = LOVE = PRODUCT!
#WinOps
@connonm
@tmTech
TOO LONG, EXPENSIVE
#WinOps
@connonm
@tmTech
DOES NOT FIT ANYMORE
#WinOps
@connonm
@tmTech
PLAY (RE)NICE? KILL -9
#WinOps
@connonm
@tmTech
UNACCEPTABLE!!!!!!
TICKETMASTER CLOUD
#WinOps
@connonm
@tmTech
SOMEBODY ELSES BOX
TRANSPARENT COSTS
#WinOps
@connonm
@tmTech
CONTAINERS – PART I
#WinOps
@connonm
@tmTech
CONTAINERS – PART II
#WinOps
@connonm
@tmTech
31
(MORE) MODERN TECH
CHANCE
TO CLEAN UP
WHAT ARE WE USING?
#WinOps
@connonm
@tmTech
TECH MATURITY
Open Source @
https://github.com/Ticketmaster/techmaturity
1. CONTAINERISE your application with Docker or AUTOMATE your Operating Systems with CHEF
2. IMPLEMENTATE BETTER INSTRUMENTATION - no SSH or RDP in production!
Log aggregation and rotation, metrics
3. Use SYNTHETIC MONITORING to understand the health of your product
Define SLAs, track & alert on availability and performance
4. AUTOMATE YOUR BUILD PIPELINE by reusing common tools
Terraform & GitLab CI are the products of choice
5. GET SECURE
Pass pen/static testing
No passwords/keys in configs
6. DESIGN A SHARED NOTHING ARCHITECTURE
No shared filesystems (e.g. NFS, CIFS)
7. BUILD FOR AVAILABLITY- NO SINGLE POINTS OF FAILURE
Use auto-scaling groups and without hard-coded IPs
OUR PRINCIPLES FOR MODERNISATION
#WinOps
@connonm
@tmTech
SUMMARY
#WinOps
@connonm
@tmTech
HISTORY IS USEFUL
CONTEXT IS EVERYTHING
DON’T BE AFRAID
Q & A
#WinOps
@connonm
@tmTech

Connon MacRae - Evolution of Ticketmaster's journey to DevOps

  • 1.
    EVOLUTION OF TICKETMASTER’S JOURNEY TODEVOPS WINOPS LDN SEPTEMBER 2017 #WinOps @connonm @tmTech
  • 2.
    WHO AM I& WHY AM I HERE? Connon MacRae VP, Technical Operations (International) #WinOps @connonm @tmTech HISTORY IS USEFUL CONTEXT IS EVERYTHING DON’T BE AFRAID
  • 3.
    MICROSOFT VS LINUX MICROSOFTLINUX NO MORE CYGWIN ON WINDOWS#WinOps @connonm @tmTech
  • 4.
  • 5.
    LIVE NATION ENTERTAINMENT– 42 MARKETS Live Nation & Ticketmaster Live Nation ONLY Ticketmaster ONLY #WinOps @connonm @tmTech
  • 6.
    #WinOps @connonm @tmTech THE HOST 8 HOURS 5-9HOURS 1975 – TM started with Albert Leffler, Peter Gadwaw, Gordon Gunn . . . . . . Also Bill Gates and Paul Allen develop a BASIC program for the Altair 8800, Microsoft becomes a registered trademark. BetaMax is introduced. Motorola obtains patent for mobile phone! 1976 – TM Starts . . . . IBM Launch the first laser printer. 1977 – ELO is first ticketed event at Johnson Gymnasium, University of New Mexico 1978 – Norway is the first international client with Oslo Nye Theatre 1981 – Ticketmaster UK Opens above the Albery Theatre 1983 – Ticketmaster UK starts to work with Arsenal 1988 – Ticketmaster Australia operations begin
  • 7.
    THE THING YOUREALLY NEED TO KNOW #WinOps @connonm @tmTech I WANT THE BEST SEAT(S) ONSALE . . . 9AM . . .
  • 8.
    #WinOps @connonm @tmTech GROWING FAST 8-9 HOURS 5-9HOURS 5-6 HOURS 10 HOURS 1-4 HOURS 5-9 HOURS 1991 – Ticket Tron in North America acquired 1996 – UEFA Euros with Ticketmaster United Kingdom 1996 – First Ticketmaster internet sale! 1997 – Archtics (Sports & Arts) integrated with and later ‘DSA’ acquired. 1997 – Synchro Systems (later Ticketmaster Sport) 1997 – Ticketmaster Ireland as a joint venture 1998 – Mexcio starts operations 1998 – ticketmaster.co.uk goes online, Windows NT 4 1999 – Joint venture in Australia 1999 - Rugby World Cup in Wales 2000 – TicketWeb, Vista and Admission Canada 2001 – First deal with Clear Channel – later to become Live Nation 2001 – Norway Billetservice acquired 2002 – Manchester Commonwealth Games 2002 – Netherlands & Denmark acquired 2004 – Athens Olympics 2004 – Sweden and Finland acquired 2005 – New Zealand established and Kartenhaus in Germany acquired 2006 – Acquires TickTackTickets in Spain and Biletix in Turkey 2007 – TM China joint venture 2008 – Beijing Olympics 2008 – TM acquires TicketsNow & GetMeIn, 2008 - Paperless tickets debut in North America 2008 – Paperless debuts in UK with 18k people attending at the O2 Arena 2008 – Ticketmaster Spun off from IAC 2008 - FrontLine Entertainment management acquired 2009 – TM UK breaks records with Michael Jackson, 50 dates at the O2. 2010 – LN and TM Merge to become Live Nation Entertainment 2010 - Ticnet in France acquired 2011 – Expansion in Spain with ServiCaixa 2012 – Ticketing for London2012, over 10.9 million tickets sold and processed. Unprecedented at the time. 2013 - Launches in the middle east with UAE 2014 – Glasgow Commonwealth Games. 2014 - TM acquires Seatwave across Europe. 2014 - Middle east expands to Qatar 2014 – Relaunch DE, ES, acquires PL and Sherpa in Belgium. 2015 – Universe and Frontgate join the company 2015 – England Rugby World Cup 2016 – TM acquires Tickethour 2017 – TM acquires TicketPro in Czech Republic and launches Ticketmaster Israel 14 CORE TICKETING PRODUCTS
  • 9.
    #WinOps @connonm @tmTech THE INTERNET -1998 WWW NT4 SQL THE ‘HOSTS’ - VAXEN 1999 8%? 20% GULP
  • 10.
    #WinOps @connonm @tmTech REBUILD v1 –2000 WWW LINUX ORACLE THE ‘HOSTS’ - VAXEN MySQL
  • 11.
    #WinOps @connonm @tmTech THEN THESE .. . . . SOX - 2002 PCI v1 - 2004
  • 12.
  • 13.
    #WinOps @connonm @tmTech WE BUILT THIS. . . DEV QA CAP STAGE 5-8 HOURS PRODUCTION . . . WE WERE PROUD OF IT!
  • 14.
    LESS THAN .. . OPTIMAL? #WinOps @connonm @tmTech
  • 15.
    Group A WINDOW OPS Service Delivery WINDOWS SYSTEMS DBA TEAM LINUX OPS LINUX SYSTEMS GroupB Group C PLATFORM Product Support CLOUD SYSTEMS TOOLING NETWORKINGDATACENTRE INTERNATIONAL TECHNOLOGY TEAMS IN 2016 #WinOps @connonm @tmTech SECURITY CORP IT ARCHITECTURE NA TEAMS Product Group 1 Product Group 2 Product Group 1 Product Group 2 Product Group 1 Product Group 2 Product Group 3 Product Group 3
  • 16.
    THE MORE YOUTIGHTEN YOUR GRIP, THE MORE STAR SYSTEMS WILL SLIP THROUGH YOUR FINGERS #WinOps @connonm @tmTech
  • 17.
    WHERE HAVE WEBEEN? Engineering Ops Syseng Database Engineering Ops Syseng Database Engineering Ops Syseng Database #WinOps @connonm @tmTech
  • 18.
    WHERE HAVE WEBEEN? #WinOps @connonm @tmTech Engineering Ops Syseng Database Platform
  • 19.
    PUSH ME, PULLYOU #WinOps @connonm @tmTech
  • 20.
    PUSH ME, PULLYOU #WinOps @connonm @tmTech
  • 21.
    Ops & Reliablity Service Management Ops& Reliablity Ops & Reliablity PLATFORM ENGINEERING Product Support Runtime Services & Data Tooling Infrastructure (including DC) Network Engineering Technical Service Delivery SCALING THE SILOS, LOWERING THE WALL PLATFORM INFRASTRUCTURE #WinOps @connonm @tmTech Group A Group B Group C Product Group 1 Product Group 2 Product Group 1 Product Group 2 Product Group 1 Product Group 2 Product Group 3 Product Group 3 Group B Product Group 1 Product Group 2 Ops & Reliablity
  • 22.
    HOW CAN WEEVOLVE? #WinOps @connonm @tmTech
  • 23.
    CAT = LOVE= PRODUCT! #WinOps @connonm @tmTech
  • 24.
  • 25.
    DOES NOT FITANYMORE #WinOps @connonm @tmTech
  • 26.
    PLAY (RE)NICE? KILL-9 #WinOps @connonm @tmTech UNACCEPTABLE!!!!!!
  • 27.
  • 28.
    SOMEBODY ELSES BOX TRANSPARENTCOSTS #WinOps @connonm @tmTech
  • 29.
    CONTAINERS – PARTI #WinOps @connonm @tmTech
  • 30.
    CONTAINERS – PARTII #WinOps @connonm @tmTech
  • 31.
    31 (MORE) MODERN TECH CHANCE TOCLEAN UP WHAT ARE WE USING? #WinOps @connonm @tmTech
  • 32.
    TECH MATURITY Open Source@ https://github.com/Ticketmaster/techmaturity
  • 33.
    1. CONTAINERISE yourapplication with Docker or AUTOMATE your Operating Systems with CHEF 2. IMPLEMENTATE BETTER INSTRUMENTATION - no SSH or RDP in production! Log aggregation and rotation, metrics 3. Use SYNTHETIC MONITORING to understand the health of your product Define SLAs, track & alert on availability and performance 4. AUTOMATE YOUR BUILD PIPELINE by reusing common tools Terraform & GitLab CI are the products of choice 5. GET SECURE Pass pen/static testing No passwords/keys in configs 6. DESIGN A SHARED NOTHING ARCHITECTURE No shared filesystems (e.g. NFS, CIFS) 7. BUILD FOR AVAILABLITY- NO SINGLE POINTS OF FAILURE Use auto-scaling groups and without hard-coded IPs OUR PRINCIPLES FOR MODERNISATION #WinOps @connonm @tmTech
  • 34.
  • 35.