Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mike Rowell - our journey to a next generation cloud


Published on

Challenge; solutions implemented; investigations into a scalable cloud solution; Apache CloudStack; Ansible, Terraform and Prometheus

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Mike Rowell - our journey to a next generation cloud

  1. 1. CloudStack Our journey to a next generation cloud Mike Rowell
  2. 2. Introduction 2 CLOUDSTACK – JOURNEY TO A NEXT GEN CLOUD ABOUT ME! Director Platform Infrastructure Remit: Infrastructure that runs majority of the Ticketmaster platform in International Ticketmaster was born in Arizona in 1976, now the world’s largest ticketing provider in over 30 countries
  3. 3. A Problem
  4. 4. PEAKY TRAFFIC 4 A PROBLEM We had: • Lots of traffic during peak times • Bare-metal capacity going under used at other times • Customer service issues with existing traffic
  5. 5. INVESTIGATION 5 A PROBLEM How do we get tens of thousands of customers an hour onto our platform whilst maintaining stability? OPTIONS:
  6. 6. An Implementation 6 A PROBLEM Tied into our internal authentication system and DNS Services Simple setup on the network; internal code so easy to update/upgrade Allows us to integrate Load Balancers into the platform as well at L2 Over a number of months a self service portal was created to allow internal users the ability to spin up instances on the fly Advantages Not a real cloud solution No API (would require extensive code updates) Reliant on single threaded code; often requires small updates to fix after patching Disadvantages
  7. 7. A Solution?
  8. 8. Roll Forward 10 Years….
  9. 9. PEAKY TRAFFIC 9 NOW We have: • Lots of traffic during peak times • No way to service it effectively with our virtualization platform; no API to build on demand or expand easily. • Some legacy code bases are not suited to going into EC2 yet
  10. 10. INVESTIGATION 10 NOW How do we get hundreds of thousands of customers an hour onto our platform whilst maintaining stability? OPTIONS:
  11. 11. Another Implementation We took some of the best of breed Open Source solutions and tested them out: 11 NOW Virtualisation platform not a cloud Lacking features but a very promising solution Massive system but hard implementation with very steep learning curve CloudStack Monolithic app but provides everything we need
  12. 12. Another set of issues 12 NOW LDAP Authentication issues L3 Network Implementation Zone setups in the beginning (VLAN vs VxLAN) The road to CloudStack has had issues: Storage issues (iSCSI vs NFS) Interesting KVM foibles Lack of HA Host support for iSCSI primary storage But we’ve stuck to our goal and have had our platform running for nearly a year now!
  13. 13. Our Solution We have two Production Clouds in two separated data-centers: 13 NOW Master / Master Distinctly separated – issues in one allows us to switch loads Java API Extensions To facilitate the migration from our legacy Xen “Cloud” Runtime JAR Talks to DNS, Inventory and legacy Xen Cloud Legacy VM Destroyed Once process is complete
  14. 14. Provisioning As with all Clouds automation is the key to management: 14 NOW Ansible Provisioning of hardware assets Automation Systems Python, Ansible, Terraform Network tenants Created a python system that pulls its configuration from a YAML config Ansible AWX To push out updates to Hypervisors, Controllers and some VMs
  15. 15. Monitoring 15 NOW Storage backend CloudStack exporter MySQL exporter During the CloudStack build out we also had to deprecate our legacy monitoring platform. New platform based on Prometheus; Multiple monitoring plugins and exporters: OVS exporter Lots of extra monitoring components that don’t come with the cloudstack Prometheus endpoint.
  16. 16. QUESTIONS
  17. 17. What features would we like to see next? •Serverless/Lambda style integration (OpenWhisk?) •Better Host HA Support •More pluggable infrastructure for adding LB/FW etc components •2FA Support •Hyper-V Support 17 QUESTIONS
  18. 18. Your Questions 18 QUESTIONS