Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart
•Download as PPTX, PDF•
4 likes•20,905 views
Riot builds a lot of software. At the start of 2015 we were looking at 3000 build jobs over a hundred different applications and dozens of teams. We were handling nearly 750 jobs per hour and our build infrastructure needed to grow rapidly to meet demand. We needed to give teams total control of the “stack” used to build their applications and we needed a solution that enabled agile delivery to our players. On top of that, we needed a scalable system that would allow a team of four engineers to support over 250. After as few explorations, we built an integrated Docker solution using Jenkins that accepts docker images submitted as build environments by engineers around the company . Our “containerized” farm now creates over 10,000 containers a week and handles nearly 1000 jobs at a rate of about 100 jobs an hour. In this occasionally technical talk, we’ll explore the decisions that led Riot to consider Docker, the evolutionary stages of our build infrastructure, and how the open source and in-house software we combined to achieve our goals at scale. You’ll come away with some best practices, plenty of lessons learned, and insight into some of the more unique aspects of our system (like automated testing of submitted build environments, or testing node.js apps in containers with Chromium and xvfb).
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart
Similar to Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart (20)
More from Docker, Inc.
More from Docker, Inc. (20)
Recently uploaded
Recently uploaded (20)
Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart
- 1. Thinking Inside the Container: A Continuous Delivery Story Maxfield Stewart Engineering Manager: Riot Games
- 3. Thinking Inside the Container
- 5. A Quick Note on Riot Games
- 6. The Scale of League MORE THAN 67 MILLION MONTHLY ACTIVE PLAYERS MORE THAN 27 MILLION DAILY ACTIVE PLAYERS 7.5 MILLION PEAK CONCURREN T PLAYERS
- 7. 1.25 Million Builds a Year
- 8. 10,000 - 14,000 Containers A Week
- 9. 120 Jobs An Hour
- 10. A Containerized Build Farm JENKINS SWARM DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker 1.10.3 4 Core/32GB RAM/120GB LVS DRYDOCK BUILD HOSTS SSH Docker API Docker API Docker API
- 11. STORY TIME! (aka. Presenting the Problem)
- 12. 3500+ Unique Build Jobs
- 13. 650+ Builds an Hour
- 14. 90+ Build Slaves
- 17. Story Time.... Engineers Build Team
- 18. What Did We Want? TEAMS NEEDED TO MOVE FAST PRODUCTS HAD TO OWN THEIR STACK CONFIGURA TION AS CODE
- 19. Maybe We Want...
- 20. “Along Came a Whale” (More Story Time!)
- 22. Oh Look! Another Way to Deploy!
- 23. Docker For Newbz
- 25. Jenkins Primer
- 26. Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” Master Slaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB”
- 27. Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” Master Slaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA”
- 28. Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” Master Slaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA” Build Job Build Job Build Job Build Job Queue
- 29. Jenkins Primer JENKINS “Win32” + “Java” + ”TeamA” Master Slaves “Cent7” + “Java” + ”TeamB” “Cent7” + “GoLang” + ”TeamB” Build Job “Win32” + “Java” + ”TeamA” Build Job Build Job Build Job Build Job Queue
- 30. What Did We Need? CONTAINER RUNNING AS A BUILD SLAVE HOW TO SCALE IT WAY TO HAVE ENGINEERS PROVIDE DOCKERFILES PROFIT?
- 31. A Build Slave Container
- 32. Add a Bit of Secret Sauce... Base Slave Image Universal Riot Tools Product/App Specific Build Stuff
- 33. A Real Example
- 34. Provisioning and Plugins + = ?
- 35. Of Whales and Plugins MESOS PLUGIN KUBERNETES PLUGIN DOCKER PLUGIN
- 36. Of Whales and Plugins MESOS PLUGIN KUBERNETES PLUGIN DOCKER PLUGIN
- 37. Quick Look
- 38. Provisioning and Plugins Image Name Label
- 39. Groovy To the Rescuettps://github.com/maxfields2000/dockerjenkins_tu orial/tree/master/jenkinsscripts
- 40. We Created A Monster
- 41. We Need to Inspect Our Containers
- 42. We Need to Inspect Our Containers ./harbormaster --registry_user "$REGISTRY_USER" --registry_token "$REGISTRY_TOKEN" --jenkins_user $USER -- jenkins_pass $PASSWORD --jenkins --dockerhost $TEST_DOCKER_HOST "$IMAGENAME" "$CLOUDNAME" "$LABELNAME"
- 43. Are We Done?
- 44. But It Needs to Scale!
- 45. If Only...
- 46. If Only...
- 47. But It Needs to Scale! JENKINS SWARM DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker 1.10.3 4 Core/32GB RAM/120GB LVS BUILD HOSTS SSH Docker API Docker API
- 48. Putting It All Together Engineer REGISTRY Image
- 49. Putting It All Together Engineer REGISTRY HARBOR MASTER Image Name + Jenkins Label
- 50. Putting It All Together Engineer REGISTRY HARBOR MASTER Runs Tests
- 51. Putting It All Together Engineer REGISTRY HARBOR MASTER SWARM DOCKERHOST DOCKERHOST DOCKERHOST Image Pulled to Swarm
- 52. Putting It All Together Engineer REGISTRY HARBOR MASTER SWARM DOCKERHOST DOCKERHOST DOCKERHOST Groovy API Called Jenkins
- 53. Build Job Quick Look
- 54. Dockerception
- 55. Where to Build Containers? JENKINS SWARM DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST DOCKERHOST cAdvisor Docker-GC Container Metrics CentOS 7.2/Docker 1.10.3 4 Core/32GB RAM/120GB LVS DRYDOCK BUILD HOSTS SSH Docker API Docker API Docker API
- 56. Mai Tai’s On the Beach Build Engineer
- 57. How Do You Actually Build It?
- 58. How Do You Actually Build It?
- 60. You Make This Sound Simple.
- 61. Lesson 1 - Docker Isn’t “Simple” DOCKERFILES REQUIRE DECENT SYSTEMS ADMIN KNOWLEDGE DOCKER IMAGES THAT BUILD DOCKER IMAGES IS DOCKERCEPTION DOCKER “VOODOO AND BLACK MAGIC” QUESTIONS
- 62. Lesson 2 - Containers != VM’s CANNOT MOUNT REMOTE FILE SYSTEMS HAVE RULES FOR DOCKERFILE VS RUN TIME EPHEMERAL CONTAINERS DON’T MAINTAIN STATE
- 63. Lesson 3 - Garbage Collection DOCKER BUILD DOCKER PULL DOCKER RUN VOLUME
- 64. Lesson 4 - Maintenance/Failure PULL HOSTS ON/OFFLINE UPDATE ALL IMAGES ROLLING RESTARTS
- 65. Lesson 5 - How Will You Upgrade?
- 66. Lesson 7 - Credentials & Security WHAT SHOULD YOU PUT IN YOUR BASE IMAGE? PASSWORDS IN YOUR SOURCE CODE? SSH KEYS IN YOUR REGISTRY? PASSING EVERYTHING IN AS A ENVIRONMENT VARIABLE?
- 68. Over 1200 New Build Jobs Created
- 69. 30% of All Environments Are Containers
- 71. We No Longer Have “Just” A Build Team
- 72. QUESTIONS?