This document summarizes a presentation about how Skytap, a cloud hosting provider, transitioned their infrastructure and services to Kubernetes over time. Some key points: - In 2014, Skytap's infrastructure was complex and difficult for developers to work with. Releases were monthly and patches were scary. - Since 2016, all new services run on Kubernetes and high-churn proprietary services were migrated. This integrated Kubernetes into their CI/CD pipeline. - Skytap runs 11 Kubernetes production clusters with 70 nodes and 185 namespaces, handling around 1,000 pods. - The transition followed guiding principles of changing as little as possible and using new tools to harmonize with existing systems. Services were categorized and moved

1. How do you eat a whale?
One byte at a time!
O’Reilly Velocity Conference 2017
Oct 3, New York, NY
Kelly Looney,
Director of DevOps Consulting
Skytap Inc
* No whales were harmed in the making of this presentation. Skytap does not promote the eating or harming of whales.

2. Topics
o Where we started
o Where we’re now
o How we got here
o Organization
o Education
o Technology
o Parting wisdom

3. Skytap: Key Stats
o Regions
o 7 Multi-tenant (3 US, TOR, EMEA,
AUS, APAC)
o 3 Single-tenant (US)
o 18,057,400 VMs deployed
o Up to 44,500 / day
o 10,356,700 virtual L2 networks deployed
o Up to 19,600 per day
o 604 petabytes of allocated virtual storage

4. Starting Situation (circa 2014)
o Complex distributed system deployed across several regions
o The service was (mostly) reliable and scalable
o Deployments once a month; patched as needed - but are scary
o Heavy involvement from operations
o Difficult for devs to develop, test, and deploy

5. Starting Point

6. Current Situation
o All new services since 1/2016 run in K8S
o All proprietary high churn
services run in K8S
o Integrated CI/CD pipeline
o Ops focused on high value projects
o Release as needed – with confidence!

7. Current Situation

8. SOME SORT OF K8S Picture
K8s clusters in Skytap
o Production
o 11 clusters
o 70 nodes
o 185 namespaces
o ~1K pods at any given time
o Staging & Preprod
o 9 clusters
o 34 nodes
o 400 pods at any given time

9. What We Were Aiming For
o Reduce the unit of deployment
o Micro-services
o Complexity will only increase
o Comprehensive monitoring,
service discovery, and orchestration
o Easy stuff first
o Stateless and immutable services

10. First Steps…
Guiding Principles
o Change as little as possible
o New tools harmonize with
existing tools
o New stuff in the new framework
Actions
o Get key players on board
o Inventory and categorize services
o Determine how to concurrently
run old/new

11. Organization
o Recruit a dedicated tools team
o Not a part-time job
o Ideally members have
o Deep technical ability
o Architectural knowledge of
o Major system components
o CI/CD Tools
o RM & Deployment Practices
o Ability to teach

12. The SRE Role
• An alternative top of the tech ladder
• Start reactive with goal of being mostly
proactive
• Fire Chief to SRE story
• Let this be your primary means of
improvement
• Make the system easy to change first
• Goal to be unafraid to replace or re-
implement when needed
• Be an educator and mentor

13. Education
o Buddy system
o Documentation
o Support channels
o Reusing existing tools

14. Our World
o Devs own image generation &
deployment config
o Prebaked templates and custom
builds
o Educational Areas
o Dependency management
o Dockerfile authoring and image
caching
o Implementing K8S health checks
o Estimating resource usage

15. Technology
o Developers are human
o Release management process
o Kube-native vs. traditional CI/CD
o Which services to move to
Kubernetes first?

16. Service Categorization
o Application Tier
• Services (Web)
• Platform
• Infra
o Communication model
• Socket based
• Message passing (MQ)
o Application type:
• Stateful
• Stateless

17. Our CI/CD & Kube
o Deploy ~150 3rd party and
proprietary services to ~1,000
machines in 10+ regions
o Custom CI/CD tools on
Capistrano and Jenkins
o Kube integrated with existing
CI/CD framework

18. Testability is a P-Zero
o Deployment tools are hard to test
o Failed deployments == Dirty test
environments
o Automated multi-fidelity
environment builds

19. Fatal Mistakes to Avoid
o Underestimating what you have
o Not considering code, state, & data
o Transient technology choices
o Trying to deliver too much at once

20. Parting Wisdom
o A customer first attitude will drive
adoption
o Start with compute
o Pick up networking & storage later
o Consider your existing toolchain
o Ability to reset environments will keep
you moving fast
o Much easier for container services to talk
to legacy than vice versa

21. Thank You!

22. Contain Yourself:
Incremental Adoption
for Modernization
CoreOS Feast, 2017
San Francisco, CA
Petr Novodvorskiy, Development Lead
Dan Jones, Director of Product Management