How do you eat a whale?
One byte at a time!
Cloud Computing Expo 2017
June 8th, New York, NY
Kelly Looney,
Director of DevOps Consulting
Skytap Inc
* No whales were harmed in the making of this presentation. Skytap does not promote the eating or harming of whales.

Topics
o Where we started
o Where we’re now
o How we got here
o Organization
o Education
o Technology
o Parting wisdom

Skytap: Key Stats
o Regions
o 7 Multi-tenant (3 US, TOR, EMEA,
AUS, APAC)
o 3 Single-tenant (US)
o 18,057,400 VMs deployed
o Up to 44,500 / day
o 10,356,700 virtual L2 networks deployed
o Up to 19,600 per day
o 604 petabytes of allocated virtual storage

Starting Situation (circa 2014)
o Complex distributed system deployed across several regions
o The service was (mostly) reliable and scalable
o Deployments once a month; patched as needed - but are scary
o Heavy involvement from operations
o Difficult for devs to develop, test, and deploy

Starting Point

Current Situation
o All new services since 1/2016 run in K8S
o All proprietary high churn
services run in K8S
o Integrated CI/CD pipeline
o Ops focused on high value projects
o Release as needed – with confidence!

Current Situation

SOME SORT OF K8S Picture
K8s clusters in Skytap
o Production
o 11 clusters
o 70 nodes
o 185 namespaces
o ~1K pods at any given time
o Staging & Preprod
o 9 clusters
o 34 nodes
o 400 pods at any given time

What We Were Aiming For
o Reduce the unit of deployment
o Micro-services
o Complexity will only increase
o Comprehensive monitoring,
service discovery, and orchestration
o Easy stuff first
o Stateless and immutable services

First Steps…
Guiding Principles
o Change as little as possible
o New tools harmonize with
existing tools
o New stuff in the new framework
Actions
o Get key players on board
o Inventory and categorize services
o Determine how to concurrently
run old/new

Organization
o Recruit a dedicated tools team
o Not a part-time job
o Ideally members have
o Deep technical ability
o Architectural knowledge of
o Major system components
o CI/CD Tools
o RM & Deployment Practices
o Ability to teach

The SRE Role
• An alternative top of the tech ladder
• Start reactive with goal of being mostly
proactive
• Fire Chief to SRE story
• Let this be your primary means of
improvement
• Make the system easy to change first
• Goal to be unafraid to replace or re-
implement when needed
• Be an educator and mentor

Education
o Buddy system
o Documentation
o Support channels
o Reusing existing tools

Our World
o Devs own image generation &
deployment config
o Prebaked templates and custom
builds
o Educational Areas
o Dependency management
o Dockerfile authoring and image
caching
o Implementing K8S health checks
o Estimating resource usage

Technology
o Developers are human
o Release management process
o Kube-native vs. traditional CI/CD
o Which services to move to
Kubernetes first?

Service Categorization
o Application Tier
• Services (Web)
• Platform
• Infra
o Communication model
• Socket based
• Message passing (MQ)
o Application type:
• Stateful
• Stateless

Our CI/CD & Kube
o Deploy ~150 3rd party and
proprietary services to ~1,000
machines in 10+ regions
o Custom CI/CD tools on
Capistrano and Jenkins
o Kube integrated with existing
CI/CD framework

Testability is a P-Zero
o Deployment tools are hard to test
o Failed deployments == Dirty test
environments
o Automated multi-fidelity
environment builds

Fatal Mistakes to Avoid
o Underestimating what you have
o Not considering code, state, & data
o Transient technology choices
o Trying to deliver too much at once

Parting Wisdom
o A customer first attitude will drive
adoption
o Start with compute
o Pick up networking & storage later
o Consider your existing toolchain
o Ability to reset environments will keep
you moving fast
o Much easier for container services to talk
to legacy than vice versa

Thank You!

Contain Yourself:
Incremental Adoption
for Modernization
CoreOS Feast, 2017
San Francisco, CA
Petr Novodvorskiy, Development Lead
Dan Jones, Director of Product Management