Adopting containers at scale is fundamentally a cultural change. In late 2015, PayPal decided to migrate en masse to containers for applications built on many different frameworks over the last 15 years. It was a bold and strategic plan that included how to showcase value of containers to leadership, a phased execution strategy, building the right team to lead, and cultural transformation. Changing application code, deployment methods, and operational tools were at onset non-negotiable. This session will share how the plan was pitched and the learnings that unfolded as PayPal carefully changed everything - and nothing at the same time - to get to 150,000 containers running in production in 2 years.

1. PayPal’s 2 year
Journey to 150k
containers with
Docker
Meghdoot Bhattacharya
Cloud Engineering @ PayPal
October, 2017

2. PayPal By Numbers
$3.14B
Revenue
$106B
Total Payment
Volume (TPV)
Processed 210
million
Active Users
200
Global
Markets
Q2 2017
At-a-Glance

3. Planning
Container
Journey!
CaaS Serverles
s
Microservices
Containerize

4. Containerization
Orchestration/CaaS Platform
Microservices
Benefits of …

5. Journey Timeline
Centralized
team formed
Mid 2015
Late 2015
Engaged with
Docker Inc.
Integration of
docker in SDLC
starts
First container in
to QA and Prod
April 2016
Late 2016
Development on
CaaS starts
Developer
productivity tools
leveraging
containers
rolled out
Early-Mid 2017
Late 2017
First apps
launched on CaaS
(QA)
CaaS for
Production
2018

6. Cardinal Rules
1
Stay Invisible to
Developers at
Onset
• Largely black box
to developers. No
perceivable
change of
PDLC/SDLC.
• Developers does
not have time to
migrate
themselves.
• Unveil at right time
2
Target Existing
Applications First
• Need big mass to
showcase ROI in
enterprise
• Greenfield does
not move needle
3
Do Not Fragment
Environments
• Take it all the way
to production. Not
just for QA
• Drifts are very
costly in enterprise
4
Do Not Promise
Miracles at Start
• Not oversell.
Benefits will be less
at beginning
• Has to be phased
approach with
increasing benefits
at every phase
5
Core Dedicated
Team to Lead
• Disruptive
technology. Best
practices if not
followed can make
life worse
• Have to give time
for ecosystem
partners to catch
up

7. Phase I
(Containerizing Apps-As-Is)

8. Mission Statement
Seamlessly integrates docker in the SDLC as
universal packaging format with minimal
ecosystem disruption, aiming to
containerize 100% of production workload
and pave the path for future towards
leveraging full Containerization benefits.
Containerizing
Applications

9. Team Dynamics: Decides Outcome
Key
Traits
Dedicated
Migration
Engineer
Prep Time
• Rockstar old timers who are well aware of all
gory details of building and shipping apps.
• Respected and Trusted to make changes in
other team’s codebase.
• Startup by itself with full stack skills: OS and
system experts, framework and app specialists,
deploy and release engineers etc
• Docker Experts.
• Team spent sufficient time
often on sidelines to
understand nuances of docker.
• Get to a confidence level that
team can handle any
challenge.
• Build relationships with docker
Inc. team before game time.• Focused 100% on migration
execution.
• Excellent communication skills
with loads of patience.

10. Mindset @ Phase 1
Recognize You May
Have One Shot
• Dealing with Legacy is tabooed.
You have to prove it wrong.
Spread The Knowledge
• Program has to scale and sustain with
time.
• Dependent teams, key partners need to
independently innovate at some point.
Work on Full Stack (kernel/OS
to SDLC changes)
• Dependent teams may not have any bandwidth to
introduce this in environment.
• They are also NOT docker experts at beginning.
• Be prepared to make changes at every level of
stack.
Guilty Until Proven Innocent
• Issues will arise during migration. Debug on
behalf of developers.
• Developers have trusted you with this black
box change. Do not lose it.
• They are not Experts. You Are!
Expect Push Back
• Certain teams will be too busy to
even support seamless migration.
• Be persistent, prove it to them by
learning their app. Convince them.
Minimal Ecosystem Changes
• Perceived Benefits less compared to major changes in monitoring,
logging and downstream tools. Instead go for least touch points.
• Deploy and Operational topology left almost intact. Just app
package and runtime changed.
100% laser focused on As-Is-Migration
• Not get tempted to couple program with microservices, app
refactoring, CaaS, BM vs VM etc. Separation of concerns.
• Fast output essential. Project can be killed if slow pace. Show
confidence in your architecture, strategy and execution plans.
Do Not Hide The Migration
• Engage with customers/developers so that
they are aware of the migration.
• Black box does not imply no communication.
In fact, over communicate.

11. Rollout Strategy
Train Key Folks
• Not black box to production operators and L1 support teams.
• They need SOP, basic triaging skills.
Communicate.
Communicate.
Communicate.
• Opening emails to dev teams discussed migration details,
benefits.
• Migration activity had channels open to both developers and
operators simultaneously.
• Make it clear, its not an optional migration!
Find Champions • Critical for word of mouth and patient on rough edges.

12. Execution Strategy & Benefits
1
Introduce Container
Format into PDLC
Largely black box to
developers.
Updated existing tool
chains in Infrastructure.
Base Docker
Infrastructure and
supporting ecosystem
put in place
2 Decoupling Deploy
Platform from
Framework Stacks
Standardize deploy
platform against
container primitives
rather than per stack
logic.
3 Modernize OS and
Kernel from Decade-
old Versions
Move to modern OS in
hosts running modern
kernel. Standardize host
configs. Significantly
smaller Container VM
image.
Some Docker applications
got 10 -20% boosts in
performance for free.
4
Paves the Way for
the Next Chapter
Foundation for future
developer-centric
capabilities and unmask
Docker from
developers.
Foundation for future
full blown Container
Ecosystem to harness
full power of containers.

13. Progress
Production Containerization Progress To-Date
# of
Applications
dockerized
Time since
Phase 1 Launch
# of Container
VM hosts
# of
Containers
Critical
Flows
~150k~45k700+18 mos
Home page
Login
Payment
Checkout
& more

14. Phase 2
(Developer Empowerment)

15. Mission Statement
Piggybacking on Phase 1 that laid down docker
foundations and containerized hundreds of
applications, Phase 2 brings best of breed
developer productivity tooling, opens up
DevOps to entirely new areas and gives
flexibility to community to start refactoring
and upgrading apps.
Developer
Empowerment

16. Execution Strategy and Benefits
1
Expose docker
to developers.
No longer a black
box.
Unlock this
technology to
developer
community. Let
them innovate.
2 Enable local
laptop
workflows
Build, deploy and
debug locally with
significant time
savings.
Cloud Cost
Savings on mass
adoption.
3 Extend devOps
Platform to
Greenfield
Apps
Provide to non
standard (custom)
stacks platform
automation.
Remove manual
ticketing for these
stacks.
4 Enables new
ecosystem apps
and tooling
Developers and
Frameworks
introduce apps
and tooling that is
powered by
modern libraries.
Avoid one off
setups and
handled through
same platform.
5 Empower key
teams to
leverage docker
Make other teams
independently
successful in their
own use case.
Key Orgs building
own tooling.

17. Progress
Platform Stack
Supporting Non-
Standard
Offerings
Run QA
Environment
Locally
Modernization
of OS Base
Image
Healthy Growth
of independent
registry Orgs
outside of PDLC
Independent
Use Cases
~25
NodeJS base
image upgraded in
one shot for all
apps
Apps in GO,
custom java etc
leveraging full
SDLC automation
50% or more
boost in build,
deploy and
test/debug cycles
with reduced
tooling touch
points for users.
Data Platform, Risk
Platform, Core App
Framework teams
building rich tooling
leveraging
containers

18. Phase 3
(Migration to CaaS Platform)

19. Mission Statement
Containerized applications are migrated
into CaaS platform across multi-
clouds providing significant benefits to
developers and operators in areas of
Availability, Security, Efficiency,
Enablement and Modernization.
CaaS
Migration

20. Operations ROI
1 Efficiency (cores
consumption)
25% less vCpu in prod and 50%
less vCpu in QA.
2 Security
Revoke access to productions hosts in
lieu of secure container access.
Automated patching of hosts without
human intervention or pool depletion
risks
3 Elimination of VM
Provisioning
Rapid flexups of application
instances under traffic surge
without VM creation overhead.
4 Major Availability
Improvements
Resiliency against faults like
device/racks/partition/partial power
failures.
Departure from static app pool
partitioning to dynamic placement.
5 Faster New AZ
Onboarding
Application deployments of an
entire AZ in few hrs.
6 One Consistent Platform
Across All Environments
Consistent tooling (monitoring,
alerting etc) and SOP
(standardized playbook across
app stacks)

21. Developers do not face
VM set up cost. Just spin
the container.
Cluster adds capacity
seamlessly..
Faster deployments
through a modern
scheduler.
Predictable auto rollback
in minutes across the
site for critical pools.
Deployment time across
multi AZ across geo
regions < 1 hr
Containers migrated in
real-time to good hosts if
hosts enter a bad state
or hosts going down
Declarative deployment
systems offer better
reliability.
Stop exposing to
developers multiple
tools/platforms for same
functions in different
environment across
stacks and across multi
clouds
(openstack/aws/gcp/azu
re)
Make different
experiences on top of
consistent platform. Do
not build infrastructure
every time for new
experience
Top Developer Benefits Pitched
One Consistent
Platform Reliability &
Resiliency
Deployment
Performance Elimination of
VM Provisioning

22. Summary

23. Multi Phase Timeline
CaaS
Migration
Developer
Empowerment
Containerizin
g Applications
Converged Container
Platform
New
Application/Migrated
Application
Deployed in
containers
Access to leading
developer tools
Realizing CaaS
benefits
End State
Phase 1 Phase 2 Phase 3