PayPal has been using OpenStack to power its private cloud at scale since 2012. It now uses OpenStack across 100,000 cores, 10 petabytes of storage, and 10,000 physical servers hosting over 10,000 VMs. PayPal has migrated 100% of its web and mid-tier applications to run on OpenStack. It has built a proprietary PaaS on top of OpenStack to automate the application development lifecycle. Some challenges in operating OpenStack at this large scale include reliability issues, difficulty upgrading, and lack of maturity in some areas like security. PayPal's future plans include building a new PaaS using Mesos and Docker on OpenStack, achieving higher efficiency, and exploring hybrid cloud options.

1. PayPal’s Private Cloud @
Scale

2. Table of contents
©2015 PayPal Inc. Confidential and proprietary. 2
Introduction
OpenStack@PayPal
Our Journey
Lessons Learned
Future
Q&A

3. Introduction to PayPal
3

4. About PayPal
©2015 PayPal Inc. Confidential and proprietary. 4
PayPal is a leading technology platform company that enables digital and mobile payments on behalf
of consumers and merchants worldwide. We put our customers at the center of everything we do. We
strive to increase our relevance for consumers, merchants, friends and family to access and move
their money anywhere in the world, anytime, on any platform and through any device.

5. Architecture @ PayPal
©2015 PayPal Inc. Confidential and proprietary. 5
Robust Infrastructure, Reusable Platforms, Payments Operating System, Delightful Experiences
CustomerFacingInfrastructure
Analytics Platform
HadoopTeradata FPTIBI
Technology Platform
Framework Data Access Tools MediaMessagingSOA CrawlerCache
Infrastructure & Operations Layer
Power Data Center Hardware Network Database OperationsToolsCloudStorage Monitoring
Experience
Payments Operating System
LoginIdentity EmailCart CouponsPayment Ship SupportRisk InvoiceWallet POS Notify Merchant

6. OpenStack@PayPal
6

7. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary.
x 100K
Cores
x 10 PB
Storage
OpenStack Cloud
> 10K
Physical
Servers
x 10K
VMs
x 1000
engineers
x 10M
Lines
x 1000
Releases
/year
> 1000
Services
x 1000
Builds/d
ay
x 1000
Deploy/day
Key Statistics
Developer Statistics
9 Availability
Zones
3 Regions

8. Our OpenStack Journey
©2015 PayPal Inc. Confidential and proprietary. 8
2012 2013 2014 2015
• OpenStack POC
with 16 servers
• Built 2 AZs with
OpenStack Essex
release (96
hypervisors)
• 5 apps in
production
• Migrated from
Essex to Folsom
• Built 3 more Azs
• 350 apps in
production
• Support for dev
and QA
environments
• Upgrade to
Havana
• Migration of Nova
to Neutron
• 1000+ apps in
production
• 100% of web and
mid tier apps on
OpenStack
• Deployed our
largest AZ with
2500 nodes using
Nova cells
• Kilo upgrade in
progress

9. The Current Stack
©2015 PayPal Inc. Confidential and proprietary. 9
Datacenters
(Geographic regions, Availability Zones)
OpenStack Cloud
(Infrastructure as a Service)
Platform as a Service
Web/Mid Tier Applications
Common Infrastructure
Common Platforms and
Services
• 100% of web and mid tier
applications on OpenStack
• Proprietary PaaS on top of
OpenStack to automate
E2E application
development life cycle
• Support for polyglot
applications
• Test and production
environments available on-
demand

10. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary.
Steps to Production 25 Manual Steps Automated
Time to Market
QA Deployment
Build Time
Prod Deployment
Infrastructure
2 years back
Now
8 Weeks
5 Hours
2.5 Hours
4 Hours
1 Month
1 Day
30 Minutes
5 Minutes
< 30 Minutes
30 Minutes
Impact On Life Of A Developer

11. Challenges And Lessons Learned
11

12. Challenges And Lessons Learned
• Operating OpenStack at scale requires building an entire ecosystem of tools and back office functionality
like monitoring, configuration management, log aggregation, hardware remediation etc.
• We faced several control plane reliability issues as our deployment grew from several hundreds of
compute to several thousands specially around RabbitMQ and services interaction
• Scaling OpenStack to 10s of thousands of nodes is quite challenging specially around Neutron and
RabbitMQ.
• Security is evolving but not mature enough yet
• Upgrading to a newer version of OpenStack is not straightforward specially with internal patches and
takes months for larger deployments with multiple regions and availability zones
• Distributed nature of OpenStack results into challenges with debug-ability of system and identifying the
root cause
• OpenStack is a journey and commitment. It’s success depends on building a talented team of engineers
that takes pride in leveraging and contributing to OpenStack community©2015 PayPal Inc. Confidential and proprietary. 12

13. The Future
13

14. The Future Stack
©2015 PayPal Inc. Confidential and proprietary. 14
Datacenters
(Geographic regions, Availability Zones)
OpenStack Cloud
(Infrastructure as a Service)
Web and Mid Tier Applications
Common Infrastructure
Common Platforms and
Services
• New PaaS is being built
using Mesos and Docker
on top of OpenStack
• Goal is to achieve industry
leading datacenter
efficiency and utilization
• High application resiliency
against infrastructure
failures
• Hybrid cloud POC
• PaaS support for stateful
applications
Public
Clouds

15. Questions?
Jdesai at paypal dot com