1) PayPal operates a large OpenStack cloud platform with over 10,000 physical servers hosting 100,000 VMs to run over 1000 services for their business. 2) They wanted to move stateful applications like messaging, streaming, caching and databases to the cloud but faced challenges with agility, efficiency, elasticity and onboarding while preserving stateful data. 3) After evaluating options like network block storage, ephemeral disks, and hyperconverged storage, they chose to use VMs with attached local disks which does not lose data when VMs are lost and has lower network bandwidth needs and costs, though storage is lost if the host fails.

1. Stateful Applications On The
Cloud – A PayPal Journey
Anant Kumar and Jasdeep Singh
PayPal Inc.
August 23, 2016

2. About Paypal
2

3. 3
x 500K
Cores
x 10 PB
Storage
OpenStack Cloud
> 10K
Physical
Servers
x
100K
VMs
x 1000
engineers
x 10M
Lines
x 1000
Releases
/year
> 1000
Services
x 2000
Builds/day
x 1000
Deploy/day
• Developer
Statistics
9 Availability
Zones
3 Regions
Key Statistics

4. 4
4
2012-2013
2014-
2015
2016-
2017+
• OpenStack POC with 16
servers
• Built 2 AZs with
OpenStack Essex
release
• 5 apps in production
• 40% of holiday traffic for
Paypal Web and APIs
• PaaS on Cloud with full
PDLC support (Node and
Java applications)
• 100% of Web, API & Mid-
tier workloads
• All Stateless
Applications in Cloud
• Built largest OpenStack
based AZ (for Dev & QA)
‒ Next frontiers in
Developer Productivity
‒ Infrastructure Cost &
Efficiency
‒ Stateful
Applications in
Cloud
PayPal Cloud Platform Journey

5. Problem Statement
• Applications Still On Bare Metal
• Messaging
• Streaming
• Caching
• Search
• Databases
• Challenges With:
• Agility
• Efficiency
• Elasticity
• Onboarding
5

6. Guiding Principles
• Hybrid Cloud Ready
• Cost Efficiency
• Container Compatible
• No Loss Of Efficiency And Performance
6

7. Constraints
• Preserve State (Petabytes Of Data)
• Storage, Storage, Storage
• Recovery from VM failure
• Network Bandwidth
• Noisy neighbor
7

8. 8
Two Columns
• Network Block Storage • VMs With Ephemeral Disk
VM 1
Hypervisor
VM 2 VM n
Disk 1 Disk 2 Disk n
Host
VM 1
Hypervisor
VM 2 VM n
Network Block Storage
Host
iSCSI

9. 9
Options
• VMs With Attached Local Disk • Hyper Converged Storage
VM 1
Hypervisor
VM 2 VM n
Disk 1 Disk 2 Disk n
Host
VM 1
Hypervisor
VM 2 VM n
Hyperconverged Storage Manager
NAS Disk 1 Disk n
Host

10. What We Chose
• Winner (For Now) : VMs With Attached Local Disks
• Advantages
• Do Not Lose Data When VM Is Lost
• Network Bandwidth Not A Concern
• Cheaper (Always Helps !!!)
• Disadvantages
• Storage Is Lost If The Host Is Lost
10

11. Conclusion
• Moving Ahead Full Steam - “VMs With Attached Local Disk”
• Keeping A Close Eye On The “Hyper-converged Storage”
Space.
• Will Likely Support Multiple Options In The Future
• Keep The Hybrid Cloud In Mind
11