This document outlines PayPal's OpenCloud platform built using OpenStack. The platform aims to provide agility, availability and innovation through a unified PaaS and IaaS stack. It uses OpenStack for compute, storage and networking with additional services for load balancing, DNS management and monitoring. The current deployment includes one OpenStack installation per data center, supporting 1300 VMs across 96 compute nodes. Lessons learned so far include fitting OpenStack into existing infrastructure and customizing availability zones. Future plans include improving networking, bare metal provisioning, and extending the platform to development, QA and other environments.

1. OpenCloud At PayPal
Anand Palanisamy
15 Dec 2012

2. VISION
Provide a platform that enables agility, availability and innovation.
Risk
Payments
Wallet
Mobile
Ads
Unified
PAAS
IAAS
Infrastructure
Services
(Compute,
Storage,
Switches,
FWs,
LBs,
DNS
)
Product
Pre-­‐
DEV
QA
PROD
PROD
M&A’s
Mgmt
Centralized
Monitoring
2 Confidential and Proprietary

3. GUIDING PRINCIPLES
Technology
! Adopt Open Source Solutions where ever possible
! No Vendor Lock-in
! Industry Best Practices
! Leverage Industry Investments
Functionality
! Self-Service
! Developers First
Methodology
! Run Operations as an Engineering Organization
! Agile Development Model and Small 3 to 4 team members
! Delivery Every 2 weeks

4. VISION
Provide a platform that enables agility, availability and innovation.
Risk
Payments
Wallet
Mobile
Ads
Unified
PAAS
IAAS
Infrastructure
Services
(Compute,
Storage,
Switches,
FWs,
LBs,
DNS
)
Product
Pre-­‐
DEV
QA
PROD
PROD
M&A’s
Mgmt
Centralized
Monitoring
4 Confidential and Proprietary

5. WHY OPENSTACK?
• Closely known Success Stories
• Lots of customization capability
• Wider support by all major vendors
• Very Active Open Source Community
• We are using Linux, Java, Hadoop
• Opportunity to hire top talents
• Foundation

6. TECHNOLOGY STACK
Operations Portal PD Deployment Portal
User Horizon, Ceilometer
Interface Traffic Mgmt Monitoring Metering Stages Workflow Monitoring
Orchestration Engine
Orchestration
Cloud Formation (Heat)
Foundational Compute LB Management DNS Management
Services Nova, Swift, Glance, Key Stone LBaaS DNSaaS
Software Cobbler Nagios
Infrastructure
Salt Bind RHEL 6.3 KVM
ISC DHCP
Hardware Compute Local Storage Network Load Balancer
Infrastructure HP SL230. Gen 8 HP 4X600 GB(Mirror) Cisco 4948 & Arista 7050 F5 10.2.2
OpEx
6 Confidential and Proprietary

7. CURRENT DEPLOYMENT
• 1 OpenStack deployment per Data Center
• 96 Compute Nodes in a Rack
• 4 Top of Rack Switches (2 PROD, 2 Management)
• /23 subnet X 2 (PROD), /23 subnet X 2 (Management)
• Supporting 2 Flavors (12 to 16 VMs per host) = ~ 1300 VMs
• Utilized existing physical L3, Firewall
• Nova, Glance, Swift, Keystone, Dashboard (Essex Stable 3)
• Integrated with LBs, DNS, Change Management, naggios monitoring

8. LESSONS LEARNED
• How to fit into existing infrastructure?
• Availability Zone Customization
• Plug-in Options for DNS and LB
• Lab Infrastructure
• OpenStack Dev. Life Cycle

9. WHAT’S NEXT
• Sophisticated networking
• Bare Metal Provisioning
• Compute Cells
• Open Hardware Spec
• Extend to Dev., QA, Staging, L&P
• Self-Service, Security
• Migration to Folsom and beyond
• Design, Develop, Fix and Contribute to community
• Inter-Operability Lab and ebay inc distribution

10. COMMUNITY TARGET
CLOUD

11. YES. WE ARE HIRING
apalanisamy@paypal.com
http://ebaycareers.com