VSVC4994 - Marriage of ESX and OpenStack at PayPal PayPal is quickly moving forward to utilize open source and open standards based technologies in the build-out of our private cloud. With our internal release of OpenStack software based on 'Grizzly' we have integrated ESX 5 support and now can deploy workloads against ESX as well as against KVM.

1. VSVC4994
Marriage of ESX and OpenStack at PayPal
Scott Carlson, PayPal
#VSVC4994

2. THE MARRIAGE OF ESX AND
OPENSTACK AT PAYPAL
MULTI-VENDOR AGILITY
VMWORLD 2013

3. ABOUT PAYPAL
PayPal offers flexible and innovative payment solutions for consumers
and merchants of all sizes.
• 132,000,000 Users.
• $300,000 Payments processed by PayPal each minute.
• 193 markets / 25 currencies.
• PayPal is the World’s Most Widely Used Digital Wallet.
3

4. WHY WE VIRTUALIZED ON ESX
• 80% of the PayPal front-end is virtualized on VSphere 5.0u1
• Primary Criteria
− Stability, performance, industry expertise, availability of experts
• Standardized on VCE VBLOCK© for initial implementation
• Fully consumable API
• Load-test harness well understood in industry (specInt & vMark)
− Predictable scaling pattern for horizontally scaled workloads
4

5. CLOUD
5

6. PAYPAL INTERNAL CLOUD
2012/2013 Shift toward an internal cloud model
• Shift from Enterprise design model to cloud-based design
• Elastically scale and self-heal infrastructure to accommodate
unpredictable usage patterns of customers and internet commerce
• Separate rapidly iterating customer experiences from core
services
• reduce overall cost per transaction within the environment
6

7. CLOUD IS THE GREAT ENABLER
ENABLE THE DEVELOPER
ENABLE THE BUSINESS
One-Click
Developer
Self Service
Global
Compute &
Data
Fulfillment
Payment Delivery
SelfOrganizing &
Optimizing
Infrastructure
System Intelligence
Driven Operation
Code
7
Deploy
Enjoy

8. PAYPAL CLOUD PLATFORM –
GUIDING PRINCIPLES
• Technology
− Adopt Open Source Solutions where ever possible
− No Vendor Lock-in
− Industry Best Practices
− Leverage Industry/ebay Inc Investments
• Functionality
− Self-Service tool for application life cycle management.
− Robust Automation & Orchestration
− Seamless On-Demand Capacity Fulfillment
8

9. OPENSTACK
PayPal deploying Openstack in order to help transform our global
infrastructure into an agile and open cloud platform.
Agility - time to market for customer facing services
Agility - speed to service developer requests for VM resources
Agility – utilize the engineering culture of PayPal to
deliver specialized cloud services where needed
9

10. INCREDIBLE INDUSTRY SUPPORT
10

11. TECHNOLOGY STACK
User
Interface
Operations Portal
DEVS Deployment Portal
Horizon, Ceilometer
Traffic Mgmt
Monitoring
Metering
Stages
Workflow
Monitoring
Orchestration Engine
Orchestration
Cloud Formation (Heat)
Foundational
Services
Nova, Cinder, Swift, Keystone, Quantum, Horizon
Software
Infrastructure
Cobbler
ISC DHCP
Hardware
Infrastructure
x86 Compute
Salt
BIND
Local Storage
RHEL 6.x
Network
LBaaS, DNSaaS
FWaaS
Hypervisor
Zabbix
Load
Balancer
PP Specific
11

12. CLOUD BEFORE INTEGRATION
WEB
F
Z
F
Z
F
Z
“Stateless & Disposable”
F
Z
F
Z
KVM
Local Disk
F
Z
F
Z
F
Z
ESX
5.0u2
KVM
Local Disk
F
Z
MID
Cloud Management Zone
VCenter Management
F
Z
F
Z
ESX
5.0u2
ESX
5.0u2
Shared
Storage
Shared
Storage
Shared
Storage
Physical
Non-virtualized
F
Z
ESX
5.0u2
Shared
Storage
Physical
Non-virtualized
DATABASE & RESTRICTED ZONE
FZ = Logical Fault Zones

13. SIDE-BY-SIDE
13

14. CLOUD AFTER INTEGRATION
F
Z
F
Z
F
Z
F
Z
F
Z
WEB
F
Z
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
MID
Cloud Management Zone
F
Z
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
DATABASE & RESTRICTED ZONE

15. COMPARING
But isn’t Openstack a direct replacement for ESX? Why would
you keep them both?
ESX/Vsphere != Openstack
NOVA != vSphere || vCenter || ESXi
NOVA =~ vCD, vCAC
KVM =~ ESX
To connect to any hypervisor, the Openstack cloud ‘proxies’
connections to any supported hypervisor via Nova. That
abstracts the ‘Cloud’ from the hypervisor
15

16. BRINGING ESX ‘INTO’ THE CLOUD
• Equivalent functionality on KVM and ESX
• Full birth to death lifecycle management of virtual machines
− Build new, power on, power off, console, rebuild, delete
• Auto-configuration of host resources following t-shirt sizes
standards
− CPU, RAM, NIC, IP, OS Version
• IP Address Management
• Build from “Snapshot”/”Template”
• Deploy resources following appropriate fault zone model
• Must work from within single Horizon/Asgard interface
16

17. HYPERVISOR REQUIREMENTS
• VSphere/ESX 5.1
− 5.0 works but many, many, many back-ports / tweaks
• Single security zone per hypervisor
− No sharing of confidential & non-confidential on same hardware (PCI)
• Openstack management network communication
− This is NOT necessarily the VKERNEL network
17

18. STORAGE REQUIREMENTS
• Shared storage required
− Data Store Cluster
− Single Data Store support [ bug fix coming ]
• DRS Enabled with auto-placement
• Data Stores must be created in advance
− No Cinder support
18

19. OPENSTACK GRIZZLY
⁃ O
penS ack C
t
ommand Line T
ools (nova-client, swif t-client, et c.)
⁃ C
loud M anagement T
ools (Right scale, E rat ius, et c.)
nst
⁃ G t ools (C
UI
yberduck, iPhone client, et c.)
Int er net
O
penS ack
t
O
bject API
O
penS ack
t
C
omput e API
O
penS ack
t
Image API
O
penStack
Identity
AP
I
O
penStack
D
ashboard
H (S)
TTP
Amazon
Web Ser vices
E 2 API
C
VNC VMRC
/
/ Spice
O
penS ack
t
Block S orage API
t
Hor izon
O
penS ack
t
Net wor k API
O
penStack
O
bject AP
I
O
penStack
Image AP
I
O
penS ack
t
O
bject API
swif t-proxy
O
penStack C
ompute
AP /
I
Admin AP
I
O
penStack
Identity
AP
I
O
penS ack
t
O
penS ack
t
Block S orage API Block S orage API
t
t
nova-api
O
penS ack
t
Image API
glance-api
(O E 2, Met adat a, Admin)
S, C
nova-comput e
nova-cert/
objectstore
glance-regist r y
cont ainer
object
cinder-api
nova-console
nova-*proxy
O
penS ack
t
Image
API
memcached
account
O
penS ack
t
Net wor k API
O
penS ack
t
Net wor k API
H
TTP(S)
cinder-volume
quant um
agent (s)
nova
dat abase
object
st ore
O
penStack
Identity
AP
I
O
penS ack O
t
bject St ore
Queue
net wor k
provider
quant um
plugin(s)
quant um
dat abase
Queue
volume provider
Queue
hyper visor
cont ainer
D
B
cinder-backup
libvirt, XenAPI, et c.
glance
dat abase
account
D
B
quant um-ser ver
cinder
dat abase
nova-conduct or
nova-consoleauth
cinder-scheduler
http://www.solinea.com
O
penStack
Identity
AP
I
O
penS ack
t
Image Ser vice
nova-scheduler
O
penS ack C
t
omput e
O
penS ack
t
Block S orage
t
O
penS ack
t
Net wor k Ser vice
O
penStack
Identity
AP
I
O
penStack
Identity AP
I
keyst one
(ser vice & admin APIs)
O
penStack
Identity
Service
t oken backend
cat alog
backend
policy
backend
O
penStack O
bject AP
I
O
penStack
Identity
AP
I
O
penStack
Identity
AP
I
ident it y
backend

20. ITS ALL ABOUT NOVA

21. CONFIG OF NOVA
Nova is the project name for OpenStack Compute, a cloud computing fabric controller,
the main part of an IaaS system. Individuals and organizations can use Nova to host
and manage their own cloud computing systems.
#compute_driver = libvirt.LibvirtDriver
compute_driver = vmwareapi.VMwareVCDriver
Can be multiple
vmwareapi_host_ip=192.168.20.50
clusters now!
vmwareapi_host_username=root
vmwareapi_host_password=vmware
vmwareapi_cluster_name=openstack_test
vmwareapi_wsdl_loc=https://192.168.20.50/sdk/vimService.wsdl
Vcenter 5.1 Appliance
21
Confidential and Proprietary

22. GLANCE AND IMAGES
Rules for Glances images for VMWare
• Saved in VMDK Format
• Imported as VMDK Format
• Thick Provisioned VMDK Required
• No split VMDK allowed (must be merged)
• In a multi-hypervisor cloud, all images are separate
‘per hypervisor’ (no launching KVM VM’s on ESX)
glance add name=”MYMACHINE.vmdk" disk_format=vmdk container_format=bare
is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated"
vmware_ostype="otherGuest" < /path/to/MYMACHINE.vmdk
22
Confidential and Proprietary

23. BUILDING AND INSTALLING OS
• Kickstart
• Build a small root disk
• Use kickstart to image machine
• Post-install with puppet to customize machine and
add additional mount points depending on
application requirements
• Image Deploy
• Currently does not support ‘config-drive’
• Need Guest Tools to ‘duplicate’ functionality
23
Confidential and Proprietary

24. WHAT ABOUT THE NETWORK
•
24
Quantum requires NVP 3.2
• Cannot talk directly to VSphere API to allocate VDS
Port to NIC
• Implemented via vAPP – integration bridge
• Configured as separate transport zone within Nicira
Confidential and Proprietary

25. WHAT’S LEFT
• Component “at-scale” testing
Currently manage “tens” at a time, need to move
to “hundreds” or “thousands”
• Most fixes scheduled to go into Havanna, every
bug-fix needs to be reviewed and possible backported to Grizzly
• Multiple Data Store enumeration on a cluster
• Full Certification on VCE© VBLOCK with Vision
Intelligent Operations, auto-upgrades, and full
Openstack support of all components
25
Confidential and Proprietary

26. READING MATERIALS
•
•
•
•
•
•
http://www.solinea.com/2013/06/15/openstack-grizzly-architecture-revisited/ - Ken Pepple
http://www.slideshare.net/kenhui65/getting-started-with-openstack?ref=http://cloudarchitectmusings.com/2013/06/16/getting-started-with-openstack/ - Kenneth Hui
http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html - config-drive doc
http://docs.openstack.org/trunk/openstack-compute/admin/content/vmware.html - Openstack VMWARE doc
http://www.ebay.com - Buy It Now
http://www.paypal.com - and then Pay for it Here!

27. Interested?
DL-PayPal-Cloud-Hiring@ebay.com

28. THANK YOU

30. VSVC4994
Marriage of ESX and OpenStack at PayPal
Scott Carlson, PayPal
#VSVC4994