These are the slides from the presentation given at the OpenStack Summit in Hong Kong in Fall 2013 PayPal has adopted a hypervisor agnostic stance within our Openstack Grizzly cloud. This presentation will cover the details surrounding our grizzly implementation and integration of both KVM and ESX hypervisors under one management umbrella. Grizzly deployment details configuration details for ESX integration Reasons for execution of this strategy benefits and pitfalls of this plan This will be an audience modified presentation of one that I am giving at VMWorld 2013 in San Francisco in August 2013.

1. MARRIAGE OF OPENSTACK
WITH KVM AND ESX AT
PAYPAL
MULTI-VENDOR AGILITY
Open Stack Summit – Hong Kong - 2013

2. ABOUT PAYPAL
PayPal offers flexible and innovative payment solutions for consumers
and merchants of all sizes.
• 137,000,000 Users.
• $300,000 Payments processed by PayPal each minute.
• 193 markets / 26 currencies.
• PayPal is the World‟s Most Widely Used Digital Wallet.
2

3. WHY WE VIRTUALIZED ON ESX
• In 2011/2012, 90% of the PayPal front-end was virtualized on ESX
5.0u1
• Primary Criteria
− Stability, performance, industry expertise, availability of experts
• Standardized on VCE VBLOCK© for initial implementation
• Fully consumable API
• Load-test harness well understood in industry (specInt -> vMark)
− Predictable scaling pattern for horizontally scaled workloads
3

4. CLOUD
4

5. PAYPAL INTERNAL CLOUD
2012/2013 Shift toward an internal cloud model
• Shift from Enterprise design model to cloud-based design
• Elastically scale and self-heal infrastructure to accommodate
unpredictable usage patterns of customers and internet commerce
• Separate rapidly iterating customer experiences from core
services
• reduce overall cost per transaction within the environment
5

6. CLOUD IS THE GREAT ENABLER
ENABLE THE DEVELOPER
ENABLE THE BUSINESS
One-Click
Developer
Self Service
Global
Compute &
Data
Fulfillment
Payment Delivery
SelfOrganizing &
Optimizing
Infrastructure
System Intelligence
Driven Operation
Code
6
Deploy
Enjoy

7. PAYPAL CLOUD PLATFORM –
GUIDING PRINCIPLES
• Technology
− Adopt Open Source Solutions where ever possible
− No Vendor Lock-in
− Industry Best Practices
− Leverage Industry/ebay Inc Investments
• Functionality
− Self-Service tool for application life cycle management.
− Robust Automation & Orchestration
− Seamless On-Demand Capacity Fulfillment
7

8. OPENSTACK
PayPal deploying Openstack in order to help transform our global
infrastructure into an agile and open cloud platform.
Agility - time to market for customer facing services
Agility - speed to service developer requests for VM resources
Agility – utilize the engineering culture of PayPal to
deliver specialized cloud services where needed
8

9. TECHNOLOGY STACK
User
Interface
Operations Portal
DEVS Deployment Portal
Horizon, Ceilometer
Traffic Mgmt
Monitoring
Metering
Stages
Workflow
Monitoring
Orchestration Engine
Orchestration
Cloud Formation (Heat)
Foundational
Services
Nova, Cinder, Swift, Keystone, Quantum, Horizon
Software
Infrastructure
Cobbler
ISC DHCP
Hardware
Infrastructure
x86 Compute
Salt
BIND
Local Storage
RHEL 6.x
Network
LBaaS, DNSaaS
FWaaS
Hypervisor
Zabbix
Load
Balancer
PP Specific
9

10. CLOUD BEFORE INTEGRATION
WEB
F
Z
F
Z
F
Z
KVM
Local Disk
“Stateless & Disposable”
F
Z
MID
Cloud Management Zone
VCenter Management
F
Z
F
Z
F
Z
KVM
Local Disk
F
Z
F
Z
ESX
5.0u2
Shared
Storage
F
Z
ESX
5.0u2
Shared
Storage
F
Z
ESX
5.0u2
Shared
Storage
Physical
Non-virtualized
F
Z
ESX
5.0u2
Shared
Storage
Physical
Non-virtualized
DATABASE & RESTRICTED ZONE
FZ = Logical Fault Zones

11. SIDE-BY-SIDE
11

12. CLOUD AFTER INTEGRATION
F
Z
F
Z
F
Z
F
Z
F
Z
WEB
F
Z
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
MID
Cloud Management Zone
F
Z
KVM
Local Disk
ESX 5.0u2
Shared Storage
Physical
Non-virtualized
DATABASE & RESTRICTED ZONE

13. COMPARING
But isn‟t Openstack a direct replacement for ESX? Why would
you keep them both?
ESX/Vsphere != Openstack
NOVA != vSphere || vCenter || ESXi
NOVA =~ vCD, vCAC
KVM =~ ESX
To connect to any hypervisor, the Openstack cloud „proxies‟
connections to any supported hypervisor via Nova. That
abstracts the „Cloud‟ from the hypervisor
13

14. BRINGING ESX „INTO‟ THE CLOUD
• Equivalent functionality on KVM and ESX
• Full birth to death lifecycle management of virtual machines
− Build new, power on, power off, console, rebuild, delete
• Auto-configuration of host resources following t-shirt sizes
standards
− CPU, RAM, NIC, IP, OS Version
• IP Address Management
• Build from “Snapshot”/”Template”
• Deploy resources following appropriate fault zone model
• Must work from within single Horizon/Asgard interface
14

15. HYPERVISOR REQUIREMENTS
• ESX 5.1
− 5.0 works but too many back-ports (for us) / tweaks
• Single security zone per hypervisor
− No sharing of confidential & non-confidential on same hardware (PCI)
• Openstack management network communication
− This is NOT necessarily the VKERNEL network
15

16. STORAGE REQUIREMENTS
• “Shared storage” required
− Data Store Cluster
− Single Data Store support
• DRS Enabled with auto-placement
• Data Stores must be created in advance
− No Cinder support
16

17. OPENSTACK GRIZZLY
⁃ O
penS ack C
t
ommand Line T
ools (nova-client, swif t-client, et c.)
⁃ C
loud M anagement T
ools (Right scale, E rat ius, et c.)
nst
⁃ G t ools (C
UI
yberduck, iPhone client, et c.)
Int er net
O
penS ack
t
O
bject API
O
penS ack
t
C
omput e API
O
penS ack
t
Image API
O
penStack
Identity
AP
I
O
penStack
D
ashboard
H (S)
TTP
Amazon
Web Ser vices
E 2 API
C
VNC VMRC
/
/ Spice
O
penS ack
t
Block S orage API
t
Hor izon
O
penS ack
t
Net wor k API
O
penStack
O
bject AP
I
O
penStack
Image AP
I
O
penS ack
t
O
bject API
swif t-proxy
O
penStack C
ompute
AP /
I
Admin AP
I
O
penStack
Identity
AP
I
O
penS ack
t
O
penS ack
t
Block S orage API Block S orage API
t
t
nova-api
O
penS ack
t
Image API
glance-api
(O E 2, Met adat a, Admin)
S, C
nova-comput e
nova-cert/
objectstore
glance-regist r y
cont ainer
object
cinder-api
nova-console
nova-*proxy
O
penS ack
t
Image
API
memcached
account
O
penS ack
t
Net wor k API
O
penS ack
t
Net wor k API
H
TTP(S)
cinder-volume
quant um
agent (s)
nova
dat abase
object
st ore
O
penStack
Identity
AP
I
O
penS ack O
t
bject St ore
Queue
net wor k
provider
quant um
plugin(s)
quant um
dat abase
Queue
volume provider
Queue
hyper visor
cont ainer
D
B
cinder-backup
libvirt, XenAPI, et c.
glance
dat abase
account
D
B
quant um-ser ver
cinder
dat abase
nova-conduct or
nova-consoleauth
cinder-scheduler
http://www.solinea.com
O
penStack
Identity
AP
I
O
penS ack
t
Image Ser vice
nova-scheduler
O
penS ack C
t
omput e
O
penS ack
t
Block S orage
t
O
penS ack
t
Net wor k Ser vice
O
penStack
Identity
AP
I
O
penStack
Identity AP
I
keyst one
(ser vice & admin APIs)
O
penStack
Identity
Service
t oken backend
cat alog
backend
policy
backend
O
penStack O
bject AP
I
O
penStack
Identity
AP
I
O
penStack
Identity
AP
I
ident it y
backend

18. ITS ALL ABOUT NOVA

19. CONFIG OF NOVA
Nova is the project name for OpenStack Compute, a cloud computing fabric controller,
the main part of an IaaS system. Individuals and organizations can use Nova to host
and manage their own cloud computing systems.
#compute_driver = libvirt.LibvirtDriver
compute_driver = vmwareapi.VMwareVCDriver
Can be multiple
vmwareapi_host_ip=192.168.20.50
clusters now!
vmwareapi_host_username=root
vmwareapi_host_password=vmware
vmwareapi_cluster_name=openstack_test
vmwareapi_wsdl_loc=https://192.168.20.50/sdk/vimService.wsdl
Vcenter 5.1 Appliance
19
Confidential and Proprietary

20. GLANCE AND IMAGES
Rules for Glances images for VMWare
• Saved in VMDK Format
• Imported as VMDK Format
• Thick Provisioned VMDK Required
• No split VMDK allowed (must be merged)
• In a multi-hypervisor cloud, all images are separate
„per hypervisor‟ (no launching KVM VM‟s on ESX)
glance add name=”MYMACHINE.vmdk" disk_format=vmdk container_format=bare
is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated"
vmware_ostype="otherGuest" < /path/to/MYMACHINE.vmdk
20
Confidential and Proprietary

21. BUILDING AND INSTALLING OS
• Kickstart
• Build a small root disk
• Use kickstart to image machine
• Post-install with puppet to customize machine and
add additional mount points depending on
application requirements
• Image Deploy
• Currently does not support „config-drive‟
• Need Guest Tools to „duplicate‟ functionality
21
Confidential and Proprietary

22. WHAT ABOUT THE NETWORK
•
22
Quantum requires NVP 3.2
• Cannot talk directly to VSphere API to allocate VDS
Port to NIC
• Uses vAPP – integration bridge or native in 5.5
• Configured as separate transport zone within Nicira
Confidential and Proprietary

23. WHAT‟S LEFT
• Component “at-scale” testing
• Currently manage “tens” at a time, need to
move to “hundreds” or “thousands”
• Most fixes in Havanna, every bug-fix needs to be
reviewed and possible back-ported to Grizzly
• Multiple Data Store enumeration on a cluster
• Full Certification on VCE VBLOCK with Vision
Intelligent Operations, auto-upgrades, and full
Openstack support of all components
23
Confidential and Proprietary

24. READING MATERIALS
•
•
•
•
•
•
http://www.solinea.com/2013/06/15/openstack-grizzly-architecture-revisited/ - Ken Pepple
http://www.slideshare.net/kenhui65/getting-started-with-openstack?ref=http://cloudarchitectmusings.com/2013/06/16/getting-started-with-openstack/ - Kenneth Hui
http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html - config-drive doc
http://docs.openstack.org/trunk/openstack-compute/admin/content/vmware.html - Openstack VMWARE doc
http://www.ebay.com - Buy It Now
http://www.paypal.com - and then Pay for it Here!

25. THANK YOU
Interested?
DL-PayPal-Cloud-Hiring@ebay.com