Marriage of ESX and OpenStack - PayPal - VMWorld US 2013


Published on

VSVC4994 - Marriage of ESX and OpenStack at PayPal

PayPal is quickly moving forward to utilize open source and open standards based technologies in the build-out of our private cloud. With our internal release of OpenStack software based on 'Grizzly' we have integrated ESX 5 support and now can deploy workloads against ESX as well as against KVM.

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Marriage of ESX and OpenStack - PayPal - VMWorld US 2013

  1. 1. VSVC4994 Marriage of ESX and OpenStack at PayPal Scott Carlson, PayPal #VSVC4994
  3. 3. ABOUT PAYPAL PayPal offers flexible and innovative payment solutions for consumers and merchants of all sizes. • 132,000,000 Users. • $300,000 Payments processed by PayPal each minute. • 193 markets / 25 currencies. • PayPal is the World’s Most Widely Used Digital Wallet. 3
  4. 4. WHY WE VIRTUALIZED ON ESX • 80% of the PayPal front-end is virtualized on VSphere 5.0u1 • Primary Criteria − Stability, performance, industry expertise, availability of experts • Standardized on VCE VBLOCK© for initial implementation • Fully consumable API • Load-test harness well understood in industry (specInt & vMark) − Predictable scaling pattern for horizontally scaled workloads 4
  5. 5. CLOUD 5
  6. 6. PAYPAL INTERNAL CLOUD 2012/2013 Shift toward an internal cloud model • Shift from Enterprise design model to cloud-based design • Elastically scale and self-heal infrastructure to accommodate unpredictable usage patterns of customers and internet commerce • Separate rapidly iterating customer experiences from core services • reduce overall cost per transaction within the environment 6
  7. 7. CLOUD IS THE GREAT ENABLER ENABLE THE DEVELOPER ENABLE THE BUSINESS One-Click Developer Self Service Global Compute & Data Fulfillment Payment Delivery SelfOrganizing & Optimizing Infrastructure System Intelligence Driven Operation Code 7 Deploy Enjoy
  8. 8. PAYPAL CLOUD PLATFORM – GUIDING PRINCIPLES • Technology − Adopt Open Source Solutions where ever possible − No Vendor Lock-in − Industry Best Practices − Leverage Industry/ebay Inc Investments • Functionality − Self-Service tool for application life cycle management. − Robust Automation & Orchestration − Seamless On-Demand Capacity Fulfillment 8
  9. 9. OPENSTACK PayPal deploying Openstack in order to help transform our global infrastructure into an agile and open cloud platform. Agility - time to market for customer facing services Agility - speed to service developer requests for VM resources Agility – utilize the engineering culture of PayPal to deliver specialized cloud services where needed 9
  11. 11. TECHNOLOGY STACK User Interface Operations Portal DEVS Deployment Portal Horizon, Ceilometer Traffic Mgmt Monitoring Metering Stages Workflow Monitoring Orchestration Engine Orchestration Cloud Formation (Heat) Foundational Services Nova, Cinder, Swift, Keystone, Quantum, Horizon Software Infrastructure Cobbler ISC DHCP Hardware Infrastructure x86 Compute Salt BIND Local Storage RHEL 6.x Network LBaaS, DNSaaS FWaaS Hypervisor Zabbix Load Balancer PP Specific 11
  12. 12. CLOUD BEFORE INTEGRATION WEB F Z F Z F Z “Stateless & Disposable” F Z F Z KVM Local Disk F Z F Z F Z ESX 5.0u2 KVM Local Disk F Z MID Cloud Management Zone VCenter Management F Z F Z ESX 5.0u2 ESX 5.0u2 Shared Storage Shared Storage Shared Storage Physical Non-virtualized F Z ESX 5.0u2 Shared Storage Physical Non-virtualized DATABASE & RESTRICTED ZONE FZ = Logical Fault Zones
  13. 13. SIDE-BY-SIDE 13
  14. 14. CLOUD AFTER INTEGRATION F Z F Z F Z F Z F Z WEB F Z KVM Local Disk ESX 5.0u2 Shared Storage Physical Non-virtualized MID Cloud Management Zone F Z KVM Local Disk ESX 5.0u2 Shared Storage Physical Non-virtualized DATABASE & RESTRICTED ZONE
  15. 15. COMPARING But isn’t Openstack a direct replacement for ESX? Why would you keep them both? ESX/Vsphere != Openstack NOVA != vSphere || vCenter || ESXi NOVA =~ vCD, vCAC KVM =~ ESX To connect to any hypervisor, the Openstack cloud ‘proxies’ connections to any supported hypervisor via Nova. That abstracts the ‘Cloud’ from the hypervisor 15
  16. 16. BRINGING ESX ‘INTO’ THE CLOUD • Equivalent functionality on KVM and ESX • Full birth to death lifecycle management of virtual machines − Build new, power on, power off, console, rebuild, delete • Auto-configuration of host resources following t-shirt sizes standards − CPU, RAM, NIC, IP, OS Version • IP Address Management • Build from “Snapshot”/”Template” • Deploy resources following appropriate fault zone model • Must work from within single Horizon/Asgard interface 16
  17. 17. HYPERVISOR REQUIREMENTS • VSphere/ESX 5.1 − 5.0 works but many, many, many back-ports / tweaks • Single security zone per hypervisor − No sharing of confidential & non-confidential on same hardware (PCI) • Openstack management network communication − This is NOT necessarily the VKERNEL network 17
  18. 18. STORAGE REQUIREMENTS • Shared storage required − Data Store Cluster − Single Data Store support [ bug fix coming ] • DRS Enabled with auto-placement • Data Stores must be created in advance − No Cinder support 18
  19. 19. OPENSTACK GRIZZLY ⁃ O penS ack C t ommand Line T ools (nova-client, swif t-client, et c.) ⁃ C loud M anagement T ools (Right scale, E rat ius, et c.) nst ⁃ G t ools (C UI yberduck, iPhone client, et c.) Int er net O penS ack t O bject API O penS ack t C omput e API O penS ack t Image API O penStack Identity AP I O penStack D ashboard H (S) TTP Amazon Web Ser vices E 2 API C VNC VMRC / / Spice O penS ack t Block S orage API t Hor izon O penS ack t Net wor k API O penStack O bject AP I O penStack Image AP I O penS ack t O bject API swif t-proxy O penStack C ompute AP / I Admin AP I O penStack Identity AP I O penS ack t O penS ack t Block S orage API Block S orage API t t nova-api O penS ack t Image API glance-api (O E 2, Met adat a, Admin) S, C nova-comput e nova-cert/ objectstore glance-regist r y cont ainer object cinder-api nova-console nova-*proxy O penS ack t Image API memcached account O penS ack t Net wor k API O penS ack t Net wor k API H TTP(S) cinder-volume quant um agent (s) nova dat abase object st ore O penStack Identity AP I O penS ack O t bject St ore Queue net wor k provider quant um plugin(s) quant um dat abase Queue volume provider Queue hyper visor cont ainer D B cinder-backup libvirt, XenAPI, et c. glance dat abase account D B quant um-ser ver cinder dat abase nova-conduct or nova-consoleauth cinder-scheduler O penStack Identity AP I O penS ack t Image Ser vice nova-scheduler O penS ack C t omput e O penS ack t Block S orage t O penS ack t Net wor k Ser vice O penStack Identity AP I O penStack Identity AP I keyst one (ser vice & admin APIs) O penStack Identity Service t oken backend cat alog backend policy backend O penStack O bject AP I O penStack Identity AP I O penStack Identity AP I ident it y backend
  21. 21. CONFIG OF NOVA Nova is the project name for OpenStack Compute, a cloud computing fabric controller, the main part of an IaaS system. Individuals and organizations can use Nova to host and manage their own cloud computing systems. #compute_driver = libvirt.LibvirtDriver compute_driver = vmwareapi.VMwareVCDriver Can be multiple vmwareapi_host_ip= clusters now! vmwareapi_host_username=root vmwareapi_host_password=vmware vmwareapi_cluster_name=openstack_test vmwareapi_wsdl_loc= Vcenter 5.1 Appliance 21 Confidential and Proprietary
  22. 22. GLANCE AND IMAGES Rules for Glances images for VMWare • Saved in VMDK Format • Imported as VMDK Format • Thick Provisioned VMDK Required • No split VMDK allowed (must be merged) • In a multi-hypervisor cloud, all images are separate ‘per hypervisor’ (no launching KVM VM’s on ESX) glance add name=”MYMACHINE.vmdk" disk_format=vmdk container_format=bare is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated" vmware_ostype="otherGuest" < /path/to/MYMACHINE.vmdk 22 Confidential and Proprietary
  23. 23. BUILDING AND INSTALLING OS • Kickstart • Build a small root disk • Use kickstart to image machine • Post-install with puppet to customize machine and add additional mount points depending on application requirements • Image Deploy • Currently does not support ‘config-drive’ • Need Guest Tools to ‘duplicate’ functionality 23 Confidential and Proprietary
  24. 24. WHAT ABOUT THE NETWORK • 24 Quantum requires NVP 3.2 • Cannot talk directly to VSphere API to allocate VDS Port to NIC • Implemented via vAPP – integration bridge • Configured as separate transport zone within Nicira Confidential and Proprietary
  25. 25. WHAT’S LEFT • Component “at-scale” testing Currently manage “tens” at a time, need to move to “hundreds” or “thousands” • Most fixes scheduled to go into Havanna, every bug-fix needs to be reviewed and possible backported to Grizzly • Multiple Data Store enumeration on a cluster • Full Certification on VCE© VBLOCK with Vision Intelligent Operations, auto-upgrades, and full Openstack support of all components 25 Confidential and Proprietary
  26. 26. READING MATERIALS • • • • • • - Ken Pepple - Kenneth Hui - config-drive doc - Openstack VMWARE doc - Buy It Now - and then Pay for it Here!
  27. 27. Interested?
  28. 28. THANK YOU
  29. 29. VSVC4994 Marriage of ESX and OpenStack at PayPal Scott Carlson, PayPal #VSVC4994