Marriage of Openstack with KVM and ESX at PayPal OpenStack Summit Hong Kong Fall 2013

3,701 views
3,463 views

Published on

These are the slides from the presentation given at the OpenStack Summit in Hong Kong in Fall 2013

PayPal has adopted a hypervisor agnostic stance within our Openstack Grizzly cloud. This presentation will cover the details surrounding our grizzly implementation and integration of both KVM and ESX hypervisors under one management umbrella. Grizzly deployment details configuration details for ESX integration Reasons for execution of this strategy benefits and pitfalls of this plan This will be an audience modified presentation of one that I am giving at VMWorld 2013 in San Francisco in August 2013.

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,701
On SlideShare
0
From Embeds
0
Number of Embeds
23
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • Paypal is a Payments companyBetter for merchantsEasier for youCan be considered your digital walletWe’re not just on the internet, but we are also in store in places like Home Depot and GNCScale is importantShopping habits importantPaypal scales to meet needs worldwide and can be deployed in-country depending on local laws.
  • PayPal moved from 0% virtualized in production to 90% virtualized in one year.We took advantage of the converged infrastructure to shorten ramp up time and utilized resources from the vendor supplierAPI used for monitoring and deployment scriptsThere is a predictible scaling pattern for the front-tier on paypal.com and measuring the resultant output of the VM’s allowed us to scale as appropriate to meet demandVBLOCK measured at 99.999% availability which gave us confidence in running paypal apps on it
  • Clouds are a work in progress
  • Shift from Enterprise design model to cloud-based designElastically scale and self-heal infrastructure to accommodate unpredictable usage patterns of customers and internet commerceSeparate rapidly iterating customer experiences from core servicesreduce overall cost per transaction within the environment
  • Open Source to enable innovation fasterDefine Standards and APIsIt enables vendors to exactly know what large scale operators need to manage their cloudIndustry best practices prevents “reinventing the wheel”
  • Two Entry Points for InfrastructurePayPal Product DevelopersCloud Operators to manage CloudCentrally Orchestrated using HeatLocal StorageHP 4X600 GB(MirrorCisco 4948 & Arista 7050Nicira NVPF5 10.2.2 LB
  • New Hardware PlatformAll new Application PlatformGive infrastructure to developersFault Zones – Built to support maintenance activities and disposable compute for stateless load balanced poolsFront and business tiers separate infrastructure to retain separation for PCI
  • New Hardware PlatformAll new Application PlatformGive infrastructure to developers
  • Although many have mistakenly made direct comparisons between OpenStack Nova and vSphere, that is actually quite inaccurate since Nova actually sits at a layer above the hypervisor layer. OpenStack in general and Nova in paticular, is most analogous to vCloud Director (vCD) and vCloud Automation Center (vCAC), and not ESXi or even vCenter. In fact, it is very important to remember that Nova itself does NOT come with a hypervisor but manages multiple hypervisors, such as KVM or ESXi. Nova orchestrate these hypervisors via APIs and drivers. The list of supported hypervisors include KVM, vSphere, Xen, and others; a detailed list of what is supported can be found on the OpenStack Hypervisor Support Matrix.
  • Marriage of Openstack with KVM and ESX at PayPal OpenStack Summit Hong Kong Fall 2013

    1. 1. MARRIAGE OF OPENSTACK WITH KVM AND ESX AT PAYPAL MULTI-VENDOR AGILITY Open Stack Summit – Hong Kong - 2013
    2. 2. ABOUT PAYPAL PayPal offers flexible and innovative payment solutions for consumers and merchants of all sizes. • 137,000,000 Users. • $300,000 Payments processed by PayPal each minute. • 193 markets / 26 currencies. • PayPal is the World‟s Most Widely Used Digital Wallet. 2
    3. 3. WHY WE VIRTUALIZED ON ESX • In 2011/2012, 90% of the PayPal front-end was virtualized on ESX 5.0u1 • Primary Criteria − Stability, performance, industry expertise, availability of experts • Standardized on VCE VBLOCK© for initial implementation • Fully consumable API • Load-test harness well understood in industry (specInt -> vMark) − Predictable scaling pattern for horizontally scaled workloads 3
    4. 4. CLOUD 4
    5. 5. PAYPAL INTERNAL CLOUD 2012/2013 Shift toward an internal cloud model • Shift from Enterprise design model to cloud-based design • Elastically scale and self-heal infrastructure to accommodate unpredictable usage patterns of customers and internet commerce • Separate rapidly iterating customer experiences from core services • reduce overall cost per transaction within the environment 5
    6. 6. CLOUD IS THE GREAT ENABLER ENABLE THE DEVELOPER ENABLE THE BUSINESS One-Click Developer Self Service Global Compute & Data Fulfillment Payment Delivery SelfOrganizing & Optimizing Infrastructure System Intelligence Driven Operation Code 6 Deploy Enjoy
    7. 7. PAYPAL CLOUD PLATFORM – GUIDING PRINCIPLES • Technology − Adopt Open Source Solutions where ever possible − No Vendor Lock-in − Industry Best Practices − Leverage Industry/ebay Inc Investments • Functionality − Self-Service tool for application life cycle management. − Robust Automation & Orchestration − Seamless On-Demand Capacity Fulfillment 7
    8. 8. OPENSTACK PayPal deploying Openstack in order to help transform our global infrastructure into an agile and open cloud platform. Agility - time to market for customer facing services Agility - speed to service developer requests for VM resources Agility – utilize the engineering culture of PayPal to deliver specialized cloud services where needed 8
    9. 9. TECHNOLOGY STACK User Interface Operations Portal DEVS Deployment Portal Horizon, Ceilometer Traffic Mgmt Monitoring Metering Stages Workflow Monitoring Orchestration Engine Orchestration Cloud Formation (Heat) Foundational Services Nova, Cinder, Swift, Keystone, Quantum, Horizon Software Infrastructure Cobbler ISC DHCP Hardware Infrastructure x86 Compute Salt BIND Local Storage RHEL 6.x Network LBaaS, DNSaaS FWaaS Hypervisor Zabbix Load Balancer PP Specific 9
    10. 10. CLOUD BEFORE INTEGRATION WEB F Z F Z F Z KVM Local Disk “Stateless & Disposable” F Z MID Cloud Management Zone VCenter Management F Z F Z F Z KVM Local Disk F Z F Z ESX 5.0u2 Shared Storage F Z ESX 5.0u2 Shared Storage F Z ESX 5.0u2 Shared Storage Physical Non-virtualized F Z ESX 5.0u2 Shared Storage Physical Non-virtualized DATABASE & RESTRICTED ZONE FZ = Logical Fault Zones
    11. 11. SIDE-BY-SIDE 11
    12. 12. CLOUD AFTER INTEGRATION F Z F Z F Z F Z F Z WEB F Z KVM Local Disk ESX 5.0u2 Shared Storage Physical Non-virtualized MID Cloud Management Zone F Z KVM Local Disk ESX 5.0u2 Shared Storage Physical Non-virtualized DATABASE & RESTRICTED ZONE
    13. 13. COMPARING But isn‟t Openstack a direct replacement for ESX? Why would you keep them both? ESX/Vsphere != Openstack NOVA != vSphere || vCenter || ESXi NOVA =~ vCD, vCAC KVM =~ ESX To connect to any hypervisor, the Openstack cloud „proxies‟ connections to any supported hypervisor via Nova. That abstracts the „Cloud‟ from the hypervisor 13
    14. 14. BRINGING ESX „INTO‟ THE CLOUD • Equivalent functionality on KVM and ESX • Full birth to death lifecycle management of virtual machines − Build new, power on, power off, console, rebuild, delete • Auto-configuration of host resources following t-shirt sizes standards − CPU, RAM, NIC, IP, OS Version • IP Address Management • Build from “Snapshot”/”Template” • Deploy resources following appropriate fault zone model • Must work from within single Horizon/Asgard interface 14
    15. 15. HYPERVISOR REQUIREMENTS • ESX 5.1 − 5.0 works but too many back-ports (for us) / tweaks • Single security zone per hypervisor − No sharing of confidential & non-confidential on same hardware (PCI) • Openstack management network communication − This is NOT necessarily the VKERNEL network 15
    16. 16. STORAGE REQUIREMENTS • “Shared storage” required − Data Store Cluster − Single Data Store support • DRS Enabled with auto-placement • Data Stores must be created in advance − No Cinder support 16
    17. 17. OPENSTACK GRIZZLY ⁃ O penS ack C t ommand Line T ools (nova-client, swif t-client, et c.) ⁃ C loud M anagement T ools (Right scale, E rat ius, et c.) nst ⁃ G t ools (C UI yberduck, iPhone client, et c.) Int er net O penS ack t O bject API O penS ack t C omput e API O penS ack t Image API O penStack Identity AP I O penStack D ashboard H (S) TTP Amazon Web Ser vices E 2 API C VNC VMRC / / Spice O penS ack t Block S orage API t Hor izon O penS ack t Net wor k API O penStack O bject AP I O penStack Image AP I O penS ack t O bject API swif t-proxy O penStack C ompute AP / I Admin AP I O penStack Identity AP I O penS ack t O penS ack t Block S orage API Block S orage API t t nova-api O penS ack t Image API glance-api (O E 2, Met adat a, Admin) S, C nova-comput e nova-cert/ objectstore glance-regist r y cont ainer object cinder-api nova-console nova-*proxy O penS ack t Image API memcached account O penS ack t Net wor k API O penS ack t Net wor k API H TTP(S) cinder-volume quant um agent (s) nova dat abase object st ore O penStack Identity AP I O penS ack O t bject St ore Queue net wor k provider quant um plugin(s) quant um dat abase Queue volume provider Queue hyper visor cont ainer D B cinder-backup libvirt, XenAPI, et c. glance dat abase account D B quant um-ser ver cinder dat abase nova-conduct or nova-consoleauth cinder-scheduler http://www.solinea.com O penStack Identity AP I O penS ack t Image Ser vice nova-scheduler O penS ack C t omput e O penS ack t Block S orage t O penS ack t Net wor k Ser vice O penStack Identity AP I O penStack Identity AP I keyst one (ser vice & admin APIs) O penStack Identity Service t oken backend cat alog backend policy backend O penStack O bject AP I O penStack Identity AP I O penStack Identity AP I ident it y backend
    18. 18. ITS ALL ABOUT NOVA
    19. 19. CONFIG OF NOVA Nova is the project name for OpenStack Compute, a cloud computing fabric controller, the main part of an IaaS system. Individuals and organizations can use Nova to host and manage their own cloud computing systems. #compute_driver = libvirt.LibvirtDriver compute_driver = vmwareapi.VMwareVCDriver Can be multiple vmwareapi_host_ip=192.168.20.50 clusters now! vmwareapi_host_username=root vmwareapi_host_password=vmware vmwareapi_cluster_name=openstack_test vmwareapi_wsdl_loc=https://192.168.20.50/sdk/vimService.wsdl Vcenter 5.1 Appliance 19 Confidential and Proprietary
    20. 20. GLANCE AND IMAGES Rules for Glances images for VMWare • Saved in VMDK Format • Imported as VMDK Format • Thick Provisioned VMDK Required • No split VMDK allowed (must be merged) • In a multi-hypervisor cloud, all images are separate „per hypervisor‟ (no launching KVM VM‟s on ESX) glance add name=”MYMACHINE.vmdk" disk_format=vmdk container_format=bare is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated" vmware_ostype="otherGuest" < /path/to/MYMACHINE.vmdk 20 Confidential and Proprietary
    21. 21. BUILDING AND INSTALLING OS • Kickstart • Build a small root disk • Use kickstart to image machine • Post-install with puppet to customize machine and add additional mount points depending on application requirements • Image Deploy • Currently does not support „config-drive‟ • Need Guest Tools to „duplicate‟ functionality 21 Confidential and Proprietary
    22. 22. WHAT ABOUT THE NETWORK • 22 Quantum requires NVP 3.2 • Cannot talk directly to VSphere API to allocate VDS Port to NIC • Uses vAPP – integration bridge or native in 5.5 • Configured as separate transport zone within Nicira Confidential and Proprietary
    23. 23. WHAT‟S LEFT • Component “at-scale” testing • Currently manage “tens” at a time, need to move to “hundreds” or “thousands” • Most fixes in Havanna, every bug-fix needs to be reviewed and possible back-ported to Grizzly • Multiple Data Store enumeration on a cluster • Full Certification on VCE VBLOCK with Vision Intelligent Operations, auto-upgrades, and full Openstack support of all components 23 Confidential and Proprietary
    24. 24. READING MATERIALS • • • • • • http://www.solinea.com/2013/06/15/openstack-grizzly-architecture-revisited/ - Ken Pepple http://www.slideshare.net/kenhui65/getting-started-with-openstack?ref=http://cloudarchitectmusings.com/2013/06/16/getting-started-with-openstack/ - Kenneth Hui http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html - config-drive doc http://docs.openstack.org/trunk/openstack-compute/admin/content/vmware.html - Openstack VMWARE doc http://www.ebay.com - Buy It Now http://www.paypal.com - and then Pay for it Here!
    25. 25. THANK YOU Interested? DL-PayPal-Cloud-Hiring@ebay.com

    ×