Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Optimizing Cloud Foundry and OpenStack for large scale deployments

13,189 views

Published on

Published in: Engineering, Technology

Optimizing Cloud Foundry and OpenStack for large scale deployments

  1. 1. 1 OpenStack Summit May 12-16, 2014 Atlanta, Georgia Optimizing OpenStack for large scale Cloud Foundry deployments Animesh Singh, Daniel Krook, Jason Anderson, Kalonji Bankole @animeshsingh @danielkrook @andersonljason @k_bankole
  2. 2. 2 © 2014 IBM Corporation OAuth OSLC Infrastructure as a Service Platform as a Service Software as a Service API economy Cloud operating environment Software- defined environment TOSCA http://www.ibm.com/developerworks/cloud/library/cl-open-architecture/ IBM cloud and open technologies At all tiers, IBM is committed to building its cloud on an open cloud architecture Products and services built on open source and open standards benefit IBM and customers
  3. 3. 3 © 2014 IBM Corporation Gold Sponsors IBM is #2 in contributions to integrated projects IBM is working to accelerate OpenStack Foundation success… Because an open interoperable Cloud is critical for flexible cloud deployment and customer success… IBM has 15 core contributors15 IBMers working on OpenStack – from formation of the Foundation to Code Quality & New Function380 IBMers active in the projects 2 107 Mar 2013 May 2014859 Contributors 8,500 Individuals 2130 Contributors 16,100+ Individuals Exponential growth Platinum Sponsors OpenStack Participant Growth
  4. 4. 4 © 2014 IBM Corporation4 ©2014 IBM Corporation …And embracing Cloud Foundry as an open source PaaS Continuing our history of strengthening and extending open source TOTAL CONTRIBUTORS LINES OF CODE PULL REQ / WK COMPANIES 1,274 Average per month 2013 average: 133 12mo average: 98 Lifetime average: 58 711k 50+ 43
  5. 5. 5 © 2014 IBM Corporation Cloud Foundry PaaS Cloud Foundry services registry and runtime management layer. Components are dynamically discoverable and loosely coupled, exposing health through HTTP endpoints so agents can collect state and act on it. Cloud Foundry architectural overview User Authentication and Authorization Router DEA Pool Service Gateway Apps Service Connector Health Manager Messaging Cloud Controller Build Packs Cloud Foundry BOSH Micro Clouds Private Clouds Public Clouds Service Nodes
  6. 6. 6 © 2014 IBM Corporation Integrate!
  7. 7. 7 © 2014 IBM Corporation PaaS BOSH Cloud Provider Interface (CPI) Initially written for VMware and Amazon Web Services OpenStack CPI • OpenStack CPI written by PistonCloud Apache2 license • PistonCloud - An enterprise OpenStack company, founded by few of OpenStack founders Cloud Foundry – OpenStack integration OpenStack Cloud Servers Networking Storage Integration IaaS Hardware
  8. 8. 8 © 2014 IBM Corporation BOSH deployment process Deployment Manifest • Release name/version • # VMs, job params • Stemcells to use Stemcell • Base OS • BOSH agent Release • Name Jobs • Software packages • Config templates • Scripts BOSH Deployed Environment Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software packages
  9. 9. 9 © 2014 IBM Corporation Sample manifest
  10. 10. 10 © 2014 IBM Corporation 10 Cloud Provider Interface for OpenStack • OpenStack CPI is an implementation of the BOSH CPI. • Leverages the Fog Ruby gem for OpenStack • OpenStack CPI gem (Ruby package), the OpenStack Registry and Agents all reside on the Cloud Foundry Bosh repository https://github.com/cloudfoundry/bosh • Stemcell Image (Base OS Image) and sample manifest files for OpenStack exist
  11. 11. 11 © 2014 IBM Corporation Requirements: • Static / floating ips • Persistent disks • Outbound Internet connectivity • Custom flavors • Increased quota • Security groups 11 Integration requirements for OpenStack
  12. 12. 12 © 2014 IBM Corporation Automate!
  13. 13. 13 © 2014 IBM Corporation OpenStack discovery: • Extend the Fog OpenStack gem used by CPI to discover and OpenStack artifacts in an automated manner in the Cloud Foundry manifest • Pass OpenStack credentials and discover OpenStack compute and network information. Deployment automation: Tip one Discover Flavors Discover Networks Discover VM Subnet Discover DHCP IP Discover Keypairs Discover Gateway IP Discover Security groups
  14. 14. 14 © 2014 IBM Corporation Deployment automation: Tip two OpenStack pre-req setup: • Extend the Fog OpenStack gem used by CPI to setup Cloud Foundry OpenStack pre-reqs • Setup tenant quota • Keypairs • Custom Flavors • Security Groups setup Create Keypair Create Flavors Router, DEA, Cloud Controller, Service Nodes Create Security Groups Cf-private, cf-public Setup tenant quota
  15. 15. 15 © 2014 IBM Corporation Deployment automation: Tip three Cloud Foundry manifest generation: • Hard to manually create and edit Cloud Foundry manifests – which can extend up to 1000+ lines • Automation around template creation – an extension from the previous efforts of OpenStack Discovery and Cloud Foundry pre-req setup. • Can create templates for both nova-network and neutron based environments • Iterates through jobs and assigns static IPs to them if we are using an environment without PowerDNS.
  16. 16. 16 © 2014 IBM Corporation Deployment automation: Tip four Stemcell automation: • Community stemcell image has empty fstab and this makes the root partition auto detection to fail, and consequently provisioning to fail. Multiple Solutions: 1. Inject file into stemcell • Mount stemcell image using qemu-nbd • Inject /etc/fstab into mounted os • Create new image based on snapshot of modified os 2. Modify Openstack Configuration: • Change “libvirt_inject_partition” parameter in nova.conf from -1 (autodetect) to 1 (use first partition) 3. Modify bosh “Stemcell Builder” script • Script injects /etc/fstab into CentOS, but not Ubuntu (as seen here) 4. Install cloud-init in the image • Best solution to make it work in conjunction with metadata service Stemcell • Base OS • BOSH Agent
  17. 17. 17 © 2014 IBM Corporation Scale!
  18. 18. 18 © 2014 IBM Corporation Sample CF sizing for 1,000 small applications ~60 virtual machines • 20 high memory DEAs • 11 CF fabric components • 26 service broker and service nodes 160 CPU ~500 GB memory ~1.5 TB VM disk ~200 GB block storage
  19. 19. 19 © 2014 IBM Corporation Controller Node Our initial OpenStack cluster was built on powerful hardware… …but it was not ideal for Cloud Foundry Compute Nodes Compute Nodes Compute Node Nova Compute Nova Network Storage Node Cinder (volume) Glance (image) 1x 20x Dedicated Servers (bare metal) Nova API Nova Scheduler Keystone Horizon Database (MySQL) Messaging (Qpid) 1x 128 GB memory 6 cores 36 TB disk each
  20. 20. 20 © 2014 IBM Corporation Weaknesses in the OpenStack layout affected the stability of our Cloud Foundry configuration  Single points of failure in the networking, messaging, and storage configuration made BOSH unhappy. ‒ Qpid on Folsom memory leaks made the head node a bottleneck. ‒ Lost messages affected VM and storage creation, leaving many resources in stuck states. ‒ Single Nova API process design was overloaded with BOSH traffic ‒ VLAN per tenant design routed much traffic through the single head node, impacting both the fabric and brokered services. ‒ Cinder and Glance storage and APIs were located on a single node.  Bare metal allocation of resources was also inefficient ‒ Every OpenStack component ran as a process directly on the OS ‒ Made reacting to workload changes difficult, i.e., difficult to redistribute OpenStack components to react to Cloud Foundry demands
  21. 21. 21 © 2014 IBM Corporation We‟ve addressed the major issues OpenStack by distributing components more intelligently Moving some components to VMs, assigning controller, compute, data, storage roles to groups of components improves availability, flexibility, scalability and maintainability Compute Nodes Compute Nodes Compute Node Nova Compute Cloud Controller Data Node database (mysql) messaging (qpid) Cloud Controller Master Node nova-api nova-scheduler Keystone Horizon Cloud Controller Data Node database (MySql) messaging (Qpid) Cloud Controller Storage Node Cinder (volume) Glance (image) Nova Network Controller Node Nova API Nova Scheduler Keystone Horizon Storage Node Cinder (volume) Glance (image) 2x Data Node Database (MySQL) Messaging (RabbitMQ) 2x 15x Load Balancer Node HA Proxy Keepalive Load Balancer Node HA Proxy Keepalive Load Balancer Node HA Proxy KeepAlive 6x3x Virtual Machines (VMs) Dedicated Servers (bare metal)
  22. 22. 22 © 2014 IBM Corporation Matching the right hardware for OpenStack to host Cloud Foundry Learn more about these topics at sessions from our team Wednesday
  23. 23. 23 © 2014 IBM Corporation Optimize!
  24. 24. 24 © 2014 IBM Corporation Deploy optimization tips • Increase OpenStack API rate limits (/etc/nova/api-paste.ini) [filter:ratelimit] paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory limits = (POST, "*", .*, 9999, MINUTE); (POST, "*/servers", ^/servers, 9999, DAY); (PUT, "*", .*, 9999, MINUTE); (GET, "*changes-since*", .*changes-since.*, 9999, MINUTE); (DELETE, "*", .*, 9999, MINUTE) BOSH BOSH Administrator NATS PostGres Redis Blob Store Health Monitor Director Registry CloudProviderInterface(CPI) Cloud Foundry Environment mongo_gw mysql_gw redis_gw postgres_gw cloud_ctrler ccdb_ng router nats debian_nfs dea uaa_db uaa loggregator health_mgr rabbit_gw mongo_node mysql_node redis_node postgres_node rabbit_node OpenStack Environment OpenStack API
  25. 25. 25 © 2014 IBM Corporation Deploy optimization tips • Increase BOSH‟s NATS timeout • A large number of BOSH agents can overwhelm NATS • Place the following in your MicroBOSH manifest file: apply_spec: properties: nats: ping_interval: 30 ping_max_outstanding: 30 BOSH NATS mongo_gwmysql_gw redis_gwpostgres_gw cloud_ctrler ccdb_ng router nats debian_nfs dea uaa_db uaa loggregator health_mgr rabbit_gw mongo_nodemysql_node redis_nodepostgres_node rabbit_node
  26. 26. 26 © 2014 IBM Corporation Deploy optimization tips • Avoid name based security groups with nova-network • Name based security groups require message bus activity and database updates proportional to the number of existing VMs • Use an OpenStack scheduler which distributes VM load • Ensure that jobs are distributed across compute nodes instead of grouping • The default scheduler has the correct behavior compute_scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler
  27. 27. 27 © 2014 IBM Corporation Security optimization tips Guiding principle: Use the most limited permissions required to complete the job 1. Use tenant credentials • Do not use full admin credentials in your BOSH manifest 2. Only open the security ports which are required
  28. 28. 28 © 2014 IBM Corporation Security optimization tips 3. Separate your VM network and OpenStack management network • One challenge with BOSH VMs deployed on OpenStack • Setup pin holes for those particular VMs (Director, OpenStack Registry) 3. Separate Cloud Foundry‟s core fabric from other brokered services • Services connected using the service broker should be on a separate network • Prevents compromised services from effecting Cloud Foundry
  29. 29. 29 © 2014 IBM Corporation What next? Join us at the Cloud Foundry Summit
  30. 30. 30 © 2014 IBM Corporation Monday, May 12 – Room B314 12:05-12:45 Wednesday, May 14 - Room B312 9:00-9:40 9:50-10:30 11:00-11:40 11:50-12:30 OpenStack is Rockin‟ the OpenCloud Movement! Who„s Next to Join the Band ? Angel Diaz, VP Open Technology and Cloud Labs David Lindquist, IBM Fellow, VP, CTO Cloud & Smarter Infrastructure Getting from enterprise ready to enterprise bliss - why OpenStack and IBM is a match made in Cloud heaven. Todd Moore - Director, Open Technologies and Partnerships Taking OpenStack beyond Infrastructure with IBM SmartCloud Orchestrator. Andrew Trossman - Distinguished Engineer, IBM Common Cloud Stack and SmartCloud Orchestrator IBM, SoftLayer and OpenStack - present and future Michael Fork - Cloud Architect IBM and OpenStack: Enabling Enterprise Cloud Solutions Now. Tammy Van Hove -Distinguished Engineer, Software Defined Systems IBM Sponsored Sessions
  31. 31. 31 © 2014 IBM Corporation Monday, May 12 3:40 - 4:20 3:40 - 4:20 Tuesday, May 13 11:15 - 11:55 2:00 - 2:40 5:30 - 6:10 5:30 - 6:10 Wednesday, May14 9:50 - 10:30 2:40 - 3:20 Thursday, May 15 9:50 - 10:30 1:30 - 2:10 2:20 - 3:00 IBM Technical Sessions
  32. 32. 32 Be sure to stop by the IBM booth to see some demos and get your rockin‟ OpenStack t-shirt while they last. Thank you !

×