Successfully reported this slideshow.

Optimizing Cloud Foundry and OpenStack for large scale deployments



OpenStack Summit
May 12-16, 2014
Atlanta, Georgia
Optimizing OpenStack for large scale
Cloud Foundry deployments
2 © 2014 IBM Corporation
as a Service
as a Service
as a Service
3 © 2014 IBM Corporation
Gold Sponsors
IBM is #2 in contributions to
integrated projects
IBM is working to accelerate Open...

YouTube videos are no longer supported on SlideShare

View original on YouTube

1 of 33
1 of 33

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Optimizing Cloud Foundry and OpenStack for large scale deployments

  1. 1. 1 OpenStack Summit May 12-16, 2014 Atlanta, Georgia Optimizing OpenStack for large scale Cloud Foundry deployments Animesh Singh, Daniel Krook, Jason Anderson, Kalonji Bankole @animeshsingh @danielkrook @andersonljason @k_bankole
  2. 2. 2 © 2014 IBM Corporation OAuth OSLC Infrastructure as a Service Platform as a Service Software as a Service API economy Cloud operating environment Software- defined environment TOSCA IBM cloud and open technologies At all tiers, IBM is committed to building its cloud on an open cloud architecture Products and services built on open source and open standards benefit IBM and customers
  3. 3. 3 © 2014 IBM Corporation Gold Sponsors IBM is #2 in contributions to integrated projects IBM is working to accelerate OpenStack Foundation success… Because an open interoperable Cloud is critical for flexible cloud deployment and customer success… IBM has 15 core contributors15 IBMers working on OpenStack – from formation of the Foundation to Code Quality & New Function380 IBMers active in the projects 2 107 Mar 2013 May 2014859 Contributors 8,500 Individuals 2130 Contributors 16,100+ Individuals Exponential growth Platinum Sponsors OpenStack Participant Growth
  4. 4. 4 © 2014 IBM Corporation4 ©2014 IBM Corporation …And embracing Cloud Foundry as an open source PaaS Continuing our history of strengthening and extending open source TOTAL CONTRIBUTORS LINES OF CODE PULL REQ / WK COMPANIES 1,274 Average per month 2013 average: 133 12mo average: 98 Lifetime average: 58 711k 50+ 43
  5. 5. 5 © 2014 IBM Corporation Cloud Foundry PaaS Cloud Foundry services registry and runtime management layer. Components are dynamically discoverable and loosely coupled, exposing health through HTTP endpoints so agents can collect state and act on it. Cloud Foundry architectural overview User Authentication and Authorization Router DEA Pool Service Gateway Apps Service Connector Health Manager Messaging Cloud Controller Build Packs Cloud Foundry BOSH Micro Clouds Private Clouds Public Clouds Service Nodes
  6. 6. 6 © 2014 IBM Corporation Integrate!
  7. 7. 7 © 2014 IBM Corporation PaaS BOSH Cloud Provider Interface (CPI) Initially written for VMware and Amazon Web Services OpenStack CPI • OpenStack CPI written by PistonCloud Apache2 license • PistonCloud - An enterprise OpenStack company, founded by few of OpenStack founders Cloud Foundry – OpenStack integration OpenStack Cloud Servers Networking Storage Integration IaaS Hardware
  8. 8. 8 © 2014 IBM Corporation BOSH deployment process Deployment Manifest • Release name/version • # VMs, job params • Stemcells to use Stemcell • Base OS • BOSH agent Release • Name Jobs • Software packages • Config templates • Scripts BOSH Deployed Environment Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software packages
  9. 9. 9 © 2014 IBM Corporation Sample manifest
  10. 10. 10 © 2014 IBM Corporation 10 Cloud Provider Interface for OpenStack • OpenStack CPI is an implementation of the BOSH CPI. • Leverages the Fog Ruby gem for OpenStack • OpenStack CPI gem (Ruby package), the OpenStack Registry and Agents all reside on the Cloud Foundry Bosh repository • Stemcell Image (Base OS Image) and sample manifest files for OpenStack exist
  11. 11. 11 © 2014 IBM Corporation Requirements: • Static / floating ips • Persistent disks • Outbound Internet connectivity • Custom flavors • Increased quota • Security groups 11 Integration requirements for OpenStack
  12. 12. 12 © 2014 IBM Corporation Automate!
  13. 13. 13 © 2014 IBM Corporation OpenStack discovery: • Extend the Fog OpenStack gem used by CPI to discover and OpenStack artifacts in an automated manner in the Cloud Foundry manifest • Pass OpenStack credentials and discover OpenStack compute and network information. Deployment automation: Tip one Discover Flavors Discover Networks Discover VM Subnet Discover DHCP IP Discover Keypairs Discover Gateway IP Discover Security groups
  14. 14. 14 © 2014 IBM Corporation Deployment automation: Tip two OpenStack pre-req setup: • Extend the Fog OpenStack gem used by CPI to setup Cloud Foundry OpenStack pre-reqs • Setup tenant quota • Keypairs • Custom Flavors • Security Groups setup Create Keypair Create Flavors Router, DEA, Cloud Controller, Service Nodes Create Security Groups Cf-private, cf-public Setup tenant quota
  15. 15. 15 © 2014 IBM Corporation Deployment automation: Tip three Cloud Foundry manifest generation: • Hard to manually create and edit Cloud Foundry manifests – which can extend up to 1000+ lines • Automation around template creation – an extension from the previous efforts of OpenStack Discovery and Cloud Foundry pre-req setup. • Can create templates for both nova-network and neutron based environments • Iterates through jobs and assigns static IPs to them if we are using an environment without PowerDNS.
  16. 16. 16 © 2014 IBM Corporation Deployment automation: Tip four Stemcell automation: • Community stemcell image has empty fstab and this makes the root partition auto detection to fail, and consequently provisioning to fail. Multiple Solutions: 1. Inject file into stemcell • Mount stemcell image using qemu-nbd • Inject /etc/fstab into mounted os • Create new image based on snapshot of modified os 2. Modify Openstack Configuration: • Change “libvirt_inject_partition” parameter in nova.conf from -1 (autodetect) to 1 (use first partition) 3. Modify bosh “Stemcell Builder” script • Script injects /etc/fstab into CentOS, but not Ubuntu (as seen here) 4. Install cloud-init in the image • Best solution to make it work in conjunction with metadata service Stemcell • Base OS • BOSH Agent
  17. 17. 17 © 2014 IBM Corporation Scale!
  18. 18. 18 © 2014 IBM Corporation Sample CF sizing for 1,000 small applications ~60 virtual machines • 20 high memory DEAs • 11 CF fabric components • 26 service broker and service nodes 160 CPU ~500 GB memory ~1.5 TB VM disk ~200 GB block storage
  19. 19. 19 © 2014 IBM Corporation Controller Node Our initial OpenStack cluster was built on powerful hardware… …but it was not ideal for Cloud Foundry Compute Nodes Compute Nodes Compute Node Nova Compute Nova Network Storage Node Cinder (volume) Glance (image) 1x 20x Dedicated Servers (bare metal) Nova API Nova Scheduler Keystone Horizon Database (MySQL) Messaging (Qpid) 1x 128 GB memory 6 cores 36 TB disk each
  20. 20. 20 © 2014 IBM Corporation Weaknesses in the OpenStack layout affected the stability of our Cloud Foundry configuration  Single points of failure in the networking, messaging, and storage configuration made BOSH unhappy. ‒ Qpid on Folsom memory leaks made the head node a bottleneck. ‒ Lost messages affected VM and storage creation, leaving many resources in stuck states. ‒ Single Nova API process design was overloaded with BOSH traffic ‒ VLAN per tenant design routed much traffic through the single head node, impacting both the fabric and brokered services. ‒ Cinder and Glance storage and APIs were located on a single node.  Bare metal allocation of resources was also inefficient ‒ Every OpenStack component ran as a process directly on the OS ‒ Made reacting to workload changes difficult, i.e., difficult to redistribute OpenStack components to react to Cloud Foundry demands
  21. 21. 21 © 2014 IBM Corporation We‟ve addressed the major issues OpenStack by distributing components more intelligently Moving some components to VMs, assigning controller, compute, data, storage roles to groups of components improves availability, flexibility, scalability and maintainability Compute Nodes Compute Nodes Compute Node Nova Compute Cloud Controller Data Node database (mysql) messaging (qpid) Cloud Controller Master Node nova-api nova-scheduler Keystone Horizon Cloud Controller Data Node database (MySql) messaging (Qpid) Cloud Controller Storage Node Cinder (volume) Glance (image) Nova Network Controller Node Nova API Nova Scheduler Keystone Horizon Storage Node Cinder (volume) Glance (image) 2x Data Node Database (MySQL) Messaging (RabbitMQ) 2x 15x Load Balancer Node HA Proxy Keepalive Load Balancer Node HA Proxy Keepalive Load Balancer Node HA Proxy KeepAlive 6x3x Virtual Machines (VMs) Dedicated Servers (bare metal)
  22. 22. 22 © 2014 IBM Corporation Matching the right hardware for OpenStack to host Cloud Foundry Learn more about these topics at sessions from our team Wednesday
  23. 23. 23 © 2014 IBM Corporation Optimize!
  24. 24. 24 © 2014 IBM Corporation Deploy optimization tips • Increase OpenStack API rate limits (/etc/nova/api-paste.ini) [filter:ratelimit] paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory limits = (POST, "*", .*, 9999, MINUTE); (POST, "*/servers", ^/servers, 9999, DAY); (PUT, "*", .*, 9999, MINUTE); (GET, "*changes-since*", .*changes-since.*, 9999, MINUTE); (DELETE, "*", .*, 9999, MINUTE) BOSH BOSH Administrator NATS PostGres Redis Blob Store Health Monitor Director Registry CloudProviderInterface(CPI) Cloud Foundry Environment mongo_gw mysql_gw redis_gw postgres_gw cloud_ctrler ccdb_ng router nats debian_nfs dea uaa_db uaa loggregator health_mgr rabbit_gw mongo_node mysql_node redis_node postgres_node rabbit_node OpenStack Environment OpenStack API
  25. 25. 25 © 2014 IBM Corporation Deploy optimization tips • Increase BOSH‟s NATS timeout • A large number of BOSH agents can overwhelm NATS • Place the following in your MicroBOSH manifest file: apply_spec: properties: nats: ping_interval: 30 ping_max_outstanding: 30 BOSH NATS mongo_gwmysql_gw redis_gwpostgres_gw cloud_ctrler ccdb_ng router nats debian_nfs dea uaa_db uaa loggregator health_mgr rabbit_gw mongo_nodemysql_node redis_nodepostgres_node rabbit_node
  26. 26. 26 © 2014 IBM Corporation Deploy optimization tips • Avoid name based security groups with nova-network • Name based security groups require message bus activity and database updates proportional to the number of existing VMs • Use an OpenStack scheduler which distributes VM load • Ensure that jobs are distributed across compute nodes instead of grouping • The default scheduler has the correct behavior compute_scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler
  27. 27. 27 © 2014 IBM Corporation Security optimization tips Guiding principle: Use the most limited permissions required to complete the job 1. Use tenant credentials • Do not use full admin credentials in your BOSH manifest 2. Only open the security ports which are required
  28. 28. 28 © 2014 IBM Corporation Security optimization tips 3. Separate your VM network and OpenStack management network • One challenge with BOSH VMs deployed on OpenStack • Setup pin holes for those particular VMs (Director, OpenStack Registry) 3. Separate Cloud Foundry‟s core fabric from other brokered services • Services connected using the service broker should be on a separate network • Prevents compromised services from effecting Cloud Foundry
  29. 29. 29 © 2014 IBM Corporation What next? Join us at the Cloud Foundry Summit
  30. 30. 30 © 2014 IBM Corporation Monday, May 12 – Room B314 12:05-12:45 Wednesday, May 14 - Room B312 9:00-9:40 9:50-10:30 11:00-11:40 11:50-12:30 OpenStack is Rockin‟ the OpenCloud Movement! Who„s Next to Join the Band ? Angel Diaz, VP Open Technology and Cloud Labs David Lindquist, IBM Fellow, VP, CTO Cloud & Smarter Infrastructure Getting from enterprise ready to enterprise bliss - why OpenStack and IBM is a match made in Cloud heaven. Todd Moore - Director, Open Technologies and Partnerships Taking OpenStack beyond Infrastructure with IBM SmartCloud Orchestrator. Andrew Trossman - Distinguished Engineer, IBM Common Cloud Stack and SmartCloud Orchestrator IBM, SoftLayer and OpenStack - present and future Michael Fork - Cloud Architect IBM and OpenStack: Enabling Enterprise Cloud Solutions Now. Tammy Van Hove -Distinguished Engineer, Software Defined Systems IBM Sponsored Sessions
  31. 31. 31 © 2014 IBM Corporation Monday, May 12 3:40 - 4:20 3:40 - 4:20 Tuesday, May 13 11:15 - 11:55 2:00 - 2:40 5:30 - 6:10 5:30 - 6:10 Wednesday, May14 9:50 - 10:30 2:40 - 3:20 Thursday, May 15 9:50 - 10:30 1:30 - 2:10 2:20 - 3:00 IBM Technical Sessions
  32. 32. 32 Be sure to stop by the IBM booth to see some demos and get your rockin‟ OpenStack t-shirt while they last. Thank you !

Editor's Notes

  • IBM is committed to accelerating the success of the OpenStack foundation because interoperability in the Cloud is critical for flexible cloud deployments and ultimately customer success. As a member of the new Board of Directors, our goal is to help progress the platform, sustain a vibrant ecosystem, and position OpenStack as the IaaS platform of choice for cloud consumers and providers. IBM made a commitment in March 2013that ALL of our cloud offerings would be based on OpenStack), including our private clousofferindsto our public cloud offerings (currently Softlayer),  our optimized Pure Systems offerings (PureApp and PureFlex, etc) :which would be  transitioned over time to be hosted on OpenStack.Out private and public cloud portfolios including Softlayer and IBM Cloud Manager are already incorporating these standards.
  • Cloud Foundry PaaSAn application runs in a DEA. The Cloud Controller orchestrates the routing and lifecycle of all DEAs in the pool. Routers manage application traffic. Health Manager reports mismatched application states to the CC. A servicegateway provides an interface for services (native or external). A messaging bus manages all system communication. Apps are accessed directly through the router while web and CLI clients access Cloud Controller via RESTful services.
  • Stock photos for IBM!/wiki/Wc2b28dd4ba19_4a82_a6c1_8aaee8ffc00a/page/Photography
  • Here we introduce some of the BOSH vocbulary as we will be using it in the course of the presentation.Stemcells: In a cloud platform, VMs are usually cloned from a template. A stemcell is a VM template containing a standard Ubuntu distribution. A BOSH agent is also embedded in the template so that BOSH can take control of VMs cloned from the stemcell.Jobs: A job is a collection of software which serves a particular purpose (e.g. MySQL or the Cloud Controller). At deployment time, each job will be install on its own stemcell VM.Releases: A release contains a number of jobs which can be deployed into the target environment. A deployment can consist of more than one release and not every job in a release must be deployed.Deployment Manifest: A deployment manifest is the set of instructions BOSH uses to create a deployment. It is written using the YAML format. The manifest file contains the following sections:NameReleasesCompilationUpdateNetworksResource PoolsJobsPropertiesSample WordPress Manifest FileSample Cloud Foundry Manifest File
  • Now delving  more into technical details - we first need to have an OpenStack environment  implmented and configured accroding to some of the requirements from Cloud Foundry,  for example static ips and persistent disks,  as well as the VMs should have connectivity to the internet.  We also need to create some custom flavors for Cloud Foundry jobs, as well as the default scheduler for OpenStack needs to be set up so that it distributes VMs across different nodes in a random order.
  • Stock photos for IBM!/wiki/Wc2b28dd4ba19_4a82_a6c1_8aaee8ffc00a/page/Photography
  • The original OpenStack configuration supported generic high performance IaaS, PaaS, and SaaS workloads, but was not optimized for Cloud FoundryThe CloudFirst Factory was a cutting edge lab intended to incubate next generation cloud innovation on a clean slate provided by OpenStack (and other open source technologies) for the next release of SmartCloud Enterprise, IBM’s incumbent IaaS.The PaaS (a Cloud Operating Environment) effort competed with IaaS (SDE), SaaS (solution APIs), and other emerging cloud workloads such as Big Data and Analytics on OpenStack.[Diagram here]So while powerful, the CFF was not able to be tailored explicitly for Cloud Foundry (due to the other workloads and tenants on the cluster). This led to some particular bottlenecks and problems.
  • Messaging was a single point of failure, and since we were using Qpid on Red Hat, this was a less tested configuration than RabbitMQ on Ubuntu. In particular, a memory leak with Qpid on Folsom that were solved in Grizzly overloaded our head node often. [1] As an innovation lab, and since we were quite new to running OpenStack, we started simple and therefore did not use HA as our primary driver. The quickest time to value was a single Controller node and many Compute nodes. Therefore any issues on the head node (network, memory) directly affected the entire cluster.Also since we were responding to milestones to show innovation quickly, we did not enable complex storage configuration. No shared storage (that would support live migrations) and all of our Cinder storage was on a single node.The single Controller node architecture also had implications for network design, as all traffic flowed through the head nodeAt first everything was a bare metal process. In the lab, this wasn’t as much a cost implication, but it, along with resiliency and better distribution of resources gave us key lessons[1]
  • So, based on those lessons on running Cloud Foundry in the Cloud First Factory, we derived some key areas for improvement.These fed into the architecture we selected for OpenStack on SoftLayer.Hardware and network selection Bare metal and virtual machine mix Networking components and designComponent distribution
  • Mitigated with Clustered NATS
  • Key Point:
  • 9:00 - Getting from enterprise ready to enterprise bliss - why OpenStack and IBM is a match made in Cloud heavenAs a founding sponsor of the OpenStack Foundation, IBM's approach to OpenStack is simple - deliver high value contributions to OpenStack to make it THE best in class IaaS open source offering and then build IBM offerings on this foundation to deliver exception enterprise value to our clients. In this kickoff presentation for the IBM Track, Todd Moore, Director of Open Technologies and Partnerships, provides the answer to one of the most frequently asked questions - how is IBM adopting OpenStack across IBM Cloud Offerings? This presentation details the imperative of an Open Cloud Architecture to ensure interoperability and to avoid vendor lock in and sets the context for the IBM track sessions which provide a deeper dive on some of the IBM offerings shipping with OpenStack today including SmartCloud Orchestrator and SoftLayer.9:50 - IBM and OpenStack: Enabling Enterprise Cloud Solutions NowWhether you're just looking to get started with building your first OpenStack cloud infrastructure, or needing to expand your current project with additional capabilities - IBM has many ways to support you on your journey to a more open cloud environment. Need compute and/or storage hardware? This session will provide you with an overview of the portfolio of physical infrastructure options that IBM can deliver with OpenStack today (including the latest on Power8 and XIV Storage). Leverage existing IT infrastructure? IBM has the right mix of cloud solutions to help you make the most of your OpenStack journey . What's next, now? This session will discuss and demonstrate advanced scheduling, automation, and file serving capabilities that you can use with OpenStack today. 11:00 - Taking OpenStack beyond Infrastructure with IBM SmartCloud OrchestratorYou're using OpenStack as the infrastructure to build out your cloud environment. What a great choice !! Here are a few questions we think you should be considering:How are you connecting the services in your OpenStack cloud to your existing IT Management Systems like Monitoring, Backup, Patch?Are you providing a self service catalog for non-IT users to request services?Are you integrating with development tools for full lifecycle management of Heat Orchestration Templates?Can you deliver OpenStack through public, private, hybrid and expert systems? Come see how IBM SmartCloud Orchestrator can help you automate cloud deployed business processes, connecting your OpenStack environment to your Enterprise IT Management services.11:50 - IBM, SoftLayer and OpenStack - present and futureGet a detailed overview on why SoftLayer is the best global platform to build high-performance, highly-scalable OpenStack clouds, along with the latest advancements from IBM to improve the OpenStack experience in 2014. In this session you will learn how SoftLayer's commitment to bare-metal infrastructure can be leveraged to rapidly deploy OpenStack environments in near real-time with advanced capabilities like floating IPs, private and public networks across multiple regions or as an extension of existing, on-premises OpenStack installation. Additional topics include:- SoftLayer native capabilities through OpenStack APIs- How SoftLayer automation capabilities can save valuable time during the installation and setup phases- How open source projects can make it even easier to install and operate OpenStack on SoftLayerOther sponsored tracksRed Hat - full day Monday and TuesdayHP - Monday afternoonDell - Tuesday afternoonNetapp - Wed morning ( concurrent to IBM track )Intel - Wed afternoonVMware - Wed afternoon
  • Tempest: Integrated OpenStack TestingAn Overview of Cloud Auditing Support for OpenStackHosting hybrid (bare-metal + virtualized) applications on OpenStack Enhancing High Availability in Context of OpenStackTraining your cluster to take care of itself and let you eat dinner in  Optimizing OpenStack for large scale Cloud Foundry deployments Turning the Heat up on DevOps: Providing a web-based editing experience around Heat templatesLinux Containers - NextGen Virtualization for CloudA practical approach to deploying a highly available and optimally performing OpenStack.Federated Identity & Federated Service Provider Support for OpenStack Clouds (joint IBM, and Rackspace)Network Policy Abstractions in Neutron (joint IBM, Cisco and Midokura)Hybrid Cloud with OpenStack: Bridging Two Worlds
  • ×