1
Cloud Foundry and OpenStack – Marriage Made in Heaven !
Apps
Animesh Singh, Egle Sigler, Jason Anderson
@animeshsingh, @eglute, @andersonljason
CF Summit, San Francisco
June 9-11
2
What Is OpenStack
• Open Source Project for with a very large community
• Open source cloud computing platform for public and private clouds
• IaaS for managing compute, storage, and networking resource
• Standard and massively scalable cloud operating system
3
Who Is OpenStack
• Founded by Rackspace and NASA in 2010
• Open Source global community of developers, testers, operators, and users
4
OpenStack Community
76K Commits
2000 Developers
38K Tickets
55K Mail Messages
… and counting!
http://activity.openstack.org/dash/releases/index.html?data_dir=data/whole_project
5
OpenStack Architecture
Collection of well integrated projects:
• Compute (Nova)
• Networking (Neutron)
• Object Storage (Swift)
• Block Storage (Cinder)
• Identity(Keystone)
• Image Service (Glance)
• Dashboard (Horizon)
…and others
6
OpenStack Compute (Nova)
Compute is a cloud computing fabric controller, the main part of an IaaS system.
• Component based architecture: Quickly add new behaviors
• Highly available: Scale to very serious workloads
• Fault-Tolerant: Isolated processes avoid cascading failures
• Recoverable: Failures should be easy to diagnose, debug, and rectify
• Open Standards: Be a reference implementation for a community-driven api
• API Compatibility: Nova strives to provide API-compatible with popular systems like
Amazon EC2
7
OpenStack Networking (Neutron)
Network connectivity as a service, a system for managing networks and IP
addresses.
Advantages of software defined networking. Create and manage:
• Networks and subnets
• Routers
• Load balancers
• Gateways
• Floating IPs
8
OpenStack Block Storage (Cinder)
OpenStack Block Storage (Cinder) provides persistent block-level storage devices
for use with OpenStack compute instances.
• Fully integrated with Compute and Dashboard components
• Allows users to manage their own storage needs
• Appropriate for performance sensitive scenarios:
• database storage
• expandable file systems
• providing a server with access to raw block level storage
• Snapshot management provides for volume/data backup and restoration
9
OpenStack Dashboard
Horizon is the canonical implementation of OpenStack’s Dashboard, which provides a
web based user interface to OpenStack services including Nova, Swift, Keystone, etc.
10
Along with Rackspace, IBM is also working to accelerate
OpenStack Foundation success…
 Contributions across all releases
 Contributions to the latest Icehouse
release
http://activity.openstack.org/dash/releases/index.html
11
Why Cloud Foundry?
Meets Developer’s Needs
Focus on app development,
not provisioning VMs,
databases, messaging servers,
etc
Agile development model
Deploy and scale in seconds
Open Cloud Platform
There is an increasing appetite
for cloud-based mobile, social
and analytics applications from
line-of-business executives -
drives the need for a more open
cloud development platform
Compelling Community
Cloud Foundry has a compelling
community and emerging ecosystem
as well as a mature set of
capabilities and robustness
Platinum Founding Sponsors
1,274 711k
LINES OF CODETOTAL CONTRIBUTORS
12
IBM BlueMix (powered by Cloud Foundry)
http://bluemix.net
 IBM initiative to develop a
Platform as a Services
offering
 IBM and partner cloud
services
 Integrated DevOps with
both Browser and Eclipse-
based tools
Services
Lifecycle
Management
(JazzHub)
Application
Runtime
Runtimes &
Frameworks
Middleware Application Operational Mobile ExternalData
node java ruby Worklight
WebSphere
Liberty
Web IDE
(Eclipse Orion)
Eclipse IDE
Application
Composition
Environment
Create & Manage Services
Test/Run Test/Run
Explore
Services
Explore
Services
IBM Bluemix ™
Check In Code Check In Code
Come, experiment with our open beta at http://bluemix.net
13
Integrate !
14
PaaS
Cloud Foundry – OpenStack integration
Cloud
Integration
IaaS
UAA
Router
DEA Pool
Service Gateway Apps
Service Connector
Health Manager
Messaging
Cloud Controller
Build Packs
Service Nodes
Cloud Foundry BOSH
Cloud Provider Interface
• Static / floating ips
• Persistent disks
• Custom VM
Configurations
• Specialized Security
groups
• Outbound Internet
connectivity
15
BOSH deployment process
Deployment Manifest
• Release
name/version
• # VMs, job params
• Stemcells to use
Stemcell
• Base OS
• BOSH agent
Release
• Name
• Software packages
• Config templates
• Scripts
BOSH
Deployed
Cloud Foundry
Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software packages
16
Automate (Leverage the power of community) !
17
Leverage the power of Open Source Community
• Open technologies give us a tremendous amount of power and flexibility to integrate them together
seamlessly
• Lets see some of the examples of what we can do with the power of Open technologies to go from
bare metal hardware to a deployed Cloud Foundry ready to push apps
Open Source for Cloud Foundry and OpenStack
Automation!
18
OpenStack Installation:
• Leverage the open source Chef Cloud infrastructure Automation framework
• Requires information about hardware, network environment and software repositories.
• Automate OpenStack installation in approximately 45 minutes
Chef for OpenStack Install Automation
Setup Compute
Setup Network
Setup Controller
Setup Storage
19
OpenStack Discovery:
• Leverage the open source Fog gem to discover OpenStack artifacts in an automated manner
• Require OpenStack credentials and discover OpenStack compute and network information.
Fog for OpenStack Discovery Automation
Discover VM
Configuration Sizes
Discover Network
Subnets
Discover Network
Security Rules
Discover DHCP , DNS
Gateway and floating IPs
Discover Security
Credentials
20
Fog for OpenStack Setup Automation
Cloud Foundry Pre-req setup on OpenStack:
• Leverage the open source Fog gem to setup Cloud Foundry requirements in an automated manner
• Setup according to best practices and guidelines – still giving users the flexibility to change if desired
Create Security
Credentials
Create VM configs for
Router, DEAs, Cloud
Controller, Service
Nodes
Create network
Security Rules
Setup tenant quota
21
RUBY
BOSH and Ruby for Cloud Foundry Deployment
Automation
Cloud Foundry Deployment Automation
• Automate base OS image or Stemcell modification
• Automate Cloud Foundry deployment manifest file genration using Ruby ERB templates
• Automate upload of Cloud Foundry core release, services and runtime frameworks, followed by Cloud
Foundry deployment
Stemcell
Creation/Modification
and Upload
Generate BOSH and
Cloud Foundry
Manifest
Upload Cloud
Foundry core,
Services and runtime
Deploy Cloud Foundry
Deploy Microbosh
BOSH
22
Automate end to end deployments in under 3 hours !
• Previous example signify the power of Open source community
• Leveraging tools like Chef, Fog, BOSH, Ruby ERB from the community can help create powerful
automation which can be repeatedly and consistently replicated in under 3 hours !
Open way is the only way!
BOSH
RUBY
23
Scale!
24
Scaling Number of Apps on Cloud Foundry
Sample sizing for 1000 small applications
Total Resources: 160 VCPU, 500GB Memory, 1.5TB of VM disk
25
Scaling OpenStack
Optimize Internal Communication
• Configure OpenStack for scaled concurrency
Optimize Performance
• Configure OpenStack scheduler to evenly distribute load
Setup Highly Available Architecture for PaaS workloads
Load Balancer
Nodes
Data
Nodes
3x
Compute
Nodes
15
xStorage
Nodes
2x
Controller
Nodes
2x 6x
26
Scaling Cloud Foundry / BOSH
Optimize Internal Communication
• Configure messaging bus for VM communication
Optimized routing and bandwidth allocation
• Isolate Cloud Foundry components using multiple networks
Maintain Cloud Foundry’s Highly Available Architecture
Service
Gateways
10
xRouters
3x
Service
Nodes
15
x
DEAs
20x
Cloud
Controllers
2x
27
Summary
Why Cloud Foundry and OpenStack are a great fit ?
• 100% Open PaaS and IaaS solutions (no vendor lock-ins) with a growing
community of contributors and sponsors on both sides
• Power of Open Source community can be leveraged to automate the
deployment and lifecycle management of Cloud Foundry on OpenStack
• OpenStack meets Cloud Foundry integration requirements, and is totally
configurable and adaptable to handle the scale of a PaaS solution like Cloud
Foundry
Bottom Line: Its indeed a match made in Heaven !
28
What next? Join us at our Silicon Valley Meetups
http://www.meetup.com/BlueMix/
http://www.meetup.com/CloudFoundry/
For more technical details, refer to the slides and video here
http://www.slideshare.net/AnimeshSingh/optimizing-cloud-foundry-and-openstack-for-large-scale-deployments
https://www.openstack.org/assets/presentation-media/A-Practical-Approach-to-HA-Final.pdf
https://www.youtube.com/watch?v=jCwtV9n_ak4
https://www.youtube.com/watch?v=FsrCGkBo4Vg

Cloud Foundry and OpenStack – Marriage Made in Heaven !

  • 1.
    1 Cloud Foundry andOpenStack – Marriage Made in Heaven ! Apps Animesh Singh, Egle Sigler, Jason Anderson @animeshsingh, @eglute, @andersonljason CF Summit, San Francisco June 9-11
  • 2.
    2 What Is OpenStack •Open Source Project for with a very large community • Open source cloud computing platform for public and private clouds • IaaS for managing compute, storage, and networking resource • Standard and massively scalable cloud operating system
  • 3.
    3 Who Is OpenStack •Founded by Rackspace and NASA in 2010 • Open Source global community of developers, testers, operators, and users
  • 4.
    4 OpenStack Community 76K Commits 2000Developers 38K Tickets 55K Mail Messages … and counting! http://activity.openstack.org/dash/releases/index.html?data_dir=data/whole_project
  • 5.
    5 OpenStack Architecture Collection ofwell integrated projects: • Compute (Nova) • Networking (Neutron) • Object Storage (Swift) • Block Storage (Cinder) • Identity(Keystone) • Image Service (Glance) • Dashboard (Horizon) …and others
  • 6.
    6 OpenStack Compute (Nova) Computeis a cloud computing fabric controller, the main part of an IaaS system. • Component based architecture: Quickly add new behaviors • Highly available: Scale to very serious workloads • Fault-Tolerant: Isolated processes avoid cascading failures • Recoverable: Failures should be easy to diagnose, debug, and rectify • Open Standards: Be a reference implementation for a community-driven api • API Compatibility: Nova strives to provide API-compatible with popular systems like Amazon EC2
  • 7.
    7 OpenStack Networking (Neutron) Networkconnectivity as a service, a system for managing networks and IP addresses. Advantages of software defined networking. Create and manage: • Networks and subnets • Routers • Load balancers • Gateways • Floating IPs
  • 8.
    8 OpenStack Block Storage(Cinder) OpenStack Block Storage (Cinder) provides persistent block-level storage devices for use with OpenStack compute instances. • Fully integrated with Compute and Dashboard components • Allows users to manage their own storage needs • Appropriate for performance sensitive scenarios: • database storage • expandable file systems • providing a server with access to raw block level storage • Snapshot management provides for volume/data backup and restoration
  • 9.
    9 OpenStack Dashboard Horizon isthe canonical implementation of OpenStack’s Dashboard, which provides a web based user interface to OpenStack services including Nova, Swift, Keystone, etc.
  • 10.
    10 Along with Rackspace,IBM is also working to accelerate OpenStack Foundation success…  Contributions across all releases  Contributions to the latest Icehouse release http://activity.openstack.org/dash/releases/index.html
  • 11.
    11 Why Cloud Foundry? MeetsDeveloper’s Needs Focus on app development, not provisioning VMs, databases, messaging servers, etc Agile development model Deploy and scale in seconds Open Cloud Platform There is an increasing appetite for cloud-based mobile, social and analytics applications from line-of-business executives - drives the need for a more open cloud development platform Compelling Community Cloud Foundry has a compelling community and emerging ecosystem as well as a mature set of capabilities and robustness Platinum Founding Sponsors 1,274 711k LINES OF CODETOTAL CONTRIBUTORS
  • 12.
    12 IBM BlueMix (poweredby Cloud Foundry) http://bluemix.net  IBM initiative to develop a Platform as a Services offering  IBM and partner cloud services  Integrated DevOps with both Browser and Eclipse- based tools Services Lifecycle Management (JazzHub) Application Runtime Runtimes & Frameworks Middleware Application Operational Mobile ExternalData node java ruby Worklight WebSphere Liberty Web IDE (Eclipse Orion) Eclipse IDE Application Composition Environment Create & Manage Services Test/Run Test/Run Explore Services Explore Services IBM Bluemix ™ Check In Code Check In Code Come, experiment with our open beta at http://bluemix.net
  • 13.
  • 14.
    14 PaaS Cloud Foundry –OpenStack integration Cloud Integration IaaS UAA Router DEA Pool Service Gateway Apps Service Connector Health Manager Messaging Cloud Controller Build Packs Service Nodes Cloud Foundry BOSH Cloud Provider Interface • Static / floating ips • Persistent disks • Custom VM Configurations • Specialized Security groups • Outbound Internet connectivity
  • 15.
    15 BOSH deployment process DeploymentManifest • Release name/version • # VMs, job params • Stemcells to use Stemcell • Base OS • BOSH agent Release • Name • Software packages • Config templates • Scripts BOSH Deployed Cloud Foundry Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software packages
  • 16.
    16 Automate (Leverage thepower of community) !
  • 17.
    17 Leverage the powerof Open Source Community • Open technologies give us a tremendous amount of power and flexibility to integrate them together seamlessly • Lets see some of the examples of what we can do with the power of Open technologies to go from bare metal hardware to a deployed Cloud Foundry ready to push apps Open Source for Cloud Foundry and OpenStack Automation!
  • 18.
    18 OpenStack Installation: • Leveragethe open source Chef Cloud infrastructure Automation framework • Requires information about hardware, network environment and software repositories. • Automate OpenStack installation in approximately 45 minutes Chef for OpenStack Install Automation Setup Compute Setup Network Setup Controller Setup Storage
  • 19.
    19 OpenStack Discovery: • Leveragethe open source Fog gem to discover OpenStack artifacts in an automated manner • Require OpenStack credentials and discover OpenStack compute and network information. Fog for OpenStack Discovery Automation Discover VM Configuration Sizes Discover Network Subnets Discover Network Security Rules Discover DHCP , DNS Gateway and floating IPs Discover Security Credentials
  • 20.
    20 Fog for OpenStackSetup Automation Cloud Foundry Pre-req setup on OpenStack: • Leverage the open source Fog gem to setup Cloud Foundry requirements in an automated manner • Setup according to best practices and guidelines – still giving users the flexibility to change if desired Create Security Credentials Create VM configs for Router, DEAs, Cloud Controller, Service Nodes Create network Security Rules Setup tenant quota
  • 21.
    21 RUBY BOSH and Rubyfor Cloud Foundry Deployment Automation Cloud Foundry Deployment Automation • Automate base OS image or Stemcell modification • Automate Cloud Foundry deployment manifest file genration using Ruby ERB templates • Automate upload of Cloud Foundry core release, services and runtime frameworks, followed by Cloud Foundry deployment Stemcell Creation/Modification and Upload Generate BOSH and Cloud Foundry Manifest Upload Cloud Foundry core, Services and runtime Deploy Cloud Foundry Deploy Microbosh BOSH
  • 22.
    22 Automate end toend deployments in under 3 hours ! • Previous example signify the power of Open source community • Leveraging tools like Chef, Fog, BOSH, Ruby ERB from the community can help create powerful automation which can be repeatedly and consistently replicated in under 3 hours ! Open way is the only way! BOSH RUBY
  • 23.
  • 24.
    24 Scaling Number ofApps on Cloud Foundry Sample sizing for 1000 small applications Total Resources: 160 VCPU, 500GB Memory, 1.5TB of VM disk
  • 25.
    25 Scaling OpenStack Optimize InternalCommunication • Configure OpenStack for scaled concurrency Optimize Performance • Configure OpenStack scheduler to evenly distribute load Setup Highly Available Architecture for PaaS workloads Load Balancer Nodes Data Nodes 3x Compute Nodes 15 xStorage Nodes 2x Controller Nodes 2x 6x
  • 26.
    26 Scaling Cloud Foundry/ BOSH Optimize Internal Communication • Configure messaging bus for VM communication Optimized routing and bandwidth allocation • Isolate Cloud Foundry components using multiple networks Maintain Cloud Foundry’s Highly Available Architecture Service Gateways 10 xRouters 3x Service Nodes 15 x DEAs 20x Cloud Controllers 2x
  • 27.
    27 Summary Why Cloud Foundryand OpenStack are a great fit ? • 100% Open PaaS and IaaS solutions (no vendor lock-ins) with a growing community of contributors and sponsors on both sides • Power of Open Source community can be leveraged to automate the deployment and lifecycle management of Cloud Foundry on OpenStack • OpenStack meets Cloud Foundry integration requirements, and is totally configurable and adaptable to handle the scale of a PaaS solution like Cloud Foundry Bottom Line: Its indeed a match made in Heaven !
  • 28.
    28 What next? Joinus at our Silicon Valley Meetups http://www.meetup.com/BlueMix/ http://www.meetup.com/CloudFoundry/ For more technical details, refer to the slides and video here http://www.slideshare.net/AnimeshSingh/optimizing-cloud-foundry-and-openstack-for-large-scale-deployments https://www.openstack.org/assets/presentation-media/A-Practical-Approach-to-HA-Final.pdf https://www.youtube.com/watch?v=jCwtV9n_ak4 https://www.youtube.com/watch?v=FsrCGkBo4Vg

Editor's Notes

  • #22 Now lets consider the example of cloud foundry deployment itself. We discussed before how BOSH deploys Cloud Foundry. Now before BOSH starts the deployment, there are few other steps like Cloud Foundry template generation which can be automated via tapping into open source ruby libraries like ERB, or Embedded Ruby etc. Coming back to BOSH, different steps like creating and uploading a stem cell to microbosh deploy to finally deploying CloudFoundry with the required runtimes and services can all be automated as well.
  • #23 Great - so we just witnessed few of the examples of the power of Open source community, Leveraging Chef, fog, BOSH from the community can help create powerful automation which can be repeatedly and consistently replicated in under 3 hours !
  • #24 So the marriage of OpenStack and Cloud Foundry happened and end to end installation and configuration can be automated. What next ? Well we need to grow the family - or in other words deploy lots and lots of applications. My colleague Jason is now going to take us thorough how Cloud Foundry and OpenStack provide inherent characteristics for scaling
  • #26 Cloud Foundry has a highly available architecture and OpenStack needs to be configured in a similar fashion. This is one of the great selling points of OpenStack is its configurability.
  • #28 Cloud Foundry has a great scalability which needs to be mimicked at the IaaS Layer Internally we wanted to build a scaled out environment in our lab and we’ve determined the following points of consideration