Cloud Foundry BOSH Where the Platform rubber meets the Infrastructure road - ChefConf

Cloud Foundry BOSH Where the Platform rubber meets the Infrastructure roadPatrick Chanezon with slides from Martin EnglundSenior Director Cloud Foundry EngineeringDeveloper Relationschanezonp@vmware.com @pmenglundhttp://twitter.com/chanezon ChefConf, San Francisco, May 2012Wednesday, May 16, 12

What is Cloud Computing? Cloud According to my daughter Eliette 2Wednesday, May 16, 12

Cloud Stack - Classic Pyramid Software As A Service Platform As A Service Infrastructure As A Service 3Wednesday, May 16, 12

Cloud Stack - Developer Perspective Software As A Service Platform As A Service Infrastructure As A Service 4Wednesday, May 16, 12

Infrastructure 5Wednesday, May 16, 12

IaaS/Virtualization getting mainstream § AWS, Joyent, Rackspace,... § Open Source projects: OpenStack, DeltaCloud, Eucalyptus § Automation: Chef, Juju § Standardization? DMTF § Inside the Firewall, Virtualization: VMware, Microsoft, Xen, KVM § 50% of workloads are virtualized § Easy to provision, manage instance...BUT § Still need to manage backups, software stacks, monitor, upgrades 6Wednesday, May 16, 12

With Infrastructure, you still need to build your own platform § Need to build a distributed platform on top of you infrastructure § Story of the AWS meltdown from last summer • http://blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html • http://www.readwriteweb.com/cloud/2010/12/chaos-monkey-how-netflix- uses.php • http://news.ycombinator.com/item?id=2477296 • http://stu.mp/2011/04/the-cloud-is-not-a-silver-bullet.html § Twilio, Smugmug, SimpleGeo survived it because they built their own distributed platform on top of IaaS § Enterprise customers want to consider Infrastructure like CDNs • Multi Cloud usage • Based on Open Source de facto standards, or full standards whenever that happens 7Wednesday, May 16, 12

Platforms Let the wild rumpus begin! 8Wednesday, May 16, 12

Platforms § Raise the Unit of currency to be application & services instead of infrastructure § Google App Engine, Cloud Foundry, SalesForce Heroku, Cloudbees, Amazon Elastic Beanstalk, Microsoft ex-Azure, AppFog § Single or a few languages, services § Start of Multi language Polyglot platforms § Enabler for Agile Developers -> Create Business value faster § Lack of standards: risk, vendor lock-in § Enterprise needs: • Control, customizability • Private/Hybrid Cloud • Avoid lock-in 9Wednesday, May 16, 12

Main Risk: Lock-In Welcome to the hotel california Such a lovely place Such a lovely face Plenty of room at the hotel california Any time of year, you can ﬁnd it here Last thing I remember, I was Running for the door I had to ﬁnd the passage back To the place I was before ’relax,’ said the night man, We are programmed to receive. You can checkout any time you like, But you can never leave! 10Wednesday, May 16, 12

Cloud Foundry: The Open PaaS • Open Source: Apache 2 Licensed • multi language/frameworks • multi services • multi cloud Ap ce pli vFabric rfa Postgres Private ca nte Data Services tio Clouds rI n e Se Public vid vFabric RabbitMQTM rvi Cloud ro Msg Services ce dP Micro ou Other Cloud Cl Services 11Wednesday, May 16, 12

Open Source 12Wednesday, May 16, 12

Open Source Advantage: moves faster § http://code.google.com/p/googleappengine/issues/detail?id=13 • https://github.com/cloudfoundry/vcap/pull/25 13Wednesday, May 16, 12

Open Source Advantage: diverse communities § Juju Charms to setup a multi node Cloud Foundry in 10 minutes § See Brian Thomason & Juan Negron’s UDS 2011 session http://cloud.ubuntu.com/2011/09/from-zero-to-drawbridge-via-ubuntu- server-ensemble-and-cloudfoundry-in-less-than-10-minutes/ 14Wednesday, May 16, 12

Open Source Advantage: more innovation 15Wednesday, May 16, 12

CloudFoundry.org 16Wednesday, May 16, 12

Cloud Foundry Open Source Workflow cloudfoundry.com production updated ~2x/week github github.com/cloudfoundry on +2 and ✓ change pushed to github Reviewers +1/-1 Gerrit Code Review +2/-2 Committers reviews.cloudfoundry.org test veriﬁcation score ✓ ✗ git push triggers test execution Jenkins CI ci.cloudfoundry.org 17Wednesday, May 16, 12

Cloud Foundry is 1 year old § Services • Caldecott – tunnel into your services, explore with standard client tools • PostgreSQL, RabbitMQ § Frameworks, Runtimes, and Tools • Java and Ruby Auto-Reconfiguration • Scala, node.JS 0.6.*, Erlang, JRuby, PHP, Python, .NET, Spring 3.1, Grails 2.0, Play 2.0 • Multi-Node Chef based deployment tools • Maven Plugin, Eclipse Integration • VMC manifests, Java Debugging, Rails Console • Standalone applications § Micro Cloud Foundry x 3 18Wednesday, May 16, 12

Cloud Foundry Ecosystem 19Wednesday, May 16, 12

Cloud Foundry Ecosystem 20Wednesday, May 16, 12

Register today http://cloudfoundry.com/signup Use Promo Code ChefConf To avoid approval queue waiting time 21Wednesday, May 16, 12

Logical View Browser VMC client STS plugin (user app access) Routers CloudControllers App App HealthManager Services DEA Pool Messaging 22Wednesday, May 16, 12

Cloud Foundry BOSH 23Wednesday, May 16, 12

Chef at VMware § Chef is used internally in several projects at VMware § To manage the Cloud Foundry deployment and Life Cycle, we developed a more specific system: BOSH 24Wednesday, May 16, 12

Production Grade Cloud Foundry Clusters § 500 – 5,000 VMs cloudfoundry.com § 40+ unique node types § 75+ unique software packages § 75+ unique environments § 2x/week cf.com updates § 24x7x365 non-stop operation § No-downtime deployments § Reliable, robust, repeatable deployments, updates, capacity adjustments § Small teams manage many instances production, staging, stress, qa, dev Google style problem è Google style solution 25Wednesday, May 16, 12

Cloud Foundry BOSH § Cloud Foundry BOSH is an open source tool-chain for release engineering, deployment, and lifecycle management of large scale distributed services • Prescriptive way of creating releases and managing systems and services • It is not a collection of shell scripts, not a pile of Perl § Built to deploy and manage production-class, large scale clusters • Production grade Cloud Foundry clusters: 500+ VMs, 40+ jobs, 75+ packages • Multi-node, multi-tier, complex clusters: e.g., our Gerrit/Jenkins Cluster § Built for devops usage and scale by a crack team of veterans • A project, not a product: command line interface, YAML, etc. • Continuous improvement, iterative development, rough edges github.com/cloudfoundry/bosh 26Wednesday, May 16, 12

BOSH Agenda § Background § Concepts § Demo § Workflow § Under the hood 27Wednesday, May 16, 12

Background § Built from the need to operate cloudfoundry.com • Over 40 different types of VMs • Hundreds of running instances of VMs • Pushing at least 2 changes per week § End-to-end management • Release engineering • Deployment • Lifecycle management § Generic solution • Any service • Any cloud • Any operating system 28Wednesday, May 16, 12

Concepts § Packages § Jobs § Releases • http://github.com/cloudfoundry/bosh-sample-release § Stemcells § Deployments 29Wednesday, May 16, 12

Packages § Source / blobs • Don’t commit large objects in the SCM § Dependencies • Compile time dependencies § Versioning • Auto-increment § Pre-packaging • Preparation for compilation § Packaging • Compiling the source 30Wednesday, May 16, 12

Packages (nginx) src/nginx -rw-r--r-- 1 martin staff 25490 Apr 11 19:17 headers-more-v0.15rc1.tgz -rw-r--r-- 1 martin staff 691501 Apr 11 19:17 nginx-1.0.11.tar.gz -rw-r--r-- 1 martin staff 1556329 Apr 11 19:17 pcre-8.21.tar.gz 31Wednesday, May 16, 12

Packages (nginx) packages/nginx/spec --- name: nginx files: - nginx/headers-more-v0.15rc1.tgz - nginx/nginx-1.0.11.tar.gz - nginx/pcre-8.21.tar.gz 32Wednesday, May 16, 12

Packages (nginx) packages/nginx/packaging # abort script on any command that exit with a non zero value set -e echo "Extracting pcre..." tar xzf nginx/pcre-8.21.tar.gz echo "Extracting headers-more module..." tar xzf nginx/headers-more-v0.15rc1.tgz echo "Extracting nginx..." tar xzf nginx/nginx-1.0.11.tar.gz echo "Building nginx..." cd nginx-1.0.11 ./configure --prefix=${BOSH_INSTALL_TARGET} --with-pcre=../pcre-8.21 --with-http_ssl_module --add-module=../headers-more-v0.15rc1 make make install 33Wednesday, May 16, 12

Jobs § Packages • Runtime dependencies § Templates • Converting a generalized form to an instance specific file § Versioned • Auto increments § Supervision • Through monit § Monitoring • Reported to the health monitor § Lifecycle hooks • E.g. drain workload from job 34Wednesday, May 16, 12

Jobs (nginx) jobs/nginx/spec --- name: nginx templates: nginx_ctl: bin/nginx_ctl nginx.conf.erb: config/nginx.conf mime.types: config/mime.types packages: - nginx 35Wednesday, May 16, 12

Jobs (nginx) jobs/nginx/monit check process nginx with pidfile /var/vcap/sys/run/nginx/nginx.pid start program "/var/vcap/jobs/nginx/bin/nginx_ctl start" stop program "/var/vcap/jobs/nginx/bin/nginx_ctl stop" group vcap 36Wednesday, May 16, 12

Jobs (nginx) jobs/nginx/templates/nginx.conf.erb user root root; worker_processes <%= properties.nginx.workers || 1 %>; error_log /var/vcap/sys/log/nginx/nginx.error.log; pid /var/vcap/sys/run/nginx/nginx.pid; events { worker_connections 8192; use epoll; } ... upstream wordpress { <% properties.wordpress.servers.each do |server| %> server <%= server %>:<%= properties.wordpress.port %>; <% end %> } server { listen 80; server_name "<%= properties.wordpress.servername %>"; server_name_in_redirect off; ... 37Wednesday, May 16, 12

Releases § Packages § Jobs § Versioned § Self contained § Incremental § Published » bosh create release » bosh upload release » bosh delete release » bosh releases 38Wednesday, May 16, 12

Stemcells § VM template § BOSH Agent § Versioned § Built using vmbuilder § IaaS Plugin » bosh upload stemcell » bosh delete stemcell » bosh stemcells 39Wednesday, May 16, 12

Deployments § Release § Networks § Resource pools § Compilation workers § Jobs § Update concurrency § Properties § Cloud properties 40Wednesday, May 16, 12

Deployment (wordpress) wordpress.yml --- name: bosh-wordpress release: name: wordpress version: 3 compilation: workers: 4 network: default cloud_properties: ram: 2048 disk: 8096 cpu: 2 update: canaries: 1 canary_watch_time: 60000 update_watch_time: 30000 max_in_flight: 4 max_errors: 1 41Wednesday, May 16, 12

Deployment (wordpress) wordpress.yml (continued) networks: - name: default subnets: - reserved: - 172.31.220.2 - 172.31.220.10 static: - 172.31.220.11 - 172.31.220.100 range: 172.31.220.0/22 gateway: 172.31.220.1 dns: - 172.30.22.153 cloud_properties: name: VLAN2220 - name: dmz subnets: - static: - 172.28.5.209 - 172.28.5.210 range: 172.28.5.208/28 dns: - 172.22.22.153 cloud_properties: name: VLAN3093 42Wednesday, May 16, 12

Deployment (wordpress) wordpress.yml (continued) resource_pools: - name: infrastructure network: default size: 6 stemcell: name: bosh-stemcell version: 0.4.6 cloud_properties: cpu: 1 disk: 8192 ram: 4096 43Wednesday, May 16, 12

Deployment (wordpress) wordpress.yml (continued) jobs: - name: mysql template: mysql instances: 1 resource_pool: infrastructure persistent_disk: 16384 networks: - name: default static_ips: - 172.31.220.20 - name: wordpress template: wordpress instances: 4 resource_pool: infrastructure networks: - name: default static_ips: - 172.31.220.30 - 172.31.220.33 44Wednesday, May 16, 12

Deployment (wordpress) wordpress.yml (continued) - name: nginx template: nginx instances: 1 resource_pool: infrastructure networks: - name: default default: [dns, gateway] static_ips: - 172.31.220.40 - name: dmz static_ips: - 172.28.5.209 45Wednesday, May 16, 12

Deployment (wordpress) wordpress.yml (continued) properties: wordpress: admin: wordpress@cloudfoundry.com port: 8008 servers: - 172.31.220.30 - 172.31.220.31 - 172.31.220.32 - 172.31.220.33 servername: wordpress.cloudfoundry.com db: name: wp user: wordpress pass: w0rdpr3ss auth_key: random key secure_auth_key: random key logged_in_key: random key mysql: address: 172.31.220.20 port: 3306 password: rootpass nginx: workers: 1 46Wednesday, May 16, 12

BOSH Demo 47Wednesday, May 16, 12

Dev Workflow bosh deployment bosh target dev ~/dev.yml write code run tests bosh create release git commit bosh deploy bosh upload release 48Wednesday, May 16, 12

QA Workflow bosh deployment bosh target qa ~/qa.yml bug reports to dev git pull run tests bosh create release bosh create release --final bosh deploy bosh upload release git commit 49Wednesday, May 16, 12

Ops Workflow bosh deployment bosh target prod ~/prod.yml bug reports to dev git pull run tests bosh upload release bosh deploy done! 50Wednesday, May 16, 12

Control § Repeatable deployments § Predictable rollout 51Wednesday, May 16, 12

Consistency § Dev / Staging / Production § BOSH targets 52Wednesday, May 16, 12

Components § Director / Workers § Agent § Monitor § CLI § NATS § Redis § Database § Blobstore 53Wednesday, May 16, 12

BOSH: under the hood BOSH User “BOSH is deployed by BOSH” bosh cli director healthmon cloudfoundry.com db redis nats workers blobs active jobs agent IaaS CPI stemcell disk 54Wednesday, May 16, 12

IaaS neutral by design vSphere: battle tested implementation, thousands of deployments CPI: code complete vCloud Director: “work in progress”, 2H 2012 functional status: “work in progress” Cloud Foundry BOSH Cloud Provider Interface (CPI) github.com/piston/openstack-bosh- cpi contribute: github.com/cloudfoundry/bosh 55Wednesday, May 16, 12

Cloud Provider Interface Stemcell create_stemcell(image, cloud_properties) delete_stemcell(stemcell) VM create_vm(agent_id, stemcell, resource_pool, networks, disk_locality, env) delete_vm(vm) configure_networks(vm, networks) Disk create_disk(size, vm_locality) delete_disk(disk) attach_disk(vm, disk) detach_disk(vm, disk) 56Wednesday, May 16, 12

BOSH and Chef? § How should they work together? § Talk to us on public mailing lists: • http://groups.google.com/a/cloudfoundry.org/group/bosh-users • http://groups.google.com/a/cloudfoundry.org/group/bosh-dev 57Wednesday, May 16, 12

Reference / Q&A § Mailing lists • http://groups.google.com/a/cloudfoundry.org/group/bosh-users • http://groups.google.com/a/cloudfoundry.org/group/bosh-dev § File a Bug • http://cloudfoundry.atlassian.net § Browse, Fork & Contribute • https://github.com/cloudfoundry/bosh • gem install gerrit-cli • gerrit clone ssh://reviews.cloudfoundry.org:29418/bosh 58Wednesday, May 16, 12

BOSH Community Resources @DrNic http://drnicwilliams.com/2012/04/16/creating-a-bosh-from-scratch-on-aws/ @BrianMMcClain http://www.brianmmcclain.com/2012/05/08/using-bosh-with-vsphere-part-2/ https://github.com/cloudfoundry/bosh https://groups.google.com/a/cloudfoundry.org/group/bosh-users/topics https://github.com/piston/openstack-bosh-cpi 59Wednesday, May 16, 12

Acknowledgement § Drawings from my daughters Eliette § Slides from Mark Lucovsky, Dave McCrory, Derek Collison, Martin Englund 60Wednesday, May 16, 12

http://www.scoop.it	55
https://twitter.com	14
https://si0.twimg.com	2
http://us-w1.rockmelt.com	2
https://twimg0-a.akamaihd.net	2
http://twitter.com	1

Cloud Foundry BOSH Where the Platform rubber meets the Infrastructure road - ChefConf

by Patrick Chanezon

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 76

Statistics

Cloud Foundry BOSH Where the Platform rubber meets the Infrastructure road - ChefConf Presentation Transcript