Lessions from building a high available cloud foudry on top of open stack
•
1 like•324 views
This document discusses lessons learned from building a highly available cloud foundry deployment on OpenStack. It covers integrating OpenStack, deploying Cloud Foundry using BOSH, using the OpenStack CPI, validating OpenStack, networking with Neutron, using Swift for blob storage, encrypting database data, load balancing with Gorouter, and backing up the Cloud Foundry platform and applications.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Lessions from building a high available cloud foudry on top of open stack
Similar to Lessions from building a high available cloud foudry on top of open stack (20)
Recently uploaded
Recently uploaded (20)
Lessions from building a high available cloud foudry on top of open stack
- 1. Lessions from Building a High Available Cloud Foudry on top of OpenStack 1
- 2. Hello World! Jiang Yi Tao IBM CDL Bluemix IaaS PaaS 2
- 3. Agenda • Open Stack • Nova Neutron Cinder Swift • Cloud Foundry • LB Gorouter • Database • Backup 3
- 4. Releases • CF 233 • OpenStack Kilo 4
- 5. OpenStack 5
- 7. • Image • Router • network, subnet • Flavor • Keypair • Security Group OpenStack Create VM 7
- 8. How to Deploy CF 1.Target a bosh director using cli 2.Upload a stemcell 3.Upload a release 4.Gererate a manifest 5.bosh deploy 5.1.prepare deployment 5.2.complie the packages 5.3.create and bind vm 5.4.pull in job configurations 5.5.create needed job instances 8
- 9. OpenStack-CPI • OpenStack CPI is an implementation of the BOSH CPI • Leverages the Fog Ruby gem for OpenStack http://fog.io 9
- 10. OpenStack Ready? • Fog Version? • bosh openstack cpi • Validate OpenStack • https://docs.cloudfoundry.org/deploying/openstack/ validate_openstack.html 10
- 11. OpenStack Ready?(cont.) gem install XXX -v XXX require 'fog' EXCON_DEBUG=true nova=Fog::Compute.new({:provider => 'openstack', :openstack_tenant => 'jiangytcn', :openstack_api_key => 'XXXX', :openstack_username => 'jiangytcn', :openstack_auth_url => 'https://XXX.cn.ibm.com:35357/v3/auth/ tokens', :openstack_domain_name => 'default', :connection_options => { :ssl_verify_peer => false}}) nova.list_servers nova.servers.create(name = 'trusty', image_ref = image, flavor_ref = flavor, key_name = my_key) Similar for Volumes, Networks, Images, etc. 11
- 12. Nova • QEMU KVM • Over Commit • Flavor • ephemeral >= 2* ram If has ephemeral • Security Groups • Only open required ports, remote CIDR/SG • VM runs OK? • ping 127.0.0.1 • (Compute Node) watch 'tail /sys/kernel/mm/ksm/*' 12
- 13. Resource Pool • Scheduler hints • AZ Affinity-Group Anti- affinity-group • No SPOF • Need more physical resource 13
- 14. Neutron • Overlap • Vlan gre vxlan • Isolated • IP translation, controlled • Flat • Flat-dhcp • Shared • Directly to switch ✤ MTU Selection PMTU ping -D -g <minsize> -G <maxsize> -h <sizeincr> <host> ✤ JUMBO FRAMES ✤ meta-data/user-data 14
- 15. Cinder • Backend • LVM CEPH ... • Disk Readonly * 15
- 16. Swift - BlobStore • user-uploaded applications, buildpacks, droplets, and application resources • Tempurl • X-Account-Meta-Temp-URL-Key • runner download packages using tempurl • Ratelimit(Optional. If no account or container limits are provided there will be no rate limiting) • NTP 16
- 17. Keystone • Tenant/Project with Large Quotas • Day one & Day two 17
- 18. Deployment • keystone • v2 or v3 • network type • manual VS dynamic VS static 18
- 21. Grouter & Load Balancer • Domains • system, apps • Gorouter only support single certs • TLS Pass-Through • Performant and a single TLS connection • TLS Termination at Load Balancer • Less performant option • Multiple certificates to be used • Internal • Haproxy • External LB • Third party • LBaaS 21
- 22. BlobStore • NFS • Webdav • Swift 22
- 23. Cloud Controller • resources clean up app_events: cutoff_age_in_days: 36 app_usage_events: cutoff_age_in_days: 36 audit_events: cutoff_age_in_days: 36 failed_jobs: cutoff_age_in_days: 7 23
- 24. Database • Cluster • Postgresql • Mysql • Backup • Transaction Logs • Full backup 24
- 26. DB Encrypt/Decrypt • Cloud Controller • db_encryption_key • row level encrypt, salt 26
- 27. def encrypt(input, salt) return nil unless input Base64.strict_encode64(run_cipher(make_cipher.encrypt, input, salt)) end def decrypt(encrypted_input, salt) return nil unless encrypted_input run_cipher(make_cipher.decrypt, Base64.decode64(encrypted_input), salt) end def make_cipher OpenSSL::Cipher::Cipher.new(ALGORITHM) end def run_cipher(cipher, input, salt) cipher.pkcs5_keyivgen(db_encryption_key, salt) cipher.update(input).tap { |result| result << cipher.final } end 27
- 29. Buildpacks • application runtime • installed buildpack • system build pack will be override every update 29
- 30. Backing up Cloud Foundry • Platform • BOSH DB • configuration files • CCDB UAADB • Blobstore • Credentials • User • apps • service instance 30
- 31. Bosh DB • DNS • for internal use • VM and Volume mapping 31
- 32. Configuration • deployment manifest bosh download manifest <DEPLOYMENT> cf.yml kept it • key pairs, certs 32
- 33. 33 You can find me at: • @jiangytcn • jiangyt.cn@gmail.com Validate OpenStack core component vm metadata service Cloud Foundry Network Type Data encrypt/decrypt Backup DB services manifest