This document discusses experiences migrating from a rented VMware environment to a self-hosted OpenStack cloud and running Cloud Foundry on OpenStack. Some key points discussed include: - Upgrades to OpenStack before the Grizzly release required a lot of manual work and could result in a full week of downtime for instances to be offline. - The upcoming Havana to Icehouse upgrade aims for less than 30 minutes of downtime using Chef to automate configuration changes and testing upgrades on a separate OpenStack staging system. - Random kernel panics, hardware outages, and other factors can kill VMs, so availability zones and aggregates can be used to spread VMs across disjunct networks/racks.

1. CloudFoundryonOpenStack
AnExperienceReport

2. Introduction

3. about.me/fischerjulian

6. The anynines Stack

7. Hardware
OpenStack
Cloud Foundry
VMware

8. We migrated from a
Rented VMware to a
self-hosted OpenStack.

9. For more details on this:
http://rh.gd/a9vmw2sos

10. Things we had to
think about

11. OpenStack Upgrades

12. Before Grizzly
OpenStack was
not ready for production

13. • The upgrade process included a lot
manual work
• No script driven upgrades
• Manual DB schema migrations
• Manual configuration file changes, etc.

14. „I scheduled a week of
total downtime with all instances
offline.“- jon@jonproulx , http://rh.gd/1sNhiiz

15. Upcoming Upgrade
Havanna > Icehouse

16. • Chef is used to roll-out Icehouse (incl.
configuration changes)
• The upgrade is well tested on a separate
multi-server OpenStack staging system

17. Goal:
<30 min downtime.

18. Let’s cross fingers :)

19. Looking forward to rolling
Upgrades with OpenStack
Icehouse
http://rh.gd/1ymhViL

20. • No need to shutdown VMs during
upgrade
• No downtime of the entire cloud

21. VM availability

22. What killes VMs?

23. • Random kernel panics (kernel bug)
http://rh.gd/1oBUeCc
• Hardware outages (hw & power failures)
• …

24. Availability Zones

25. • Build disjunct networks, racks, etc.
• Each disjunct zone = availability zone
• Tell OpenStack about availability zones
• On provision you can choose the AZ
• Build Bosh releases accordingly

26. Aggregates

27. • Similar to AZ
• Not about failover
• Select hosts with certain attributes
• E.g. SSD-aggregate
• On provision choose host with SSD disks

28. Load Balancing

29. • Not inherently clustered
• LBaaS failover can be realized using
• Pacemaker/corosync
• GlusterFS (share LB configuration)

30. VM Failover Strategies

31. Resurrect

32. • Monitor VM
• Re-Build VMs automatically
• e.g. using Cloud Foundry Bosh
• + Easy
• - Takes long (minutes not seconds)
• - Open Stack doesn’t release persistent
disks automatically

33. Failover to Standby VM

34. • Provide stand-by VM
• Monitor VM and perform failover
• e.g. using Pacemaker
• + Fast failover (seconds)
• - Pacemaker is not easy to use
• - Increased resource usage by stdby
VM(s)

35. IP Failover

36. Three ways to failover IPs

37. Load Balancer

38. • + Fast
• + Easy (use lb weights)
• - LB becomes a bottleneck
• When OS supports HA Proxy (L3) this
drawback can be eliminated

39. Floating-IPs

40. • + Easy
• + Fast
• - Only for public networking

41. NIC Re-attachment

42. • + No network bottleneck
• + No dependencies to other services
• - Slightly higher failover time (several
seconds)

43. Implications for
Cloud Foundry

44. Accept that VMs are
ephemeral

45. Distribute CF components
across OS availability zones

46. • 2 * UAA
• 2 * CC
• 2 * n * DEAs
• 2 * Health Manager
• …

47. UAA & CC DB
=
SPOF

48. HA Postgres

49. • UAA and Cloud Controller database
• Single point of failure for Cloud Foundry

50. • Postgres not inherently clusterable >
failover with standby vm
• Master/slave replication
• Pacemaker/corosync
• IP-Failover using NIC-reattachment

51. That’s half way towards a
PostgreSQL CF Service

52. • Add a V2 Service Broker
• Add a provisioning logic
• Provision 2-node db cluster on
cf create service postgres medium-cluster

53. CF Service Design

54. • Use clusterable services if possible
• Implement automatic failover if not
• Autoprovisioning using Bosh
• Organize self-healing
• (Semi-)Automatic recovery from
degraded mode

55. Summary

56. • VMware’s high availability options are
nice
• OpenStack helped us to save 50% costs
• OS is stable enough to run Cloud
Foundry on top
• OS hardening is required and feasible

57. Open Source OpenStack
and Open Source Cloud
Foundry are SME’s best
friends!

58. Questions?

59. Thank you!

60. Preparing for
disaster recovery

61. • Cinder Volume Snapshots

62. OpenStack Backups

63. OpenStack Swift

64. • Open source Amazon S3 replacement
• Object store with RESTful interface
• Scales horizontally to petabyte
dimensions
• Fully redundant, highly available
• CF service > App Asset Storage

65. Code
require "fileutils"
require "find"
require "fog"
!
class Blobstore
def initialize(connection_config, directory_key, cdn=nil, root_dir=nil)
@root_dir = root_dir
@connection_config = connection_config
@directory_key = directory_key
@cdn = cdn
end
!
def local?
@connection_config[:provider].downcase == "local"
end
!
def exists?(key)
!file(key).nil?
end