It’s no news that containers represent a portable unit of deployment, and OpenStack has proven an ideal environment for running container workloads. However, where it usually becomes more complex is that many times an application is often built out of multiple containers, as well as hybrid environments - diverse clouds, bare metal and even non-virtualized infrastructure. What’s more, setting up a cluster of container images can be fairly cumbersome because you need to make one container aware of another and expose intimate details that are required for them to communicate which is not trivial especially if they’re not on the same host. These scenarios have instigated the demand for some kind of orchestrator. The list of container orchestrators is growing fairly fast. This session will compare the different orchestration projects out there - from Heat to Kubernetes to Mesos & Cloudify - and help you choose the right tool for the job.

1. Container Orchestration
Roundup
Trammell Scruggs,
Arthur Berezin,
GigaSpaces

2. Agenda
◇ Orchestration 101
◇ Method of comparison
◇ Tools overview

3. Orchestration 101

4. Orchestration 101
◇ Common Characteristics
￭ Use DSL to define “blueprint”
￭ Execute a process based on input from the blueprint
￭ Pass context information between the deployed entities
◇ Different assumptions lead to different approaches
￭ Application Architecture
￭ Infrastructure
￭ Scope of automation

5. Method of Comparison
◇ Same application requirements
◇ “Production grade” deployment
◇ Broken into three main groups
￭ Pure Play – Cloudify/TOSCA, Terraform
￭ Container Centric – Kubernetes
￭ Infrastructure Centric - OpenStack Heat

6. ◇ PaaS: CloudFoundry, OpenShift, DEIS
◇ Config Management: Chef, Puppet, Ansible, Salt
◇ Cloud Application Management: Apache Brooklyn, Juju
◇ Cloud management platforms: RH CloudForms, Scalr
◇ Cluster schedules: Mesos, Fleet, Yarn, Nomad
This is by no means an
exhaustive comparison!
A Lot of Relevant Tools
Which We Won’t Cover Today :(

7. The Test Application
Test
Application

8. ◇ Manage dependencies
◇ Reproducible
◇ Cloneable
◇ Recoverable
◇ Updateable
Common Requirements

9. ◇ Setup monitoring and log collection
◇ Manual/Auto healing
◇ Manual/Auto scaling
◇ “Day 2”
￭ Backup and restore
￭ Update code
￭ Infrastructure upgrades and patches
Beyond Just Setup

10. “Pure-Play” Orchestration

11. Terraform by Hashicorp

12. Introduction to Terraform
◇ Part of the Hashicorp “ecosystem”
◇ Command line tool
◇ Simple (in a good way) configuration
◇ Extendable using plugins

13. Solution Overview
◇ Single top level configuration file
◇ Create all infrastructure install and
configure application with scripts.

14. Sample Configuration
resource "openstack_compute_instance_v2" "mongod_host" {
count = "3"
region = ""
name = "mongod_host"
image_name = "${var.image_name}"
flavor_name = "${var.flavor_name}"
key_pair = "tf-keypair-1"
security_groups = ["mongo_security_group"]
network {
uuid = "${openstack_networking_network_v2.tf_network.id}"
}
...
provisioner "remote-exec" {
scripts = [
"scripts/install_mongo.sh"
"start_mongod.sh"
]
}
}

15. Deployment Management
◇ Option 1: Static Allocation/Provisioning in
the configuration file.
◇ Option 2: Third-party Tool
￭ Consul/Surf
￭ Other, e.g. AWS Autoscale

16. Cons:
◇ Configurations are not
portable across cloud
providers
◇ Outsources advanced
configuration management
to other tools
◇ Context awareness is not
built in
Terraform: Pros and Cons
Pros:
◇ Simple & elegant
◇ Solid Openstack
support
◇ IaaS & Framework
neutral
◇ Display Plan before
deployment
◇ Idempotency
◇ Hashicorp stack

17. TOSCA/Cloudify

18. Topology
Nodes,
relationships,
requirements &
capabilities
Workflows
Lifecycle
operations
implementation
(install, scale…)
Policy
Events,
conditions,
action
mapping
Topology Orchestration Spec for Cloud Applications
What is TOSCA?

19. What is TOSCA?
◇ Standard for Cloud Orchestration
◇ Describes how to map application
requirements to infrastructure capabilities

20. Application
Blueprint
(TOSCA)
IaaS
Plugins
Container
Plugins
Conf Mgmt
Plugins
● Provision
● Configure
● Monitor
● Manage
Monitoring &
Alarming
Cloudify: an Open Source
Implementation of TOSCA

21. Solution Overview
◇ Single blueprint file with peripheral
scripts and plugins
◇ Create all infrastructure install and
configure application with scripts
◇ Describe future application management
and workflows
◇ Relationships describe which code to
execute when one node event affects
another node.

22. nodecellar_container:
type: nodecellar.nodes.NodecellarContainer
properties:
Name: nodecellar
image:
Repository: nodecellar
relationships:
- type: node_contained_in_nodejs
target: nodejs_host
Sample Configuration

23. nodejs_host:
type: cloudify.openstack.nodes.Server
interfaces:
cloudify.interfaces.monitoring:
start:
implementation: diamond.diamond_agent.tasks.add_collectors
inputs:
collectors_config:
default:
CPUCollector: {}
MemoryCollector: {}
LoadAverageCollector: {}
DiskUsageCollector:
config:
devices: x?vd[a-z]+[0-9]*$
NetworkCollector: {}
Sample Configuration

24. Deployment Management
◇ Manager is aware of node instances’ state
◇ logging and monitoring
◇ Policy engine handles metric streams
◇ Manager auto-executes workflows in
response to lifecycle requirements

25. ◇ Built in workflows
￭ Install
￭ Uninstall
￭ Heal
￭ Scale
￭ Execute_operation
￭ (Single common ‘cfy install’ in 3.4 )
◇ Dependency awareness through graph
navigation
◇ Remote/Local execution
Handling Post Deployment
through Workflow & Policies

26. Cons:
◇ Complex
◇ Delay in full support of
TOSCA spec
◇ Not enough
implementations - user
needs to write most of it
TOSCA/Cloudify: Pros and Cons
*Implementation specific
Pros:
◇ Infrastructure &
Framework neutrality
◇ Complete Life Cycle
Management
◇ Handles Infrastructure
& Software
◇ Post deployment
handling*
￭ Monitoring
￭ Logging
◇ Can be tied to any
peripheral system

27. Container Orchestration
Kubernetes

28. Container deployment management
◇ Pods: a group of containers usually run together
◇ Replication controller: ensures state of pod
◇ Networking: pods are networked together
◇ Service: Load balanced endpoint for a set of pods
Quick Intro to K8s

29. K8s Architecture

30. Solution Overview
◇ Single configuration file
◇ Uses pools of infrastructure resources
◇ Describe future application management
and workflows
◇ Relationships are maintained through
Pods

31. apiVersion: v1
kind: Service
metadata:
name: nodecellar-service
spec:
selector:
app: nodecellar
ports:
- protocol: TCP
port: 3000
targetPort: 3000
nodePort: 30000
type: NodePort
Sample Configuration
apiVersion: v1
kind: ReplicationController
metadata:
name: nodecellar
spec:
replicas: 2
selector:
app: nodecellar
template:
metadata:
name: nodecellar
labels:
app: nodecellar
spec:
containers:
- name: nodecellar
Image: nodecellar
workingDir: /
command:
["bash","start.sh"]
ports:
- containerPort: 3000
hostIP: 0.0.0.0

32. Cons:
◇ Not a lot of infrastructure
focus
◇ Limited Openstack support
(mostly Rackspace)
◇ No topology awareness
◇ Complex setup
Kubernetes: Pros and Cons
Pros:
◇ (almost) zero
configuration autoheal
◇ Out of the box load
balancer
◇ Simple scaling
◇ Advanced behaviors
are pretty simple to
implement

33. Infrastructure Centric

34. What Is Heat?
Provides a
mechanism for
orchestrating
OpenStack resources
through the use of
modular templates

35. Heat Architecture
OpenStack
APIs

36. Solution Overview
Output:
Replica set node
hosts
ssh-key, private ip
Input:
# of replicas per shard
Input:
# of nodeJS instances
# MongoConfig hosts
# Mongo shards hosts
Output:
mongos node hosts
App EndPoint = Load-
Balancer IP/path
Output:
mogocfg node hosts
Input:
# of config instances

37. resources:
secgroup:
type: OS::Neutron::SecurityGroup
properties:
name:
str_replace:
template: mongodb-$stackstr-secgroup
params:
$stackstr:
get_attr:
- stack-string
- value
rules:
- protocol: icmp
- protocol: tcp
port_range_min: 22
port_range_max: 22
- protocol: tcp
port_range_min: 27017
port_range_max: 27019
Infrastructure setup

38. mongo_host:
type: OS::Nova::Server
properties:
name:
str_replace:
template: $stackprefix-$stackstr
params:
$stackprefix:
get_param: stack-prefix
$stackstr:
get_attr:
- stack-string
- value
Image:
get_param: image
flavor:
get_param: flavor
security_groups:
- get_param: security_group
Create Compute Instances

39. mongodb_peer_servers:
type: "OS::Heat::ResourceGroup"
properties:
count: { get_param: peer_server_count }
resource_def:
type: { get_param: child_template }
properties:
server_hostname:
str_replace:
template: '%name%-0%index%'
params:
'%name%': { get_param: server_hostname }
image: { get_param: image }
flavor: { get_param: flavor }
ssh_key: { get_resource: ssh_key }
ssh_private_key: { get_attr: [ssh_key, private_key] }
kitchen: { get_param: kitchen }
chef_version: { get_param: chef_version }
Create MongoDB Replica Servers

40. server_setup:
type: "OS::Heat::ChefSolo"
depends_on:
- mongodb_peer_servers
properties:
username: root
private_key: { get_attr: [ssh_key, private_key] }
host: { get_attr: [mongodb_peer_servers, accessIPv4, 0] }
kitchen: { get_param: kitchen }
chef_version: { get_param: chef_version }
node:
mongodb:
ruby_gems:
mongo: '1.12.0'
bson_ext: '1.12.0'
bind_ip: { get_attr: [mongodb_peer_servers, privateIPv4, 0] }
use_fqdn: false
replicaset_members: { get_attr: [mongodb_peer_servers, privateIPv4] }
config:
replset: myreplset
run_list: [ "recipe[config_replset]" ]
Configure the Replica Servers

41. nodestack_chef_run:
type: 'OS::Heat::ChefSolo'
depends_on: nodestack_node
properties:
...
node:
nodejs_app:
...
deployment:
id: { get_param: stack_id }
app_id: nodejs
run_list: ["recipe[apt]",
"recipe[nodejs]",
"recipe[ssh_known_hosts]",
"recipe[nodejs_app]"]
data_bags:
nodejs:
id: { get_param: stack_id }
nodejs_app:
password: { get_attr: [nodejs_user_password, value] }
deploy_key: { get_param: deploy_key }
database_url:
str_replace:
template: 'mongodb://%dbuser%:%dbpasswd%@%dbhostname%'
params:
'%dbuser%': { get_param: database_username }
'%dbpasswd%': { get_param: database_user_password }
'%dbhostname%': { get_param: db_server_ip }
Create NodeJS Container

42. Cons:
◇ OpenStack Only
◇ Software configuration is
limited
◇ Lack of built-in workflow
abstraction
◇ Post deployment
orchestration is limited
￭ Requires integration
with other tools/
projects
Heat: Pros and Cons
Pros:
◇ Native To OpenStack
◇ Built-in mapping of
most OpenStack
infrastructure resource
types

43. So, what’s the Right
Tool for me?
◇ Are you hell bent on containers? OpenStack?
◇ Do you have legacy workloads?
◇ Do you consider infrastructure resources as part of the
process?
◇ Do you have a heterogenous environment?
◇ Do you want the same tool to also handle post
deployment?

44. The Tools Presented Are
Not Necessarily Mutually
Exclusive!

45. Final Words

46. The Only Constant is Change

47. Thank You
Questions?
Find us at:
◇ Twitter @CloudifySource
◇ email info@getcloudify.org