This is a talk from the Austin OpenStack summit. It demonstrates how a resilient, elastic and load-balanced cluster can be deployed using senlin, heat, ceilometer, lbaas v2, nova.

1. Deploy an Elastic, Resilient,
Load-Balanced Cluster in 5
Minutes with Senlin
Mark Voelker (VMware)
Qiming Teng (IBM)
Xinhui Li (VMware)
1

2.  Requirements for Resource Pool Management
 Senlin Clustering Service Overview
 An All-in-one Sample
2
Agenda

3. Managing Resource Pools
3

4.  Manageability
 cluster membership
 Elasticity
 key feature of a cloud, pay-as-you-go
 Load-Balanced
 distribute loads evenly on all members
 Flexibility
 customizable to meet the needs of real-life use cases
 Extensibility
 easy to add resource types and policies
4
Generic Requirements

5.  Heat: The Orchestrator
 AutoScalingGroup +
Scaling Policy + Alarm
 Heat AutoScaling modeled after AWS
AutoScaling
 It may suit some limited use cases, but not
all scenarios considering that OpenStack
aims to build a platform for both private and
public clouds
 But, Heat's Mission...
 Orchestrate composite cloud apps using a declarative template format
through an OpenStack-native REST API
 It is not about any new service, not auto-scaling, not HA, not load-balance ...
5
What we have on OpenStack
Heat as a Compiler
YAML
Template
Heat
Nova
Cinder
Neutron
Keystone
• CREATE
• GET
• UPDATE
• DELETE

6.  Cross Availability Zone Placement
 Cross Region Placement
 Anti-Affinity Placement
 Choose a Specific Node to Delete when Scaling-In
 Trigger Autoscaling Using Application-Level Metrics
 Manual Scaling
 Auto Recover Node when Failures Detected
 Migrate Nodes from a Stand-By Cluster for Rapid Provisioning
 Soft Scaling
 ...
6
Some Specific Requirements

7. Senlin Overview
7

8. 8
What Do We Really Need?
Scalable
Load-Balanced
Highly-Available
Manageable
......
of any (OpenStack) objects
- What is missing from
OpenStack?
A Clustering Service
- Auto-scaling?
Just one of the usage
scenario of a cluster.
- Auto-Healing (HA)?
Just another usage
scenario.
- We can address the
concerns by making
policies orthogonal

9. Senlin Architecture
Senlin Engine
Senlin
API
Senlin
Database
Senlin
Client
REST RPC
Profiles
Policies
9

10. 10
Senlin Features
 Profiles: A specification for the objects to be managed
 Policies: Rules to be checked/enforced before/after actions are performed
Docker
Senlin
Nova
Kubernetes
Heat
Ironic BareMetal
VMs
Stacks
Containers
placement
deletion
scaling
health
load-balance
batching
Policies as Plugins Profiles as Plugins Cluster/Nodes Managed

11. 11
Senlin Server Architecture
openstacksdk
identity
compute
orchestration
network
...
engineengine lock
scheduler
actions
nodecluster
service
registry
receiverparser
drivers
openstack
dummy
(others)
dbapi
rpc client
policies
placement
deletion
scaling
health
load-balance
batching
receiver
webhoook
MsgQueue
extension points
for external
monitoring
services
extension points
facilitating a
smarter cluster
management
extension points to talk to different
endpoints for object CRUD operations
extension points for interfacing
with different services or clouds
profiles
os.heat.stack
(others)
os.nova.server
senlin-api
WSGI
middleware
apiv1

12. 12
Senlin Operations (Actions)
 Cluster
• CREATE
• DELETE
• UPDATE
• LIST
• SHOW
• ADD_NODES
• DEL_NODES
• SCALE_OUT
• SCALE_IN
• POLICY_ATTACH
• POLICY_DETACH
• POLICY_UPDATE
 Node
• CREATE
• DELETE
• UPDATE
• LIST
• SHOW
 Policy
• CREATE
• UPDATE
• DELETE
• LIST
• SHOW
 Profile
• CREATE
• UPDATE
• DELETE
• LIST
• SHOW
 Action
• LIST
• SHOW
 Event
• LIST
• SHOW
 Receiver
• CREATE
• DELETE
• LIST
• SHOW

13. 13
profile/cluster/node operations
$ senlin profile-create -s <spec> <name>
$ senlin profile-list
$ senlin profile-show <profile>
$ senlin profile-delete <profile>
$ senlin profile-update -s <spec> <profile>
$ senlin cluster-create -p <profile> -n <size> <name>
$ senlin cluster-delete <cluster>
$ senlin cluster-list
$ senlin cluster-show <cluster>
$ senlin node-create -p <profile> -c <cluster> <name>
$ senlin node-delete <node>
$ senlin node-show <node>
$ senlin node-join -c <cluster> <node>
$ senlin node-leave <node>
$ senlin cluster-node-add -n <nodes> <cluster>
$ senlin cluster-node-del -n <nodes> <cluster>
$ senlin cluster-node-list <cluster>

14. 14
policies operations
$ senlin policy-create -s <spec> <name>
$ senlin policy-list
$ senlin policy-show <policy>
$ senlin policy-delete <policy>
$ senlin policy-update -n <name> ... <policy>
$ senlin cluster-policy-attach <cluster> <policy>
$ senlin cluster-policy-detach <cluster> <policy>
$ senlin cluster-policy-list ... <cluster>
$ senlin cluster-policy-show <cluster> <policy>
$ senlin cluster-policy-enable <cluster> <policy>
$ senlin cluster-policy-disable <cluster> <policy>
$ senlin cluster-policy-update ... <cluster> <policy>

15. 15
Other Useful Operations
$ senlin cluster-scale-in [-c <count>] <cluster>
$ senlin cluster-scale-out [-c <count>] <cluster>
$ senlin cluster-resize <options> <cluster>
$ senlin profile-type-list
$ senlin profile-type-show <type>
$ senlin policy-type-list
$ senlin policy-type-show <type>
$ senlin action-list
$ senlin action-show <action-id>
$ senlin event-list
$ senlin event-show <type>

16. 16
senlin cluster-resize
usage: senlin cluster-resize [-c <CAPACITY>] [-a <ADJUSTMENT>] [-p <PERCENTAGE>]
[-t <MIN_STEP>] [-s] [-n MIN] [-m MAX] <CLUSTER>
Positional arguments:
<CLUSTER> Name or ID of cluster to operate on.
Optional arguments:
-c <CAPACITY>, --capacity <CAPACITY>
The desired number of nodes of the cluster.
-a <ADJUSTMENT>, --adjustment <ADJUSTMENT>
A positive integer meaning the number of nodes to add,
or a negative integer indicating the number of nodes
to remove.
-p <PERCENTAGE>, --percentage <PERCENTAGE>
A value that is interpreted as the percentage of size
adjustment. This value can be positive or negative.
-t <MIN_STEP>, --min-step <MIN_STEP>
An integer specifying the number of nodes for
adjustment when <PERCENTAGE> is specified.
-s, --strict A boolean specifying whether the resize should be
performed on a best-effort basis when the new capacity
may go beyond size constraints.
-n MIN, --min-size MIN New lower bound of cluster size.
-m MAX, --max-size MAX New upper bound of cluster size. A value of -1
indicates no upper limit on cluster size.
EXACT CAPACITY?
CAPACITY BY PERCENTAGE?
CHANGE IN CAPACITY?
MINIMUM STEP?
STRICT/BEST-EFFORT?
ADJUST MIN_SIZE?
ADJUST MAX_SIZE?

17. ALL-In-ONE:
AS (AutoScaling) + HA (High-Availability) + LB (Load-Balancing)
17

18. 18
Architecture Overview
Heat Template
Cluster
Scaling
Policy LB Policy
Health
Policy
LBaaS
Alarm
Receivers
Compute
SCALE IN
SCALE OUT
RECOVER
Profile

19. 19
Profile and Cluster
profile:
type: OS::Senlin::Profile
properties:
type: os.nova.server-1.0
properties:
flavor: {get_param: flavor}
image: {get_param: image}
key_name: {get_param: key_name}
networks:
- network: {get_param: network}
security_groups:
- {get_resource: security_group}
cluster:
type: OS::Senlin::Cluster
properties:
desired_capacity: 2
min_size: 2
profile: {get_resource: profile}

20. 20
LB Policy
lb_policy:
type: OS::Senlin::Policy
properties:
type: senlin.policy.loadbalance-1.0
bindings: [cluster: {get_resource: cluster}]
properties:
pool:
protocol: HTTP
protocol_port: 80
subnet: {get_param: pool_subnet}
vip:
subnet: {get_param: vip_subnet}
protocol: HTTP
protocol_port: 80
health_monitor:
type: HTTP
delay: 10
timeout: 5
max_retries: 4

21. 21
Scaling Policies
scale_in_policy:
type: OS::Senlin::Policy
properties:
type: senlin.policy.scaling-1.0
bindings:
- cluster: {get_resource: cluster}
properties:
event: CLUSTER_SCALE_IN
adjustment:
type: CHANGE_IN_CAPACITY
number: 1
scale_out_policy:
type: OS::Senlin::Policy
properties:
type: senlin.policy.scaling-1.0
bindings:
- cluster: {get_resource: cluster}
properties:
event: CLUSTER_SCALE_OUT
adjustment:
type: CHANGE_IN_CAPACITY
number: 1

22. 22
Health Policy
health_policy:
type: OS::Senlin::Policy
properties:
type: senlin.policy.health-1.0
bindings:
- cluster: {get_resource: cluster}
properties:
detection:
type: NODE_STATUS_POLLING
options:
interval: 60
recovery:
actions:
- RECREATE

23. 23
Receivers and Alarms
receiver_scale_in:
type: OS::Senlin::Receiver
properties:
cluster: {get_resource: cluster}
action: CLUSTER_SCALE_IN
type: webhook
scale_in_alarm:
type: OS::Ceilometer::Alarm
properties:
meter_name:network.services.lb.incoming.bytes.rate
statistic: avg
period: 180
evaluation_periods: 1
threshold: 12000
repeat_actions: True
alarm_actions:
- get_attr:
- receiver_scale_in
- channel
- alarm_url
comparison_operator: le

24. 24
Receivers and Alarms
receiver_scale_out:
type: OS::Senlin::Receiver
properties:
cluster: {get_resource: cluster}
action: CLUSTER_SCALE_OUT
type: webhook
scale_out_alarm:
type: OS::Ceilometer::Alarm
properties:
meter_name: network.services.lb.incoming.bytes.rate
statistic: avg
period: 60
evaluation_periods: 1
threshold: 28000
repeat_actions: True
alarm_actions:
- get_attr:
- receiver_scale_out
- channel
- alarm_url
comparison_operator: ge

25. Short Video, For Free, :-)
25

26. Questions?
26

27. 27
Blueprints on Reworking Heat Autoscaling
BP Priori
ty
Description
autoscaling-api-
resources
high Heat resources invoking AS APIs
as-api-group-
resource
high ScalingGroup resource wrapping AS API's
group functionality
as-api-policy-
resource
high ScalingPolicy resource wrapping AS API's policy
functionality
as-api-
webhook-
resource
high Webhook resource wrapping AS API's execution
of webhooks
autoscaling-api-
client
high A python client for Heat to interact with AS API
autoscaling-api - A separate service for the implementation of
autoscaling w/ Heat
as-engine - A separate engine/service for autoscaling
support AS API
as-engine-db - A DB dedicated to autoscaling, using schema
created in as-lib-db
as-lib - A separate module to be used by the AS service
as-lib-db - A DB for autoscaling bookkeeping

28. 28
Relation To Other Projects
 Senlin provides the array data type for cloud programming
Senlin
Ceilometer
Heat
Nova Cinder Neutron Swift Keystone
Horizon
Primitive Data Types
Complex Data Types
struct person {
int age;
char name[0];
}
person team[10]; // Senlin cluster of Heat stacks
// Senlin cluster of nova servers
// Heat stack containing senlin clusters
Mistral MagnumSaharaMonasca Tacker Zaqar