This document describes an active/active high availability solution for central agents in OpenStack ceilometer. It discusses using taskflow to manage the discovery and polling tasks performed by central agents to provide high availability. Discovery agents monitor the sources of telemetry data and polling agents collect the data. Taskflow would store the tasks in a job table coordinated by Zookeeper to ensure tasks are rescheduled if an agent fails. This provides a more robust solution than a simple homebrew approach using RabbitMQ.

1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Active/Active HA for
Central Agent
Fabio Giannetti
July 2014

2. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2
The Source (No Discovery)
- name: cpu_source
interval: 400
meters:
- "cpu"
sinks:
- cpu_sink
Compute (Nova)
Name Type Unit Resource
instance g Instanc
e
inst ID
instance:<type> g Instanc
e
inst ID
memory g MB inst ID
memory.usage g MB inst ID
cpu c Ns inst ID
cpu_util g % inst ID
…
ceilometer.poll.compute =
….
cpu = ceilometer.compute.pollsters.cpu:CPUPollster

3. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
The Source (Discovery)
- name: lb_pool_source
interval: 600
meters:
- "network.services.lb.pool"
discovery:
- "lb_pools"
sinks:
- meter_sink
Name Type Unit Resource
network.services.lb.pool g pool pool ID
network.services.lb.vip g vip vip ID
network.services.lb.member g member member ID
…
Load Balancer as a Service (LBaaS)
lb_pools =
ceilometer.network.services.discovery:LBPoolsDiscovery
network.services.lb.pool =
ceilometer.network.services.lbaas:LBPoolPollster

4. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
Step 1: Normalization of Sources
- name: lb_pool_source
interval: 600
meters:
- "network.services.lb.pool"
discovery:
- "lb_pools"
sinks:
- meter_sink
source interval pollster discovery
lb_pool_source 600 ceilometer.network.services.lbaas:LBPoolPollster ceilometer.network.services.discovery:LBPoo
lsDiscovery
cpu_source 400 ceilometer.compute.pollsters.cpu:CPUPollster ceilometer.compute.discovery:InstanceDi
scovery
- name: cpu_source
interval: 400
meters:
- "cpu"
sinks:
- cpu_sink

5. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
Step 2a: Sources Discovery
source interval pollster discovery agentID agentPID agentIP timestamp Status
lb_pool_source 600 LBPoolPollster LBPoolsDiscovery 0002 25425 aa.aa.aa.aa T0 Ready
cpu_source 400 CPUPollster InstanceDiscovery 000N 76746 bb.bb.bb.bb T0 Ready
Discovery
Agent 1
Discovery
Agent 2
Discovery
Agent 3
Discovery
Agent N
…

6. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
Step 2b: Sources Discovery
source interval pollster discovery agentID agentPID agentIP timestamp status
lb_pool_source 600 LBPoolPollster LBPoolsDiscovery 0002 25425 aa.aa.aa.aa T1 Working
Discovery
Agent 2
LBPoolsDiscovery
Thread
ID source resource interval pollster agentID agentPID agentIP timestamp status
001 lb_pool_source location_1 600 LBPoolPollster T2 Ready
002 lb_pool_source location_2 600 LBPoolPollster T2 Ready
003 lb_pool_source location_3 600 LBPoolPollster T2 Ready
Create or Update (if changes)

7. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Step 2b: Sources Discovery
Discovery
Agent N
InstanceDiscovery
Thread
ID source resource interval pollster agentID agentPID agentIP timestamp status
004 cpu_source location_4 400 CPUPollster T2 Ready
005 cpu_source location_5 400 CPUPollster T2 Ready
Create or Update (if changes)
source interval pollster discovery agentID agentPID agentIP timestamp status
cpu_source 400 CPUPollster InstanceDiscovery 0000N 76746 bb.bb.bb.bb T1 Working

8. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Step 3: Polling Task Assignments (Scenario 1)
ID source resource interval pollster agentID agentPID agentIP timestamp status
001 lb_pool_source location_1 600 LBPoolPollster 0003 T2 Ready
002 lb_pool_source location_2 600 LBPoolPollster 0004 T2 Ready
003 lb_pool_source location_3 600 LBPoolPollster 0005 T2 Ready
004 cpu_source location_4 400 CPUPollster 0006 T2 Ready
005 cpu_source location_5 400 CPUPollster 000M T2 Ready
Polling Agent 1
…
Polling Agent 2
Polling Agent 3
Polling Agent 4
Polling Agent 5
Polling Agent 6
Polling Agent
M

9. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Step 4: Polling Execution (Scenario 1)
CPUPollster
Thread
Polling Agent 6
RabbitMQ
Resource
PollPublish
ID source resource interval pollster agentID agentPID agentIP timestamp status
004 cpu_source location_4 400 CPUPollster 0006 3242 xx.xx.xx.xx T2 Ready
ID source resource interval pollster agentID agentPID agentIP timestamp status
004 cpu_source location_4 400 CPUPollster T3 Working
RabbitMQ
Subscriber
Agent 1
Sends ping with status=Working
Receives ping and update status

10. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Step 4: Polling Completion → Discovery Ready
CPUPollster
Thread
Polling Agent 6
ID source resource interval pollster agentID agentPID agentIP timestamp status
004 cpu_source location_4 400 CPUPollster 0006 3242 xx.xx.xx.xx T3 Working
ID source resource interval pollster agentID agentPID agentIP timestamp status
004 cpu_source location_4 400 CPUPollster T4 Completed
RabbitMQ
Subscriber
Agent 5
Sends ping with status=Completed
Receives ping and update status

11. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
Step 3: Polling Task Assignments (Scenario 2)
ID source resource interval pollster agentID agentPID agentIP timestamp status
001 lb_pool_source location_1 600 LBPoolPollster 0001 … … … Ready
002 lb_pool_source location_2 600 LBPoolPollster 0002 … … … Ready
003 lb_pool_source location_3 600 LBPoolPollster 0003 … … … Ready
004 cpu_source location_4 400 CPUPollster 0001 … … … Ready
005 cpu_source location_5 400 CPUPollster 0002 … … … Ready
Polling Agent 1
Polling Agent 2
Polling Agent 3

12. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Step 4: Polling Execution (Scenario 2)
CPUPollster
Thread
Polling Agent 1
RabbitMQ
Resource
Poll
Publish
LBPoolPollster
Thread
Resource
Poll
Publish
ID source resource interval pollster agentID agentPID agentIP timestamp Status
001 lb_pool_source location_1 600 LBPoolPollster 00001 … … … Ready
004 cpu_source location_4 400 CPUPollster 00001 … … … Ready

13. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Discovery/Polling Agents HA: Homebrew Solution
Discovery
Agent 1
Discovery
Agent 2
Discovery
Agent 3
Discovery
Agent N
…
RabbitMQ
Subscriber
Agent 1
Subscriber
Agent 2
Ping with Status
source inte
rval
pollster discovery agentID agentPID agentIP timestamp status
lb_pool_source 600 LBPoolPollster LBPoolsDiscovery 00002 25425 aa.aa.aa.aa hh:mm:ssTdd:mm:yyyy Working
cpu_source 400 CPUPollster InstanceDiscovery 0000N 76746 bb.bb.bb.bb hh:mm:ssTdd:mm:yyyy No
Response
Updates

14. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Discovery/Polling Agents HA: Homebrew Solution
Discovery
Agent N
RabbitMQ
Subscriber
Agent 1
source inte
rval
pollster discovery agentID agentPID agentIP timestamp status
lb_pool_source 600 LBPoolPollster LBPoolsDiscovery 00002 25425 aa.aa.aa.aa hh:mm:ssTdd:mm:yyyy Working
cpu_source 400 CPUPollster InstanceDiscovery Ready
Restart

15. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Discovery/Polling Agents HA: Taskflow?
Discovery
Agent 1
Discovery
Agent 2
Discovery
Agent 3
Discovery
Agent N
…
Taskflow
source inte
rval
pollster discovery agentID agentPID agentIP timestamp status
lb_pool_source 600 LBPoolPollster LBPoolsDiscovery 00002 25425 aa.aa.aa.aa hh:mm:ssTdd:mm:yyyy OK
cpu_source 400 CPUPollster InstanceDiscovery 0000N 76746 bb.bb.bb.bb hh:mm:ssTdd:mm:yyyy No
Response
Job Table
Zookeeper

16. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you