See what happens when you overlay Ceilometer functionalities on top of a scalable solution like Monasca ... This is Ceilosca

1. Monasca + Ceilometer = Ceilosca
Fabio Giannetti, Srinivas Sakhamuri (Cisco)
Roland Hochmuth, Dan Dyer (HP)

2. Presentation Outline
• Motivations
• Ceilometer Architecture
• Ceilosca Architecture
• Performance Testing
• Status and Availability
• What Next?
• Acknowledgments

3. Address the Data
Explosion
Logs, Metrics, Events,
Active/Passive Checks,
…
End-to-End Debugging
Understand what the real
issue is and what is affected
when errors occur
Visibility
Deliver centralized
intelligence for cloud
operations at scale
Operator Needs
Resource
Utilization
Understand resource
availability and
utilization
Solution Requirements
Able to Collect,
Store and
Access all
types of data in
one place
Highly
Performant
and Scalable
Platform
Flexible Processing Pipeline
that can support multiple use
cases: diagnostics, root cause
analysis, SLA calculations,
utilization reporting, …
Extensible Platform
that can be extended
to support new types
of data and processing
Motivations: Operator Needs and Solution Requirements

4. Ceilosca, the What and Why …
WHAT?
Ceilosca is Ceilometer built on top of Monasca.
Two components are used:
1. The Ceilometer Publishing Agent and pipeline to send samples to the
Monasca API.
2. The Ceilometer Monasca Storage Driver, to query Monasca. This enables the
Ceilometer API providing backward compatibility for existing applications
WHY?
1. Improve HA, performance and scale of telemetry and monitoring
2. Provide monitoring with all the OpenStack resource metrics/events
3. Put all the telemetry and monitoring data into one place to enable more
effective OpenStack operations and insights.
We see this as a Win-Win.

5. Open Source, Extensible, Scalable & High Performance Platform based
on a Micro-services architecture
Log storage
and
processing
Metrics
storage,
retrieval and
statistics
Pluggable
Notification &
Integration
Support
Real-time
OpenStack
Notification/Event
stream processing
Real time
threshold &
alarm
engine
Operating System, Hardware
and Application
Metrics/Status/Checks
OpenStack
Notifications
OpenStack
Service Metrics
Log
Data
(ELK)
Monasca API
Ceilometer API
Monasca + Ceilometer = Ceilosca

6. MongoDB
Ceilometer
Agent
Swift Nova Cinder Glance
Ceilometer
Collector
Ceilometer
V2 API
Notification Bus
(RabbitMQ)
Ceilometer Bus
(RabbitMQ)
Ceilometer Current Architecture
poll

7. InfluxDB
Ceilometer
Agent
Swift Nova Cinder Glance
Monasca
API
Ceilometer
V2 API
Notification Bus
(RabbitMQ)
Ceilosca Architecture
Monasca
Persister
Monasca
Message Queue
(Kafka)
poll
Cassandra
(under dev.)

8. InfluxDB
Ceilometer Agent
Swift Nova Cinder Glance
Monasca
API
Ceilometer V2 API
Notification Bus
(RabbitMQ)
Ceilosca Architecture Details
Monasca
Persister
Monasca
Message Queue
(Kafka)
Publisher
Interface
Monasca
Publisher
Storage Base
Interface
Monasca Driver
poll
Cassandra
(under dev.)

9. Test Scenarios and Environment
Private Cloud Simulation Public Cloud Simulation
• 5 tenants
• 10 resources each (4 compute, 4
volume, 2 image)
• 5K measure per day (20 second
interval)
• 250K total per day
• 7.5 Mil = 1 Month
• 500 tenants
• 1 resources each (1 compute or 1
volume)
• 0.5K measure per day (~3 minute
interval)
• 250K total per day
• 7.5 Mil = 1 Month
Environment: Nova Virtual Machine
16 VCPU
32 GB RAM
50 GB Root Disk
Environment: Baremetal
Intel(R) Xeon(R) E5-2637 CPU, 4 Cores
96 GB RAM
1 TB Root Disk

10. Ceilometer
Agents
Notification Bus
(RabbitMQ)
Load Test: How it was performed for Ceilometer
Oslo Messaging
Simulator
Ceilometer
Collector
Ceilometer Bus
(RabbitMQ)
MongoDB

11. Ceilometer
Agents
Notification Bus
(RabbitMQ)
Load Test: How it was performed for Ceilosca
Oslo Messaging
Simulator
InfluxDB
Monasca
API
Monasca
Persister
Monasca
Message Queue
(Kafka)

12. VM Load Test: Public Cloud Simulation Results
1.2M 2.4M 4.8M 7.5M 9M
Ceilometer Time 5:55:38 6:57:29
Ceilosca Time 1:42:17 2:07:28 3:01:56 5:46:33
0:00:00
1:12:00
2:24:00
3:36:00
4:48:00
6:00:00
7:12:00
8:24:00
Timetocomplete(h:mm:ss)
Failed
Failed
Takeaways:
1. Ceilosca is 3.5x faster
2. Ceilosca consumes 2x to 3x less space

13. VM Query Test: Private Cloud Simulation Results
1.2M 2.4M 4.8M 7.5M 9M
Ceilometer Time 2:09:58 3:30:54
Ceilosca Time 0:40:03 1:04:29 1:51:02 2:40:58
0:00:00
0:28:48
0:57:36
1:26:24
1:55:12
2:24:00
2:52:48
3:21:36
3:50:24
Timetocomplete(h:mm:ss)
Failed
Failed
Takeaways:
• Ceilosca is 2.9x faster
• Ceilosca consumes 2x to 3x less space
• Both store faster when less Tenants/Resources

14. Query Test: How it was performed for both
Monasca
API
MongoDB
Ceilometer
V2 API
curl
'http://10.0.2.15:8777/v2/meters/volume?q.field=proj
ect_id&q.field=timestamp&q.field=timestamp&q.op=
eq&q.op=ge&q.op=le&q.type=&q.type=&q.type=&q.v
alue=…&q.value=…&q.value=…
• Disabled Keystone
• One query per tenant and
24h timestamps range
• 10 repetitions for 5 tenants

15. VM Query Test: Private Cloud Simulation Results
300K, 600K, 1.2M and 2.4M Samples – Instance Query
300K 600K 1.2M 2.4M
Ceilosca 10.96196 20.41848 40.09712 81.98306
Ceilometer 24.201712 50.76138 102.66366 204.93828
0
50
100
150
200
250
seconds
Takeaways:
• Ceilosca is 2.4x faster

16. VM Query Test: Public Cloud Simulation Results
300K, 600K, 1.2M and 2.4M Samples – Instance Query
300K 600K 1.2M 2.4M
Ceilosca 0.30624 0.54474 1.2114 2.20994
Ceilometer 3.493556 7.33076 13.88872 29.19672
0
5
10
15
20
25
30
35
seconds
Takeaways:
• Ceilosca is 11x faster
• Ceilometer performance significantly degrades
when the number of tenants or resources increases.

17. Baremetal Query Test: Public Cloud Simulation Results
7.5M Samples – Instance Query
Ceilometer Ceilosca
7.5M 53.1 2.907
0
10
20
30
40
50
60
time(s)
Takeaways:
• Ceilosca is 18x faster

18. Baremetal Query Test: Public Cloud Simulation Results
7.5M Samples – Instance Query
Ceilosca Monasca
7.5M 2.907 1.621
0
0.5
1
1.5
2
2.5
3
3.5
time(s)
Takeaways:
• Monasca API is 1.8x faster
than Ceilosca so we can
improve a lot!

19. Features/Capabilities currently supported
Ceilometer Agent
Publisher
Interface
Monasca
Publisher
Ceilometer V2 API
Storage Base
Interface
Monasca Driver
Query Type Meters Resources Samples Statistics
Simple
Metadata
Complex
Pagination
Group by
Aggregation
Max, Min, Sum, Avg, Count
Stddev, cardinality
Connects to Monasca API through Monasca client
Uses Keystone to authenticate POSTs
Supports pipeline since it affects only the publisher part
Fully configurable dimensions/metadata selection

20. Where to get it and how to run it …
Code is fully available at:
https://github.com/openstack/monasca-ceilometer
What is already there:
Monasca Publisher code, Monasca Storage code, Unit Tests and
automated Ansible Deployment on Devstack + Load Simulator.
Getting Ceilosca to work on Devstack is as easy as …
1. Clone repo in baremetal or VM
2. Run ./deployer/ceilosca.sh
3. Go for a walk … (It takes around 30/45min)
4. Have Fun with Ceilosca

21. What Next? … Roadmap
Monasca and Ceilosca code is now under Openstack repo, so we
hope to integrate the Ceilometer portions of Ceilosca back in the
Ceilometer repo (it makes a lot of sense).
Extend Ceilosca to also publish events to Monasca
Further improve Ceilosca query times with faster JSON parsing
and parallel queries to Monasca API.
Collaborate with Ceilometer to fully take advantage of inline
alarming available in Monasca.

22. Acknowledgements
It was a collective effort … so our thank goes to:
Atul Aggarwal
Jenny Wei
Ken Owens
Pauline Yeung
Putta Challa
Rohit Jaiswal
Steven Irvin

23. Thank You