openATTIC is the webUI for managing ceph storage in SUSE Linux and it is also accepted upstream as part of the default managing UI. Within openATTIC, it contains lots of different projects to provide all the functions including salt, Grafana and Prometheus. This will not be a ceph or openATTIC focus talk, even we use it as the example of showing how everything work together. Instead we will look into openATTIC to see how each project work with each other. Mainly focus on Grafana and Prometheus, they are not only very useful for ceph and openATTIC but also equally powerful for monitoring your clusters, vm or cloud / container status. Both Grafana and Prometheus are very easy to extend, which allow user or administrator to build the dashboard which fit their own need. Even the presentation example will be base on ceph / storage. Participants can use the same idea to monitor any system status they wanted after understand how they work.

1. openATTIC using
Grafana and
Prometheus

2. Alex Lau
Technical Consultant
SUSE
alau@suse.com

3. Things related to openATTIC
● Ceph
● Salt and DeepSea
● Grafana + Prometheus

4. Ceph Overview
❖ https://github.com/ceph
❖ Ceph is a distributed storage system
❖ Object, Block and File storage all in one!
❖ Self Recovery and Self healing!
❖ openATTIC become part of the management
interface
➢ https://www.openattic.org/posts/ceph-manager-dashboard-v2/

5. Ceph Architecture

6. Salt Overview
❖ https://github.com/saltstack/salt
❖ Salt is configuration management like ansible
➢ possible more :)
❖ Agent based distributed remote execution system,
➢ possible agentless with ssh
❖ Python base DevOps Automation tools using ZeroMQ
❖ Communication back channel for multiple product in
SUSE product line, including SES: DeepSea, SUSE
Manager and Cloud

7. Salt Architecture

8. DeepSea Overview
❖ Orchestration with Salt to discover, deploy, config and
manage Ceph life cycle
❖ Scalable and unified interface for different cluster size
❖ Run DeepSea Stages:
➢ 0 – Preparation
■ sync salt, install dependencies/updates
➢ 1 – Discovery
■ query hardware and network
➢ 2 – Configuration
■ merge configuration, generate keys
➢ 3 – Deployment
■ install ceph, deploy MONs and OSDs
➢ 4 – Additional Services
■ deploy MDS, RGW, etc.
➢ 5 – Removal
■ if necessary to decommission system(s)

9. DeepSea Architecture
Salt
Deployment
HardDrive
Storage
Network
Discovery
OSD MDS
Service
CephFS
RADOS Gateway
openATTIC
iSCSI Gateway
NFS Ganesha
MON MSG
RGW iSCSI
openatti
c
ganes
ha
DeepSea

10. OpenATTIC Overview
❖ https://bitbucket.org/openattic/openattic/
❖ Ceph Management and Monitoring GUI
❖ A easy to use administrative tools that actually work
❖ Highly customizable and scale out with ceph
❖ Expert / Local configuration aware without creating
inconsistent ceph cluster
❖ In 3.X above Grafana and Prometheus included

11. OpenATTIC Dashboard

12. Grafana Overview
❖ https://github.com/grafana
❖ A beautiful Visualization and Monitoring platform
❖ Time series Analytics tool support multiple data store
❖ Highly customizable Dashboard design
❖ Query Editors, Alert Notification Service
❖ Community marketplace for Dashboard

13. Grafana Dashboard

14. Prometheus Overview
❖ https://github.com/prometheus
❖ Multi-dimensional data collects and store with time
series by metric / key value pairs
❖ Pull distributed model over http, scalable and
effective
➢ Push gateway is also supported
❖ Querying using PromQL is flexible and easy to use
❖ Grafana and other graphing dashboarding supported

15. Prometheus Architecture

16. Node Exporter Overview
❖ https://github.com/prometheus/node_exporter
❖ default running on port 9100
❖ Node_exporter is a metrics collectors written in
golang
❖ It act like agent allow prometheus collect data

17. ❖ http://node:9090/graph
Prometheus Web interface

18. Create your own Node Exporter
module
❖ After forking the node_exporter
➢ git clone https://github.com/AvengerMoJo/node_exporter.git
❖ Use other collect as example to start
➢ https://github.com/AvengerMoJo/node_exporter/blob/lio_exporter/collector/lio.go
❖ create iscsi module using NewLioCollector
70 func init() {
71 registerCollector("iscsi", defaultEnabled, NewLioCollector)
72 }

19. ❖ newLioMetric is internal function to create data structure
for node_exporter being pull from Prom
❖ lioCollector is internal structure store all the statistic data
❖ NewLioCollector return lioCollector with newLioMetric
statistics in the following code sample
75 func NewLioCollector() (Collector, error) {
… ( sysfs is node_exporter format need follow upstream )
80
81 metrics, _ := newLioMetric()
82
83 return &lioCollector{
84 fs: fs,
85 metrics: metrics}, nil
86 }

20. ❖ newLioMetric create data structure as below:
➢ namespace + lio_rbd + iops = iscsi_lio_rbd_iops
➢ namespace + lio_rbd + read = iscsi_lio_rbd_read
➢ namespace + lio_rbd + write = iscsi_lio_rbd_write
103 func newLioMetric() (*lioMetric, error) {
104
105 return &lioMetric{
… ( different backstore support not included here )
138 lioRbdIops: prometheus.NewDesc(
139 prometheus.BuildFQName(namespace, lioRbdSubsystem, "iops"),
140 "iSCSI RBD backstore transport operations.",
141 []string{"iqn", "tpgt", "lun", "rbd", "pool", "image"}, nil,
142 ),
143 lioRbdRead: prometheus.NewDesc(
144 prometheus.BuildFQName(namespace, lioRbdSubsystem, "read"),
145 "iSCSI RBD backstore Read in byte.",
146 []string{"iqn", "tpgt", "lun", "rbd", "pool", "image"}, nil,
147 ),
148 lioRbdWrite: prometheus.NewDesc(
149 prometheus.BuildFQName(namespace, lioRbdSubsystem, "write"),
150 "iSCSI RBD backstore Write in byte.",
151 []string{"iqn", "tpgt", "lun", "rbd", "pool", "image"}, nil,
152 ),

21. ❖ Ｕpdate return lioCollector status called by Prom
➢ fs.ISCSIStats() is from procfs module, sys status path helper
89 func (c *lioCollector) Update(ch chan<- prometheus.Metric) error {
90
91 stats, err := c.fs.ISCSIStats()
…
97 c.updateStat(ch, s)
...
176 func (c *lioCollector) updateStat(ch chan<- prometheus.Metric, s
*iscsi.Stats) error {
...
214 if err := c.updateRBDStat(ch, label); err != nil {
...
327 func (c *lioCollector) updateRBDStat(ch chan<- prometheus.Metric, label
graphLabel) error {
328

22. ❖ reading status from sys
335 readMB, writeMB, iops, err := iscsi.ReadWriteOPS(label.iqn, label.tpgt,
label.lun)
❖ setting variable to the right unit
340 fReadMB := float64(readMB << 20)
344 fWriteMB := float64(writeMB << 20)
348 fIops := float64(iops)
❖ setting the return Prom.Metrics
351 ch <- prometheus.MustNewConstMetric(c.metrics.lioRbdRead,
352 prometheus.CounterValue, fReadMB, label.iqn, label.tpgt, label.lun,
353 rbd.Name, rbd.Pool, rbd.Image)
354
355 ch <- prometheus.MustNewConstMetric(c.metrics.lioRbdWrite,
356 prometheus.CounterValue, fWriteMB, label.iqn, label.tpgt, label.lun,
357 rbd.Name, rbd.Pool, rbd.Image)
358
359 ch <- prometheus.MustNewConstMetric(c.metrics.lioRbdIops,
360 prometheus.CounterValue, fIops, label.iqn, label.tpgt, label.lun,
361 rbd.Name, rbd.Pool, rbd.Image)

23. Build and Run Node Exporter
module
❖ You need to copy your update into the your home go dir
➢ go build github.com/prometheus/node_exporter
❖ Default collector doesn’t start / enable
❖ Enable module -collectors or --collectors after 0.15
version
➢ node_exporter -collectors.enabled iscsi
❖ Then you can check your metrics and data in port 9100
➢ http://node:9100/metrics

24. Node Exporter
http://node:9100/metrics
# HELP iscsi_lio_rbd_iops iSCSI RBD backstore transport operations.
# TYPE iscsi_lio_rbd_iops counter
iscsi_lio_rbd_iops{image="demo",iqn="iqn.2016-11.org.linux-
iscsi.igw.x86:sn.demo",lun="lun_0",pool="iscsi-
images",rbd="rbd_0",tpgt="tpgt_1"} 474
# HELP iscsi_lio_rbd_read iSCSI RBD backstore Read in byte.
# TYPE iscsi_lio_rbd_read counter
iscsi_lio_rbd_read{image="demo",iqn="iqn.2016-11.org.linux-
iscsi.igw.x86:sn.demo",lun="lun_0",pool="iscsi-
images",rbd="rbd_0",tpgt="tpgt_1"} 4.194304e+06
# HELP iscsi_lio_rbd_write iSCSI RBD backstore Write in byte.
# TYPE iscsi_lio_rbd_write counter
iscsi_lio_rbd_write{image="demo",iqn="iqn.2016-11.org.linux-
iscsi.igw.x86:sn.demo",lun="lun_0",pool="iscsi-
images",rbd="rbd_0",tpgt="tpgt_1"} 1.048576e+07

26. Grafana Template

28. Thank You
SUSE is hiring!
Start creating your own
dashboard!