20161027 hands on-gnocchicloudkitty

Configure your cloud to be able to charge
your users using official OpenStack
components!
27 octobre 2016
Ref 20160424-ObjectifLibre-
HandsOntoConfigureYourCloudtoBeAbletoChargeYourUsersUsingOfficialOpenStackComponents
Objectif Libre
http://www.objectif-libre.com
contact@objectif-libre.com

Table des matières
1 Let’s meet your hosts ! 2
2 Today’s plate ! 3
2.1 Start by getting your image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Grab an IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3 Your infrastructure ! 4
3.1 What’s to expect in that image ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4 All you need is good tools ! 5
4.1 Ceilometer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4.2 Gnocchi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.3 CloudKitty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5 Today’s setup 7
5.1 What do we use ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
5.2 Why do we need Ceilometer and Gnocchi ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
6 Let’s cook gnocchis ! 8
6.1 Who are you ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
6.2 Installing components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
6.3 Configuring Keystone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
6.4 Creating Gnocchi database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.5 Configuring Gnocchi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.6 Enable keystone Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.7 Update the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.8 Start the daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
7 Collecting metrics 11
7.1 Installing ceilometer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
8 CloudKitty’s turn ! 12
8.1 Installing components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
8.2 Configuring Keystone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
8.3 Creating the CloudKitty database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
8.4 Configuring CloudKitty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
8.5 Setting up rating rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
9 Here comes the pricing policy ! 16
9.1 First of all, restart apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9.2 Let’s create your user/project ! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9.3 Tell CloudKitty to charge your tenant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9.4 Let’s use Horizon for the policy ! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
9.5 Let’s fill some data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
9.6 Resetting calculations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
9.7 Reporting view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
9.8 Predictive Pricing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
i

10 It is time for questions ! 21
ii

Conﬁgure your cloud to be able to charge your users using ofﬁcial OpenStack components!
∙ Let’s meet your hosts
∙ Your infrastructure !
∙ All you need is good tools !
∙ Our setup
∙ Let’s cook gnocchis !
∙ Collecting the metrics
∙ Cloudkitty’s turn
∙ Here comes the pricing policy
∙ Reporting
∙ It is time for questions
Table des matières 1

Let’s meet your hosts !
∙ Stéphane Albert (Objectif Libre)
Co-father of CloudKitty / Former CloudKitty PTL
∙ Christophe Sauthier (Objectif Libre)
CEO of Objectif Libre / PTL and co-Father of CloudKitty
∙ Julien Danjou (Red Hat)
PTL Telemetry / Father of Gnocchi
∙ Maxime Cottret (Objectif Libre)
Cloud Consultant / CloudKitty Contributor
∙ Luka Peschke (Objectif Libre)
DevOps trainee / Cloudkitty Contributor
2

Today’s plate !
Start by getting your image
Go to http ://olib.re/handson-bcn and save it !
Grab an IP
Go to https ://quotidien.framapad.org/p/osa-b and put your name next to an IP
3

Your infrastructure !
What’s to expect in that image ?
Some essential OpenStack components :
∙ A mariadb database.
∙ An AMQP (rabbitmq)
∙ memcached
∙ Keystone bootstrapped with an user (admin, password : adminpass) on a single project (admin).
∙ Glance
∙ Nova
∙ Horizon
∙ Neutron
Login : root / password : root
* PLEASE CHANGE IT * (using passwd once loggued by exemple)
4

All you need is good tools !
Ceilometer
OpenStack measurement project
Ceilometer (Telemetry) collects the usage of all the resources in an OpenStack Cloud.
Those informatios are stored in databases, and a REST API is provided to access them.
Some advanced capabilities (like alarming with aodh)
Architecture
Ceilometer is composed of various parts, the main ones are :
∙ ceilometer-api (controler) : API server
∙ ceilometer-collector (controler) : reads AMQP messages (from other components like nova)
∙ ceilometer-agent-central (controller) : polls some metrics directly
∙ ceilometer-agent-compute (compute node) : fetch informations related to instances (like CPU
usage)
Metrics
Metrics are the elements storing cloud usage, like :
∙ cpu_util : CPU usage percentage
∙ network.incoming.bytes : number of octets received on a network port
∙ disk.read.bytes : number of bytes read from a disk
Metrics can be of different types :
∙ gauge : value at the precise moment of collection
∙ cumulative : sum of all the previous values
∙ delta : difference since the last value
Samples
samples are the basis of ceilometer calculations : raw data stored after each polling or message received.
Each sample is composed of a record date, a name, a type, a value and an unit.
Scalability issue
A scalability issue raised due to the following reasons :
∙ Many requests done by API requires full scan of all stored samples
∙ one can ﬁlter on any ﬁelds
∙ metadata are not indexed (since free-form)
Some improvements have been tried, but many felt it wasn’t the good approach...
5

Gnocchi
TDBaaS (Time Series Database as a Service) project started under the Ceilometer program um-
brella.
The idea is to do aggregation of samples over a period of time to only store a reduced amount of data.
Gnocchi can be used as a storage backend for Ceilometer, and since it provides its own API some third party
software can uses the mecanism avoiding the scalability issue of Ceilometer.
Archive Policy
The number of samples can be different over the time
∙ 1 every second over the last day
∙ 1 every minute for the last week
∙ 1 every hour for the last 6 months
The indexer
Gnocchi supports many indexer (Postgres, Mysql) that are simply in charge of the relationship between metrics and
resources.
Every known OpenStack resources have a model, dropping the free-form metadata that were used in Ceilometer
(and thus improving speed/scalability)
CloudKitty
An Open Source rating/chargeback component for OpenStack
∙ Official component since Sept 2015
∙ Developped in python with all OpenStack best practices
∙ Integrated with Horizon
∙ Interfaced with both Ceilometer and Gnocchi
∙ Uber Modular
Modularity
4 Levels of modularity
∙ Collection of usage
∙ Pricing Policy
∙ Output format
∙ Output Storage
A Closer look at the Hashmap module
Aims at matching arbitrary values from a service type to resource metadata
∙ Group : Define set of rules that are processed together
∙ Service : Match a CK Service (compute, network,..)
∙ Field : Match a resource metadata field (flavor, image_id,...)
∙ Mapping
∘ Can be attached to a service or a field
∘ Map the value of a metadata to a cost and a type of cost
∙ Threshold
∘ Can be attached to a service or a field
∘ Define thresold levels on a metadata to a cost
4.2. Gnocchi 6

Today’s setup
What do we use ?
We are going to use Ceilometer/Gnocchi to collect metrics, CloudKitty for the rating, and Gnocchi to store them.
Why do we need Ceilometer and Gnocchi ?
Ceilometer is able to collect AND to store metrics. However, Gnocchi’s storage is much more efﬁcient, that’s why
we use it.
7

Let’s cook gnocchis !
Who are you ?
First of all, export some environment variables needed for keystone authentication :
source /root/admin.sh
Installing components
Gnocchi daemons
yum -y install openstack-gnocchi-{api,carbonara,indexer-sqlalchemy,metricd}
Gnocchi client
yum -y install python-gnocchiclient
Conﬁguring Keystone
Adding service and enpoint information
openstack service create --name gnocchi metric
Field Value
enabled True
id c1daec9e8fc44abb84c4b047b10de33a
name gnocchi
type metric
openstack endpoint create --region RegionOne
GNOCCHI_ID internal
http://127.0.0.1:8000/
8

Field Value
enabled True
id 6435d3148bda4384900a4a4a3a2c339e
interface internal
region RegionOne
region_id RegionOne
service_id GNOCCHI_ID
service_name gnocchi
service_type metric
url http://127.0.0.1:8000/
GNOCCHI_ID admin
http://127.0.0.1:8000/
Field Value
enabled True
id c6abad5d91364ef2ab1784d3d0987449
interface admin
region RegionOne
region_id RegionOne
service_type metric
url http://127.0.0.1:8000/
GNOCCHI_ID public
http://127.0.0.1:8000/
Field Value
enabled True
id 52fa79e7fbd74c9d994c1eaca04c530e
interface public
region RegionOne
region_id RegionOne
service_type metric
url http://127.0.0.1:8000/
Creating gnocchi user
openstack user create --project service --password password gnocchi
Field Value
default_project_id f4a80b4777554cbcaa3074c95be7fe3e
domain_id default
enabled True
id 7fb1ce78bb35436098cdb46cbafff67f
name gnocchi
openstack role add --user gnocchi --project service admin
Creating Gnocchi database
6.3. Conﬁguring Keystone 9

mysql -uroot -pmysqlpass << EOF
CREATE DATABASE gnocchi;
GRANT ALL PRIVILEGES ON gnocchi.* TO 'gnocchi'@'localhost'
IDENTIFIED BY 'gnocchidbpassword';
GRANT ALL PRIVILEGES ON gnocchi.* TO 'gnocchi'@'%'
IDENTIFIED BY 'gnocchidbpassword';
EOF
The mysql root password is mysqlpass
Configuring Gnocchi
cp /root/handson_files/gnocchi.conf /etc/gnocchi/gnocchi.conf
We are going to use Gnocchi’s file storage :
[storage]
driver = file
file_basepath = /var/lib/gnocchi
Enable keystone Authentication
Edit the /etc/gnocchi/api-paste.ini file by adding keytone authentication to pipeline:main
[pipeline:main]
pipeline = gnocchi+auth
Update the database
By default, only the default resource type exists in gnocchi. This will create all resource types needed by ceilometer.
gnocchi-upgrade --create-legacy-resource-types
Start the daemons
The openstack-gnocchi-api service requires Ceilometer to be installed, so we are going to use the gnocchi
daemons instead.
First, install screen :
yum -y install screen
Then, start gnocchi daemons :
screen -d -m /usr/bin/gnocchi-api
screen -d -m /usr/bin/gnocchi-metricd
6.4. Creating Gnocchi database 10

Collecting metrics
Installing ceilometer
When gnocchi is installed, Ceilometer’s API and storage are automatically disabled, so we don’t need to install
them.
yum -y install openstack-ceilometer-{collector,polling,notification}
Creating the ceilometer user
openstack user create --project service --password password ceilometer
Field Value
domain_id default
enabled True
id 7fb1ce78bb35436098cdb46cbafff67f
name ceilometer
openstack role add --user ceilometer --project service admin
Conﬁguring Ceilometer
cp /root/handson_files/ceilometer.conf /etc/ceilometer/ceilometer.conf
Starting Ceilometer daemons
systemctl start openstack-ceilometer-collector
systemctl start openstack-ceilometer-polling
systemctl start openstack-ceilometer-notification
11

CloudKitty’s turn !
CloudKitty daemons
yum -y install openstack-cloudkitty-{api,processor}
CloudKitty dashboard
yum -y install openstack-cloudkitty-ui
CloudKitty client
yum -y install python-cloudkittyclient
Conﬁguring Keystone
Adding service and enpoint information
openstack service create --name cloudkitty rating
Field Value
enabled True
id 88a1304ef6124b888463fd4cbc7c0e51
name cloudkitty
type rating
CLOUDKITTY_ID internal
http://127.0.0.1:8889/
Field Value
enabled True
id 6435d3148bda4384900a4a4a3a2c339e
interface internal
region RegionOne
region_id RegionOne
service_id CLOUDKITTY_ID
service_name cloudkitty
service_type rating
url http://127.0.0.1:8889/
12

CLOUDKITTY_ID admin
http://127.0.0.1:8889/
Field Value
enabled True
id c6abad5d91364ef2ab1784d3d0987449
interface admin
region RegionOne
region_id RegionOne
service_type rating
url http://127.0.0.1:8889/
CLOUDKITTY_ID public
http://127.0.0.1:8889/
Field Value
enabled True
id 52fa79e7fbd74c9d994c1eaca04c530e
interface public
region RegionOne
region_id RegionOne
service_type rating
url http://127.0.0.1:8888/
Adding rating role
openstack role create rating
Field Value
domain_id id name None 66af44925dfd403e8027ba89fdcd48e1 rating
Creating cloudkitty user
openstack user create --project service --password password cloudkitty
Field Value
domain_id default
enabled True
id 3a524d3f98cb49cea49d46e73433299c
name cloudkitty
openstack role add --user cloudkitty --project service admin
Creating the CloudKitty database
mysql -uroot -pmysqlpass << EOF
CREATE DATABASE cloudkitty;
GRANT ALL PRIVILEGES ON cloudkitty.* TO 'cloudkitty'@'localhost'
IDENTIFIED BY 'ckdbpassword';
8.3. Creating the CloudKitty database 13

GRANT ALL PRIVILEGES ON cloudkitty.* TO 'cloudkitty'@'%'
IDENTIFIED BY 'ckdbpassword';
EOF
Conﬁguring CloudKitty
cp /root/handson_files/cloudkitty.conf /etc/cloudkitty/cloudkitty.conf
By default, the collect period is 3600s, and only the compute service is enabled. To get quick results, we will set the
collect period to 90s.
[collect]
collector=gnocchi
period=90
services=compute, image, volume
Updating the database
cloudkitty-dbsync upgrade
| 2016-10-13 02:24:13.908 2816 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'cloudkitty
| INFO [alembic.runtime.migration] Context impl MySQLImpl.
| INFO [alembic.runtime.migration] Will assume non-transactional DDL.
| INFO [alembic.runtime.migration] Running upgrade -> 464e951dc3b8, Initial migration
| INFO [alembic.runtime.migration] Running upgrade 464e951dc3b8 -> 2ac2217dcbd9, Added support fo
| INFO [alembic.runtime.migration] Running upgrade 2ac2217dcbd9 -> 385e33fef139, Added priority t
| INFO [alembic.runtime.migration] Running upgrade -> 3dd7e13527f3, Initial migration
| INFO [alembic.runtime.migration] Running upgrade 3dd7e13527f3 -> 4fa888fd7eda, Added threshold
| INFO [alembic.runtime.migration] Running upgrade 4fa888fd7eda -> 54cc17accf2c, Fixed constraint
| INFO [alembic.runtime.migration] Running upgrade -> 4f9efa4601c0, Initial migration.
Setting up the storage
cloudkitty-storage-init
| INFO [alembic.runtime.migration] Running upgrade -> 4c2f20df7491, Initial migration.
Restarting the cloudkitty-api
systemctl restart cloudkitty-api
Setting up rating rules
List available rating modules :
cloudkitty module-list
8.4. Conﬁguring CloudKitty 14

Module Enabled
hashmap False
noop True
pyscripts False
Enable the hashmap rating module :
cloudkitty module-enable -n hashmap
Module Enabled
hashmap True
8.5. Setting up rating rules 15

Here comes the pricing policy !
First of all, restart apache
systemctl restart httpd
Let’s create your user/project !
openstack project create summit
openstack user create --password password --project summit summit_user
openstack role add --user summit_user --project summit admin
Tell CloudKitty to charge your tenant
openstack role add --user cloudkitty --project summit rating
Let’s use Horizon for the policy !
Log into horizon at the following url http://YOUR_IP/dashboard using the newly created user (sum-
mit_user/password).
It is time to create the pricing policy !
Creation of the service
Go to the configuration panel of the Hashmap module (inside Admin > Rating > Hashmap). Create a service called
“compute”
Creation of the instance_uptime group
Create a group (on the Groups tab). A group is a logical groupment of rating rules that will be defined later.
Creation of the field
Then, we need to choose the name we want to give to the metric in our policy. It will be named flavor_id in our
case.
Click on the name of the newly created field and select “Create new Mapping”.
∙ Value : the id of the flavor that will be charged by this rule. fooflavor in our case
∙ Type : it is a flat price for that rule
16

9.4. Let’s use Horizon for the policy ! 17

9.4. Let’s use Horizon for the policy ! 18

∙ Group : select the group you just created (instance_uptime)
That’s it ! You have defined a rule to charge your users depending on the used flavor. Let’s create a second one with
a different flavor id : barflavor.
Let’s fill some data
You can wait here for hours to use your “Cloud” or use the script :)
python /root/handson_files/gnocchi_script.py
Resetting calculations
systemctl stop cloudkitty-processor
mysql -u root -pmysqlpass -e 'delete from ghybrid_dataframes;' cloudkitty
systemctl start cloudkitty-processor
Now we will have to wait for the processor to fill the database.
G**Pro-tip -> This is the trainee’s h@Xx0r command to check :**
while true
do mysql -uroot -pmysqlpass -e
'select * from ghybrid_dataframes order by id desc limit 1' cloudkitty
| tail -n 1
sleep 5
done
Reporting view
If you go into Project/rating/reporting, you should have something like this :
9.5. Let’s fill some data 19

Predictive Pricing
openstack flavor list
Create a new mapping in the flavor_id field use the id of the m1.nano flavor as value.
Create an image_id field, and create a new mapping inside it with the ID of the glance image myimage the
same way you added the m1.nano flavor. Decide that it is a rate and it should be 10% more expensive than the
usual price.
Look at the predictive pricing !
Launch a few instances with the m1.nano flavor, and wait for a few minutes.
9.8. Predictive Pricing 20

It is time for questions !
Or later :
∙ stephane.albert@objectif-libre.com
∙ christophe.sauthier@objectif-libre.com
∙ maxime.cottret@objectif-libre.com
∙ luka.peschke@objectif-libre.com
∙ julien@danjou.info
We’ll be happy to send you the latest release of these slides !
21

20161027 hands on-gnocchicloudkitty

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 20161027 hands on-gnocchicloudkitty

Similar to 20161027 hands on-gnocchicloudkitty (20)

Recently uploaded

Recently uploaded (20)

20161027 hands on-gnocchicloudkitty