CloudStack Metering
Working with the Usage Data
Tariq Iqbal
Senior Consultant
tariq.iqbal@shapeblue.com
Twitter: @TariqIqb...
@ShapeBlue #CloudStack #CCCNA14
 Involved with CloudStack before donation to Apache
 Built and deployed CloudStack / Clo...
@ShapeBlue #CloudStack #CCCNA14
“ShapeBlue are expert builders of public & private clouds. They
are the leading global ind...
@ShapeBlue #CloudStack #CCCNA14
 CloudStack Usage
 CloudStack Configuration
 Usage Data Generation
 Usage Data Access
...
@ShapeBlue #CloudStack #CCCNA14
 Organisations need to be able to meter their cloud's
consumption to:
 Monetize the clou...
@ShapeBlue #CloudStack #CCCNA14
 CloudStack does not offer any
native usage monitoring or
billing/chargeback capability
...
@ShapeBlue #CloudStack #CCCNA14
 CloudStack includes a Usage Server:
 Optional Component
 Separately installed service ...
@ShapeBlue #CloudStack #CCCNA14
Usage Server Configuration
@ShapeBlue #CloudStack #CCCNA14
Usage Server Configuration Examples
@ShapeBlue #CloudStack #CCCNA14
 As CloudStack resources are created, consumed and destroyed,
appropriate Event records a...
@ShapeBlue #CloudStack #CCCNA14
1) CloudStack events are logged into cloud.usage_event table
2) The cloudstack_usage job g...
@ShapeBlue #CloudStack #CCCNA14
Usage Types
@ShapeBlue #CloudStack #CCCNA14
Usage Types
API: http://<HOSTIP>:8096/client/api?command=listUsageTypes
@ShapeBlue #CloudStack #CCCNA14
 Usage Records show the amount of resources consumed by guest instances.
 The 8 Usage Re...
@ShapeBlue #CloudStack #CCCNA14
VM Usage Record Format
@ShapeBlue #CloudStack #CCCNA14
 To Create, Start, Stop and Restart a VM:
Usage Record Example
@ShapeBlue #CloudStack #CCCNA14
Accessing Usage Data
CloudStack Usage Data
@ShapeBlue #CloudStack #CCCNA14
 The usage records can be accessed through the CloudStack API
 This is best done using t...
@ShapeBlue #CloudStack #CCCNA14
 The base API request is:
http://<HOST>:<8080>/client/api?command=listUsageRecords&startd...
@ShapeBlue #CloudStack #CCCNA14
 <listusagerecordsresponse>
<count>1</count>
<usagerecord>
<account>user5</account>
<acco...
@ShapeBlue #CloudStack #CCCNA14
 CloudMonkey is a CLI tool for CloudStack and can be used as an
interactive shell or from...
@ShapeBlue #CloudStack #CCCNA14
CloudMonkey list usagerecords
The command to retrieve Usage Records is:
cloudmonkey list u...
@ShapeBlue #CloudStack #CCCNA14
 The usage records can be also accessed by directly querying the
cloud_usage.cloud_usage ...
@ShapeBlue #CloudStack #CCCNA14
 CloudStack 4.1 started publishing events onto a message queue
 Uses RabbitMQ as the mes...
@ShapeBlue #CloudStack #CCCNA14
 Once you are able to access the Usage Data, you will need to
design the mapping between ...
@ShapeBlue #CloudStack #CCCNA14
Customer Use Cases
CloudStack Usage Data
@ShapeBlue #CloudStack #CCCNA14
 MS Excel can be used to analyse CloudStack Usage Records
 The usage records are importe...
@ShapeBlue #CloudStack #CCCNA14
 Citrix CloudPortal Business Manager tightly integrates with the
cloud_usage database on ...
@ShapeBlue #CloudStack #CCCNA14
 Splunk> is an operational intelligence tool that was initially used
for collecting and i...
@ShapeBlue #CloudStack #CCCNA14
 Provides usage visibility, cost control and IT billing for Private and
Hybrid Clouds
 D...
@ShapeBlue #CloudStack #CCCNA14
 Slides: www.slideshare.net/shapeblue
 Blogs: www.shapeblue.com/blogs
 CloudStack Progr...
Working with
CloudStack Usage Data
Tariq Iqbal
Senior Consultant
tariq.iqbal@shapeblue.com
Twitter: @TariqIqbal_ @ShapeBlue
Upcoming SlideShare
Loading in …5
×

CloudStack Metering - Working with Usage Data #CCCNA14

2,244 views

Published on

Organisations looking to build and offer Cloud services on Apache CloudStack need to be able to either monetize their offerings and charge for usage or monitor and report on their Cloud's consumption. Majority of such organisations already have existing billing or business support systems and do not require an integrated billing or reporting system provided the usage data can be exported from CloudStack in a standard and structured format such as XML, JSON, or CSV.
CloudStack includes a Usage Server that creates summary usage records for the various resources consumed in CloudStack. Tariq covers how usage of such resources is metered in CloudStack and also:
· What usage metrics are recorded
· Configuration of the Usage Server
· Creation of the Usage Data
· Explore various methods of accessing the Usage Data
· Overview of solutions for analysing or processing the Usage Data such as MS Excel, CloudPortal (CPBM), Splunk, Amysta.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,244
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

CloudStack Metering - Working with Usage Data #CCCNA14

  1. 1. CloudStack Metering Working with the Usage Data Tariq Iqbal Senior Consultant tariq.iqbal@shapeblue.com Twitter: @TariqIqbal_ @ShapeBlue
  2. 2. @ShapeBlue #CloudStack #CCCNA14  Involved with CloudStack before donation to Apache  Built and deployed CloudStack / CloudPlatform based clouds for Enterprises and Service Providers globally - SunGard, Ascenty  Specialise in integrating CloudStack with Business Support and Operational Support systems, which include:  CloudPortals (including Payment Gateways, Fraud Control solutions)  CRM/ERP Systems – SAPB/1, Salesforce.com  Monitoring Solutions – Nagios, Splunk, Zenoss  Ticketing Systems – ServiceDesk, Email About Me
  3. 3. @ShapeBlue #CloudStack #CCCNA14 “ShapeBlue are expert builders of public & private clouds. They are the leading global independent CloudStack / CloudPlatform integrator & consultancy” “First to offer professional 24x7x365 support of Apache CloudStack and now CSForge” “ShapeBlue is absolutely one of the top experts on deploying CloudStack. Great company and very deep skill set” About ShapeBlue
  4. 4. @ShapeBlue #CloudStack #CCCNA14  CloudStack Usage  CloudStack Configuration  Usage Data Generation  Usage Data Access  Customer Use Cases Overview
  5. 5. @ShapeBlue #CloudStack #CCCNA14  Organisations need to be able to meter their cloud's consumption to:  Monetize the cloud offerings by charging for usage  Report on the cloud’s resource usage for monitoring/capacity planning  Majority of organisations already have existing BSS/OSS and just need access to the usage data  Organisations tend to have different billing criteria/charging models and require access to usage data for their specific needs Usage Requirements
  6. 6. @ShapeBlue #CloudStack #CCCNA14  CloudStack does not offer any native usage monitoring or billing/chargeback capability  Root Admin can view current System Capacity  Domain Admin and Users can view VM statistics Usage in CloudStack today
  7. 7. @ShapeBlue #CloudStack #CCCNA14  CloudStack includes a Usage Server:  Optional Component  Separately installed service called ‘cloudstack-usage’  Runs once a day (Default)  Creates aggregated usage records for the various resources consumed in CloudStack  The usage records are stored in a separate database called ‘cloud_usage’ Usage Server
  8. 8. @ShapeBlue #CloudStack #CCCNA14 Usage Server Configuration
  9. 9. @ShapeBlue #CloudStack #CCCNA14 Usage Server Configuration Examples
  10. 10. @ShapeBlue #CloudStack #CCCNA14  As CloudStack resources are created, consumed and destroyed, appropriate Event records are created in cloud.usage_events  There are 110 different events. Full list in the Programmer Guide  Network usage is the only exception which doesn't use Events CloudStack Events
  11. 11. @ShapeBlue #CloudStack #CCCNA14 1) CloudStack events are logged into cloud.usage_event table 2) The cloudstack_usage job gets a list of the latest usage events 3) Inserts these events into cloud_usage.usage_events table 4) Parses cloud_usage.usage_event and populates Helper tables 5) Helper table data used to populate cloud_usage.cloud_usage table with aggregation range wise data Usage Record Generation
  12. 12. @ShapeBlue #CloudStack #CCCNA14 Usage Types
  13. 13. @ShapeBlue #CloudStack #CCCNA14 Usage Types API: http://<HOSTIP>:8096/client/api?command=listUsageTypes
  14. 14. @ShapeBlue #CloudStack #CCCNA14  Usage Records show the amount of resources consumed by guest instances.  The 8 Usage Record formats are:  Allocated & Running VMs  Network Usage  IP Address  Disk Volume  Template, ISO and Snapshot  Load Balancer or Port Forwarding  Network Offering  VPN User  Details of the Usage Record formats can be found in the Developers Guide Usage Records
  15. 15. @ShapeBlue #CloudStack #CCCNA14 VM Usage Record Format
  16. 16. @ShapeBlue #CloudStack #CCCNA14  To Create, Start, Stop and Restart a VM: Usage Record Example
  17. 17. @ShapeBlue #CloudStack #CCCNA14 Accessing Usage Data CloudStack Usage Data
  18. 18. @ShapeBlue #CloudStack #CCCNA14  The usage records can be accessed through the CloudStack API  This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed  CloudStack provides the following Root Admin API calls and responds with XML and JSON responses:  listUsageTypes – lists available Usage Types  listUsageRecords - provides Usage records for a date range  generateUsageRecords - asynchronous usage record generation CloudStack API
  19. 19. @ShapeBlue #CloudStack #CCCNA14  The base API request is: http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y yyy-MM-dd&enddate=yyyy-MM-dd&signature  Optional request parameters: ListUsageRecords API
  20. 20. @ShapeBlue #CloudStack #CCCNA14  <listusagerecordsresponse> <count>1</count> <usagerecord> <account>user5</account> <accountid>10004</accountid> <domainid>1</domainid> <zoneid>1</zoneid> <description>i-3-4-WC running time (Service Offering: 1) (Template: 3)</description> <usage>2.95288 Hrs</usage> <usagetype>1</usagetype> <rawusage>2.95288</rawusage> <virtualmachineid>4</virtualmachineid> <name>i-3-4-WC</name> <offeringid>1</offeringid> <templateid>3</templateid> <usageid>245554</usageid> <type>XenServer</type> <startdate>2009-09-15T00:00:00-0700</startdate> <enddate>2009-09-18T16:14:26-0700</enddate> </usagerecord> </listusagerecordsresponse> ListUsageRecords Response
  21. 21. @ShapeBlue #CloudStack #CCCNA14  CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API  It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing  Filtering can be used to limit the result set  It supports argument passing and shell automation  The host, port, apikey and secretkey can be configured on setup CloudMonkey CLI
  22. 22. @ShapeBlue #CloudStack #CCCNA14 CloudMonkey list usagerecords The command to retrieve Usage Records is: cloudmonkey list usagerecords domainid=7ded1404-d7fc-11e2-a70f- 080027cfaf0b startdate=2013-06-01 enddate=2013-06-23 accountid=2
  23. 23. @ShapeBlue #CloudStack #CCCNA14  The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack  Usage Data can be exported in a CSV format from a SELECT query  The SQL query can become complex if the ID fields in the data need to be de-referenced  Typically multiple passes of the query results may be required to extract the required information SQL
  24. 24. @ShapeBlue #CloudStack #CCCNA14  CloudStack 4.1 started publishing events onto a message queue  Uses RabbitMQ as the message broker, but likely to work with other AMQP- based brokers  Use case: A third-party cloud usage solution can subscribe to CloudStack events and generate usage data which can be consumed by their billing software  Great blog from Chip Childers on configuring a CloudStack management server to publish events, and some sample code that prints each event to the console as they are received  http://www.chipchilders.com/blog/2013/7/16/tapping-into-apache-cloudstack- events-via-amqp.html  http://cloudstack.apache.org/docs/en- US/Apache_CloudStack/4.1.0/html/Admin_Guide/events.html CloudStack Message Queue
  25. 25. @ShapeBlue #CloudStack #CCCNA14  Once you are able to access the Usage Data, you will need to design the mapping between the fields in the CloudStack Usage Data and the fields of the target system  Considerations:  Do you actually need all the Usage Data for all Resources?  Any UUID values of the CloudStack resources should also be imported into the target system for reference purposes  Will you need to synchronise accounts and resources in CloudStack with the target system? Mediation
  26. 26. @ShapeBlue #CloudStack #CCCNA14 Customer Use Cases CloudStack Usage Data
  27. 27. @ShapeBlue #CloudStack #CCCNA14  MS Excel can be used to analyse CloudStack Usage Records  The usage records are imported from the cloud_usage table into MS Excel via an MySQL ODBC connection  Either PivotTables or reports can be used to present the data  This is convenient and great for integration testing between CloudStack and external systems MS Excel
  28. 28. @ShapeBlue #CloudStack #CCCNA14  Citrix CloudPortal Business Manager tightly integrates with the cloud_usage database on CloudStack/CloudPlatform  The Usage Data is cross referenced against the utility pricing and subscription pricing in CPBM  Scheduled Billing and Invoicing jobs generate the invoices in XML or PDF based on the users billing cycle Citrix CloudPortal
  29. 29. @ShapeBlue #CloudStack #CCCNA14  Splunk> is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs  We then connected splunk> to the CloudStack Usage database and automated the collection and indexing of Usage Data  Setup feature-rich dashboards and the capability to drill down into the Usage Data for BI reporting Splunk>
  30. 30. @ShapeBlue #CloudStack #CCCNA14  Provides usage visibility, cost control and IT billing for Private and Hybrid Clouds  Dashboard Integrated in the CloudStack UI  Pricing, Alerting, Reporting, Capacity Mgmt  Amysta consolidates public cloud expenses e.g. AWS and private cloud consumption in one dashboard view Amysta
  31. 31. @ShapeBlue #CloudStack #CCCNA14  Slides: www.slideshare.net/shapeblue  Blogs: www.shapeblue.com/blogs  CloudStack Programmer Guide, API Documentation & Wiki: http://cloudstack.apache.org/  CloudStack Mailing Lists (Users, Development, Marketing): http://cloudstack.apache.org/mailing-lists.html  CloudMonkey: https://cwiki.apache.org/CLOUDSTACK/cloudstack+cloudmonkey+cli  How to Use MS Excel to Analyze CloudStack Usage Records: http://support.citrix.com/article/CTX132030 Further Information
  32. 32. Working with CloudStack Usage Data Tariq Iqbal Senior Consultant tariq.iqbal@shapeblue.com Twitter: @TariqIqbal_ @ShapeBlue

×