CloudStack Metering – Working with the Usage Data


Published on

Organisations looking to build and offer Cloud services on Apache CloudStack need to be able to either monetize their offerings and charge for usage or monitor and report on their Cloud's consumption.

Majority of such organisations already have existing billing or business support systems and do not require an integrated billing or reporting system, provided the usage data can be exported from CloudStack in a standard and structured format such as XML, JSON, or CSV.

Tariq Iqbal of ShapeBlue provides an overview of the Apache CloudStack Usage Metrics, Usage Server and the multitude of ways for accessing the Usage Data for billing and reporting purposes, including real world use cases of some of the integration projects he has worked on.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • eSkyCityBroker BinSunGardCiscoOrangeT-Mobile
  • eSkyCityBroker BinSunGardCiscoOrangeT-Mobile
  • CloudStack Metering – Working with the Usage Data

    1. 1. CloudStack Metering Working with the Usage Data Tariq Iqbal Senior Consultant Twitter: @TariqIqbal_ @ShapeBlue
    2. 2. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  Involved with CloudStack before donation to Apache  Built and deployed CloudStack/CloudPlatform based clouds for Enterprises and Service Providers globally  Specialise in integrating CloudStack with Business Support and Operational Support systems, which include:  CloudPortals (including Payment Gateways, Fraud Control solutions)  Ticketing Systems  CRM/ERP Systems  Monitoring Solutions About Me
    3. 3. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 “ShapeBlue are expert builders of public & private clouds. They are the leading global independent CloudStack / CloudPlatform integrator & consultancy” About ShapeBlue
    4. 4. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudStack metering - What and how?  CloudStack configuration to enable Usage capture  Usage Data generation process  Methods of accessing the Usage Data  Real World Use Cases Overview
    5. 5. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  Organisations need to be able to meter their cloud's consumption and monetize the cloud offerings by charging for usage  Majority of organisations already have existing billing or business support systems and just need access to CloudStack Usage Data  Organisations tend to have different billing criteria and charging models requiring a comprehensive integrated solution Metering Requirements
    6. 6. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 CloudStack Usage? CloudStack Metering
    7. 7. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudStack does not offer any integrated usage monitoring or billing/chargeback capability  Root Admin can view System Capacity  Domain Admin and Users can view VM status Metering in CloudStack today
    8. 8. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudStack includes a Usage Server that creates summary usage records for the various resources consumed in CloudStack  The Usage Server is an optional, separately-installed component called ‘cloudstack-usage’  The Usage Server runs at least once per day. It can be configured to run multiple times per day and it’s behaviour is controlled by the following global configuration settings: • Cloud Usage Server
    9. 9. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  As CloudStack resources are created, consumed and destroyed, appropriate Event records are created in the ‘cloud’ database table: usage_events  There are 110 different Events: (51 shown) CloudStack Events
    10. 10. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The Usage Server schedules a job at the time specified by usage.stats.job.exec.time parameter. This job queries the ‘cloud’ database table: usage_events and generates summary usage records in the ‘cloud_usage’ database table: cloud_usage as follows: 1. CloudStack events have been logged into the cloud.usage_event table 2. cloudstack_usage job gets a list of the latest usage events 3. Inserts the latest usage events into the cloud_usage.usage_events table 4. Helper tables populated from cloud_usage.usage_event table after parsing 5. cloud_usage.cloud_usage table populated with aggregation range wise data. Helper table data is used here Usage Record Generation
    11. 11. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The Usage Server collects 14 different types of resource usage: Usage Types
    12. 12. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The Usage Records show the amount of resources consumed by guest instances.  The Usage Record format for Allocated & Running VMs is shown below:  The other 7 Usage Record formats are:  Network Usage  IP Address  Disk Volume  Template, ISO & Snapshot  Load Balancer & Port Fwd  Network Offering  VPN User Usage Records
    13. 13. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 Usage Record Example
    14. 14. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 Accessing Usage Data CloudStack Metering
    15. 15. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The usage records can be accessed through the CloudStack API  This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed  CloudStack provides the following Usage API calls with Root Admin access and supports XML and JSON response formats  listUsageTypes – list of Usage Types accessible  generateUsageRecords - asynchronous usage record generation  listUsageRecords - provides Usage records for a date range CloudStack API
    16. 16. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The base command is: http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y yyy-MM-dd&enddate=yyyy-MM-dd&signature  Append the various options: ListUsageRecords API
    17. 17. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  <listusagerecordsresponse> <count>1</count> <usagerecord> <account>user5</account> <accountid>10004</accountid> <domainid>1</domainid> <zoneid>1</zoneid> <description>i-3-4-WC running time (Service Offering: 1) (Template: 3)</description> <usage>2.95288 Hrs</usage> <usagetype>1</usagetype> <rawusage>2.95288</rawusage> <virtualmachineid>4</virtualmachineid> <name>i-3-4-WC</name> <offeringid>1</offeringid> <templateid>3</templateid> <usageid>245554</usageid> <type>XenServer</type> <startdate>2009-09-15T00:00:00-0700</startdate> <enddate>2009-09-18T16:14:26-0700</enddate> </usagerecord> </listusagerecordsresponse> ListUsageRecords Response
    18. 18. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API  It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing  Filtering can be used to limit the result set  It supports argument passing and shell automation  The host, port, apikey and secretkey can be configured on setup CloudMonkey CLI
    19. 19. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 CloudMonkey list usagerecords The command to retrieve Usage Records is: cloudmonkey list usagerecords domainid=7ded1404-d7fc-11e2-a70f- 080027cfaf0b startdate=2013-06-01 enddate=2013-06-23 accountid=2
    20. 20. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack  Usage Data can be exported in a CSV format from a SELECT query  The SQL query can become complex if the ID fields in the data need to be de-referenced  Typically multiple passes of the query results may be required to extract the required information SQL
    21. 21. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  Once you are able to access the Usage Data, you will need to design the mapping between the fields in the CloudStack Usage Data and the fields of the target system.  Considerations:  Do you actually need all the Usage Data for all Resources?  Any UUID values of the CloudStack resources should also be imported into the target system for reference purposes  Decide which system will be the master for the user account data and if synchronisation is required Mediation
    22. 22. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 Real World Use Cases CloudStack Metering
    23. 23. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  MS Excel can be used to analyse CloudStack Usage Records  The usage records are imported from the cloud_usage. cloud_usage table into MS Excel via an MySQL ODBC connection  Either PivotTables or reports can be used to present the data.  This is convenient and great for integration testing between CloudStack and external systems. MS Excel
    24. 24. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  Citrix CloudPortal Business Manager tightly integrates with the cloud_usage database on CloudStack/CloudPlatform.  The Usage Data is cross referenced against the utility pricing and subscription pricing in CPBM.  Scheduled Billing and Invoicing jobs generate the invoices in XML or PDF based on the tenants billing cycle. Citrix CloudPortal
    25. 25. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  splunk> is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs  We then connected splunk> to the CloudStack Usage database and automated the collection and indexing of Usage Data  Setup feature-rich dashboards and the capability to drill down into the Usage Data for BI reporting splunk>
    26. 26. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudStack Developers Guide, API Documentation & Wiki:   CloudMonkey  cli.html  How to Use MS Excel to Analyze CloudStack Usage Records:   CloudStack Mailing Lists (Users, Development, Marketing):  Further Information
    27. 27. CloudStack Metering Working with the Usage Data Tariq Iqbal Senior Consultant Twitter: @TariqIqbal_ @ShapeBlue