• Save
CloudStack Metering – Working with the Usage Data
Upcoming SlideShare
Loading in...5
×
 

CloudStack Metering – Working with the Usage Data

on

  • 3,781 views

Organisations looking to build and offer Cloud services on Apache CloudStack need to be able to either monetize their offerings and charge for usage or monitor and report on their Cloud's ...

Organisations looking to build and offer Cloud services on Apache CloudStack need to be able to either monetize their offerings and charge for usage or monitor and report on their Cloud's consumption.

Majority of such organisations already have existing billing or business support systems and do not require an integrated billing or reporting system, provided the usage data can be exported from CloudStack in a standard and structured format such as XML, JSON, or CSV.

Tariq Iqbal of ShapeBlue provides an overview of the Apache CloudStack Usage Metrics, Usage Server and the multitude of ways for accessing the Usage Data for billing and reporting purposes, including real world use cases of some of the integration projects he has worked on.

Statistics

Views

Total Views
3,781
Slideshare-icon Views on SlideShare
3,754
Embed Views
27

Actions

Likes
8
Downloads
0
Comments
0

2 Embeds 27

http://www.linkedin.com 26
https://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • eSkyCityBroker BinSunGardCiscoOrangeT-Mobile
  • eSkyCityBroker BinSunGardCiscoOrangeT-Mobile

CloudStack Metering – Working with the Usage Data CloudStack Metering – Working with the Usage Data Presentation Transcript

  • CloudStack Metering Working with the Usage Data Tariq Iqbal Senior Consultant tariq.iqbal@shapeblue.com Twitter: @TariqIqbal_ @ShapeBlue
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  Involved with CloudStack before donation to Apache  Built and deployed CloudStack/CloudPlatform based clouds for Enterprises and Service Providers globally  Specialise in integrating CloudStack with Business Support and Operational Support systems, which include:  CloudPortals (including Payment Gateways, Fraud Control solutions)  Ticketing Systems  CRM/ERP Systems  Monitoring Solutions About Me
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 “ShapeBlue are expert builders of public & private clouds. They are the leading global independent CloudStack / CloudPlatform integrator & consultancy” About ShapeBlue
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudStack metering - What and how?  CloudStack configuration to enable Usage capture  Usage Data generation process  Methods of accessing the Usage Data  Real World Use Cases Overview
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  Organisations need to be able to meter their cloud's consumption and monetize the cloud offerings by charging for usage  Majority of organisations already have existing billing or business support systems and just need access to CloudStack Usage Data  Organisations tend to have different billing criteria and charging models requiring a comprehensive integrated solution Metering Requirements
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 CloudStack Usage? CloudStack Metering
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudStack does not offer any integrated usage monitoring or billing/chargeback capability  Root Admin can view System Capacity  Domain Admin and Users can view VM status Metering in CloudStack today
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudStack includes a Usage Server that creates summary usage records for the various resources consumed in CloudStack  The Usage Server is an optional, separately-installed component called ‘cloudstack-usage’  The Usage Server runs at least once per day. It can be configured to run multiple times per day and it’s behaviour is controlled by the following global configuration settings: • Cloud Usage Server
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  As CloudStack resources are created, consumed and destroyed, appropriate Event records are created in the ‘cloud’ database table: usage_events  There are 110 different Events: (51 shown) CloudStack Events
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The Usage Server schedules a job at the time specified by usage.stats.job.exec.time parameter. This job queries the ‘cloud’ database table: usage_events and generates summary usage records in the ‘cloud_usage’ database table: cloud_usage as follows: 1. CloudStack events have been logged into the cloud.usage_event table 2. cloudstack_usage job gets a list of the latest usage events 3. Inserts the latest usage events into the cloud_usage.usage_events table 4. Helper tables populated from cloud_usage.usage_event table after parsing 5. cloud_usage.cloud_usage table populated with aggregation range wise data. Helper table data is used here Usage Record Generation
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The Usage Server collects 14 different types of resource usage: Usage Types
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The Usage Records show the amount of resources consumed by guest instances.  The Usage Record format for Allocated & Running VMs is shown below:  The other 7 Usage Record formats are:  Network Usage  IP Address  Disk Volume  Template, ISO & Snapshot  Load Balancer & Port Fwd  Network Offering  VPN User Usage Records
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 Usage Record Example
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 Accessing Usage Data CloudStack Metering
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The usage records can be accessed through the CloudStack API  This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed  CloudStack provides the following Usage API calls with Root Admin access and supports XML and JSON response formats  listUsageTypes – list of Usage Types accessible  generateUsageRecords - asynchronous usage record generation  listUsageRecords - provides Usage records for a date range CloudStack API
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The base command is: http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y yyy-MM-dd&enddate=yyyy-MM-dd&signature  Append the various options: ListUsageRecords API
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  <listusagerecordsresponse> <count>1</count> <usagerecord> <account>user5</account> <accountid>10004</accountid> <domainid>1</domainid> <zoneid>1</zoneid> <description>i-3-4-WC running time (Service Offering: 1) (Template: 3)</description> <usage>2.95288 Hrs</usage> <usagetype>1</usagetype> <rawusage>2.95288</rawusage> <virtualmachineid>4</virtualmachineid> <name>i-3-4-WC</name> <offeringid>1</offeringid> <templateid>3</templateid> <usageid>245554</usageid> <type>XenServer</type> <startdate>2009-09-15T00:00:00-0700</startdate> <enddate>2009-09-18T16:14:26-0700</enddate> </usagerecord> </listusagerecordsresponse> ListUsageRecords Response
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudMonkey is a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API  It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing  Filtering can be used to limit the result set  It supports argument passing and shell automation  The host, port, apikey and secretkey can be configured on setup CloudMonkey CLI
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 CloudMonkey list usagerecords The command to retrieve Usage Records is: cloudmonkey list usagerecords domainid=7ded1404-d7fc-11e2-a70f- 080027cfaf0b startdate=2013-06-01 enddate=2013-06-23 accountid=2
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  The usage records can be also accessed by directly querying the cloud_usage.cloud_usage MySQL table in CloudStack  Usage Data can be exported in a CSV format from a SELECT query  The SQL query can become complex if the ID fields in the data need to be de-referenced  Typically multiple passes of the query results may be required to extract the required information SQL
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  Once you are able to access the Usage Data, you will need to design the mapping between the fields in the CloudStack Usage Data and the fields of the target system.  Considerations:  Do you actually need all the Usage Data for all Resources?  Any UUID values of the CloudStack resources should also be imported into the target system for reference purposes  Decide which system will be the master for the user account data and if synchronisation is required Mediation
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013 Real World Use Cases CloudStack Metering
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  MS Excel can be used to analyse CloudStack Usage Records  The usage records are imported from the cloud_usage. cloud_usage table into MS Excel via an MySQL ODBC connection  Either PivotTables or reports can be used to present the data.  This is convenient and great for integration testing between CloudStack and external systems. MS Excel
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  Citrix CloudPortal Business Manager tightly integrates with the cloud_usage database on CloudStack/CloudPlatform.  The Usage Data is cross referenced against the utility pricing and subscription pricing in CPBM.  Scheduled Billing and Invoicing jobs generate the invoices in XML or PDF based on the tenants billing cycle. Citrix CloudPortal
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  splunk> is an operational intelligence tool that was initially used for collecting and indexing infrastructure logs  We then connected splunk> to the CloudStack Usage database and automated the collection and indexing of Usage Data  Setup feature-rich dashboards and the capability to drill down into the Usage Data for BI reporting splunk>
  • @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013  CloudStack Developers Guide, API Documentation & Wiki:  http://cloudstack.apache.org/  CloudMonkey  https://cwiki.apache.org/CLOUDSTACK/cloudstack-cloudmonkey- cli.html  How to Use MS Excel to Analyze CloudStack Usage Records:  http://support.citrix.com/article/CTX132030  CloudStack Mailing Lists (Users, Development, Marketing):  http://cloudstack.apache.org/mailing-lists.html Further Information
  • CloudStack Metering Working with the Usage Data Tariq Iqbal Senior Consultant tariq.iqbal@shapeblue.com Twitter: @TariqIqbal_ @ShapeBlue