WSO2 Stratos - Tenent CPU    usage metering    Lasindu Charith Vidana Pathiranage          University of Moratuwa
Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
Current implementation for tenant usage     metering and billing in Stratos         ✔   Number of users         ✔   Disk S...
What are the available CPU usage      metering methods ?       ●           Per Hour       ●           Per Instance        ...
Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
Java ThreadMXBean to rescue..●    The management interface for the thread system    of the JVM.●    Java virtual machine i...
Sample Code Usageif (Metering is enabled) { //check the system property   (carbon.xml)if(tenant!= carbon.super && context ...
CPU time of requests passing through                  Tomcat to ODEThread Pool                 Request Request            ...
ESB CPU Time …           Synapse             Proxy Service               In Sequence                Class                 ...
How it works ?? Thread Execution Component  CpuUsageStatisticsContainer                                              CpuUs...
Usage Agent -> BAM                                 Publish    Usage Agent Component                            Usage      ...
Overall Architecture
BpsCpuUsageStatisticsContainer                                                 Data Retriever         <CpuUsageStatisticsE...
Why different … ??✔   Every Request does not go through Tomcat    servlet transport (eg: ESB uses nhttp requests)✔   Some ...
Solution✔   Specifically capture the CPU time for the products    which has above constraints.✔   Separate Component to re...
How to use tenant CPU usage Statistics      Metered CPU Statistics will be summarized in BAM.      Data will be used for...
How they all fit - in …???                           er             Throttling                 st to Serv            Reque...
Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
Demo✔   CPU time for a            ✔   Remotely debug for                                  correctness     •         Sample...
Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
Problems[1] Products are different            Thread handling is done differently in some        products. Had to remotel...
Problems Continued ..[2] Retrieving data from different dependencies              Cannot add direct dependencies to ode/s...
Problems Continued ..[3] Accurate CPU Usage data ..??              Request execution live time and CPU time are very     ...
Problems Continued ..[4] Performance Hit ...??              EnableMetring is set to false by default in         carbon.xm...
Performance Comparison with Apache Jmeter                     ESB Echo Proxy Service – 1000 Samples            No of Threa...
Performance Comparison with Apache Jmeter                    ESB Echo Proxy Service – 1000 Samples            No of Thread...
Performance Comparison with Apache Jmeter          ESB Proxy Service (Class mediator) – 1000 Samples       No of Threads :...
Problems Continued ..[5] Product/version problems              Different Products used different versions of the         ...
Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
Automation Hackathon    With GREG team for almost 2 months.    Wrote a lot of test cases and ported old tests to    Clar...
Technical Knowledge
Above them all ...    Obviously learnt a load of technical things.    How to take important architectural decisions and ...
Above them all ...    How to test software, automate the functionality and    how QA functions.    How to use mailing li...
Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
Questions ..?
References    http://maharachchi.blogspot.com/2011/08/metering-    throttling-and-billing-in.html    http://sanjeewamala...
Thank you !!
Internship Project (Lasindu) WSO2
Upcoming SlideShare
Loading in...5
×

Internship Project (Lasindu) WSO2

2,201

Published on

The Presentation of the Internship Project I did in WSO2. Stratos tenant CPU Usage Metering for billing and throttling.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,201
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Internship Project (Lasindu) WSO2"

  1. 1. WSO2 Stratos - Tenent CPU usage metering Lasindu Charith Vidana Pathiranage University of Moratuwa
  2. 2. Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
  3. 3. Current implementation for tenant usage metering and billing in Stratos ✔ Number of users ✔ Disk Storage ✔ Bandwidth usage ✗ CPU Usage ???
  4. 4. What are the available CPU usage metering methods ? ● Per Hour ● Per Instance Fixed Rate ● Per User ● Per Tenant Request ??
  5. 5. Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
  6. 6. Java ThreadMXBean to rescue..● The management interface for the thread system of the JVM.● Java virtual machine implementation supports measuring the CPU time for the current thread or for any thread.● getCurrentThreadCpuTime() Returns the total CPU time for the current thread in nanoseconds.● Does not account for thread sleep/idle time.
  7. 7. Sample Code Usageif (Metering is enabled) { //check the system property (carbon.xml)if(tenant!= carbon.super && context equals “services” or “webapps”) {startCpuTime=threadMXBean.getCurrentThreadCpuTime() //get thread cpu time }}//Executable codeif (Metering is enabled) {if(tenant!= carbon.super && context equals “services” or “webapps”) {endCpuTime = Get thread Cpu timethreadCpuTime = (endCpuTime – startCpuTime)/1000000 //to millisecondsif(threadCpuTime> 0) Add cpuStatisticsEntry to Queue }}
  8. 8. CPU time of requests passing through Tomcat to ODEThread Pool Request Request BPEL Response Axis 2 Component Tomcat Valves Invoke Tomcat Servlet Transport (Job) Apache ODE Internal Thread Pool Scheduler Simple
  9. 9. ESB CPU Time … Synapse Proxy Service In Sequence Class Endpoint Mediator ServiceConsumer ServerWorker Provider Out Sequence Class Mediator ClientWorker
  10. 10. How it works ?? Thread Execution Component CpuUsageStatisticsContainer CpuUsageStatistics Retreive retrieval ComponentCpuUsageStatisticsEntries Queue Thread CPU Time per Request Send Existing (To be used for Billing Publish Stratos Usage Agent and Throttling) Component
  11. 11. Usage Agent -> BAM Publish Usage Agent Component Usage < tenantID, measurement, value > Bandwidth Database CPU Usage Usage Usage
  12. 12. Overall Architecture
  13. 13. BpsCpuUsageStatisticsContainer Data Retriever <CpuUsageStatisticsEntry> BPS Usage Agent Component <<send>> SimpleScheduler PublisherUtils ode Data Persister <<send>> Data Retriever EsbCpuUsageStatisticsContainer <CpuUsageStatisticsEntry> Data Retriever ESB Usage Agent Component ServerWorker ClientWorker Stratos Usage Agent synapse-nhttp- tranport <<retrieve>> CpuUsageStatisticsContainer New Agent Component <CpuUsageStatisticsEntry>TransportStatisticsContainer<CpuUsageStatisticsEntry> ThreadMXBeanCarbonStuckThreadDetection Valve Extensible Cpu Time Capturing Component Tomcat Ext
  14. 14. Why different … ??✔ Every Request does not go through Tomcat servlet transport (eg: ESB uses nhttp requests)✔ Some products uses their internal thread pools and thread execution mechanisms. (eg : BPS uses Apache Ode & ESB uses Apache Synapse)✔ BAM script execution is handled by a separate JVM
  15. 15. Solution✔ Specifically capture the CPU time for the products which has above constraints.✔ Separate Component to retrieve product specific CPU Usage Statistics and send them to Stratos Usage Agent Component.✔ Should add CPU Statistics to the same Usage Agent instance, once it is registered as an OSGI Service.
  16. 16. How to use tenant CPU usage Statistics Metered CPU Statistics will be summarized in BAM. Data will be used for billing and throttling. Tenants will be throttled and billed at the end of the month according to their CPU usage. Summarized Data in BAM using a Hive Script
  17. 17. How they all fit - in …??? er Throttling st to Serv Reque Agent Client Metering Data Store Mediator Agent ( Optional ) Server Usage Agent
  18. 18. Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
  19. 19. Demo✔ CPU time for a ✔ Remotely debug for correctness • Sample Web-service ✔ Summarize data at BAM • BPEL Process side • ESB Proxy Service
  20. 20. Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
  21. 21. Problems[1] Products are different  Thread handling is done differently in some products. Had to remotely debug each an every products dependent apache code (ode/synapse/hive/hadoop) and find the thread execution part and capture the CPU time of each request  Usually tenant information is not associated with each request/response in apache code. I had to send the tenant domain/id in certain cases as a parameter in the invoke method from the particular component or set it as a property so that I could find which request comes from which tenant.
  22. 22. Problems Continued ..[2] Retrieving data from different dependencies  Cannot add direct dependencies to ode/synapse in Stratos usage agent component since it is not used in every WSO2 product. I had to write new component to do the data retrieval/persistence tasks for each product, where I had to capture CPU time, except for Tomcat.ext  Had to register UsageDataPersistenceManager in usage agent as an OSGI service, so that ESB/BPS components can add the CPU usage data to the same instance that is used by the org.wso2.carbon.usage.agent components persistence queue.
  23. 23. Problems Continued ..[3] Accurate CPU Usage data ..??  Request execution live time and CPU time are very close values, but CPU time is less than the live time.  Thread sleep time is not captured as CPU time.  Thread CPU time is aggregated in ThreadMXBean. Had to take the difference of thread CPU time always for a particular request.
  24. 24. Problems Continued ..[4] Performance Hit ...??  EnableMetring is set to false by default in carbon.xml. CPU time measuring code is executed only if metering is enabled.  Tested for Tomcat.ext after metering is enabled. No noticeable change in SOAPUI for a of web service call burst.  Tested for several types of ESB proxy services with and without code from Apache Jmeter and there is no sign of change in TPS.
  25. 25. Performance Comparison with Apache Jmeter ESB Echo Proxy Service – 1000 Samples No of Threads : 100 Ramp-up period : 5s Loop Count : 10 Average Median 90% Line Min Max Error Throughput 3 3 6 2 26 0.0% 199.7/sec 3 3 6 2 18 0.0% 199.4/secWithout Code 3 2 6 2 34 0.0% 199.6/sec 2 2 6 2 32 0.0% 199.3/sec 3 2 6 2 22 0.0% 199.7/sec Average Median 90% Line Min Max Error Throughput 4 3 7 2 38 0.0% 199.1/secWith Code 3 3 6 2 21 0.0% 199.3/sec 3 3 6 2 38 0.0% 199.6/sec 3 3 6 2 21 0.0% 199.5/sec 2 3 6 2 25 0.0% 199.0/sec
  26. 26. Performance Comparison with Apache Jmeter ESB Echo Proxy Service – 1000 Samples No of Threads : 50 Ramp-up period : 5s Loop Count : 20 Average Median 90% Line Min Max Error Throughput 3 3 4 2 19 0.0% 199.0/sec 3 3 4 2 13 0.0% 198.8/secWithout Code 3 3 4 2 21 0.0% 197.5/sec 2 3 4 2 12 0.0% 199.0/sec 2 3 3 2 19 0.0% 199.3/sec Average Median 90% Line Min Max Error Throughput 4 3 7 2 29 0.0% 196.9/secWith Code 4 3 7 2 22 0.0% 199.3/sec 4 3 7 2 21 0.0% 199.5/sec 3 3 6 2 22 0.0% 198.1/sec 2 3 3 2 17 0.0% 199.2/sec
  27. 27. Performance Comparison with Apache Jmeter ESB Proxy Service (Class mediator) – 1000 Samples No of Threads : 100 Ramp-up period : 1s Loop Count : 10 Without With Code Code 760.5/sec 766.8/sec 762.2/sec 749.1/sec 754.1/sec 746.8/sec 745.2/sec 746.3/sec 751.9/sec 751.3/sec 763.4/sec 748.5/sec 764.5/sec 757.6/sec 753.6/sec 749.6/sec– Checked out for several types of Proxy services at the same time and total throughput seems to be quite even.
  28. 28. Problems Continued ..[5] Product/version problems  Different Products used different versions of the same component  While project goes on several changes to dependent components occurred
  29. 29. Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
  30. 30. Automation Hackathon With GREG team for almost 2 months. Wrote a lot of test cases and ported old tests to Clarity framework. Learnt on Greg LCs, Rxts, APIs, URIs, Handlers, Permissions etc. Learnt to writie axis2 clients to test CRUD Operation support and Discovery Proxy for GREG. Automated several Support Patches.
  31. 31. Technical Knowledge
  32. 32. Above them all ... Obviously learnt a load of technical things. How to take important architectural decisions and flexibility of carbon architecture. How to Communicate ideas with others and get the necessary help. Was able to get the help of lot of people and work in several products Carbon, AS, ESB, BPS, GREG, BAM, DSS, Stratos etc. Learnt best practices in software engineering and coding conventions.
  33. 33. Above them all ... How to test software, automate the functionality and how QA functions. How to use mailing lists effectively. How to manage time and meet deadlines. How does a company function and how a company prepares for a release. Got to know a bunch of good friends/people. Enjoyed every minute of it.
  34. 34. Outline1. Requirements2. Implementation3. Demo4. Problems5. Things Learnt6. Q & A
  35. 35. Questions ..?
  36. 36. References http://maharachchi.blogspot.com/2011/08/metering- throttling-and-billing-in.html http://sanjeewamalalgoda.blogspot.com/2011/08/wso2- stratos-usage-and-throttling_22.html http://wso2.org/library/articles/2011/11/usage-metering- cloud-environment-using-wso2-stratos http://docs.oracle.com/javase/6/docs/api/java/lang/manage ment/ThreadMXBean.html http://jmeter.apache.org
  37. 37. Thank you !!

×