IWSM Presentation

1. Performance
measurement for cloud
computing applications
using ISO 25010
standard characteristics
Anderson Ravanello, Jean-Marc Desharnais, Luis Eduardo Bautista Villalpando,
Alain April, Abdelouahed Gherbi
ravanello@gmail.com, jean-marc.desharnais@etsmtl.net, luis.bautistav@gmail.com,
alain.apri@etsmtl.ca, Abdelouahed.Gherbi@etsmtl.ca

2. Background of cloud computing
 Cloud computing is an emerging technology.
 This technology is being broadly adopted by the
industry as means to achieve mobility, reduced costs
and ubiquity. (Voas and Zhang, 2009)
 One of the most important challenge in delivering
Cloud Services is to ensure that they are fault tolerant
(Bautista et al., 2013)
 The system performance is unreliable due to the
complexity of the infrastructure

3. Characteristics of cloud computing
• Cloud computing is expanding
• Measuring performance for this infrastructure is
complex
• Measuring performance from log data involves
evaluating large amounts of data
 Rapid processing of query results to user is important
(ex. Google) and is a part of the performance

4. Objective of this research
 The main objective of this research is to show how
base and derived measures can be map to reveal the
performance of a cloud-based application
 This will be tested in the context of a large Microsoft
Exchange application installed in a private cloud and its
distributed clients (10000 servers around the world)
 To take in account the complexity of the infrastructure we
will implement the measurement framework developed by
Bautista. This measurement framework is using quality
characteristics of ISO 25010 (ISO SQuaRE series)

5. Background: comparative complexity of standard computing X client
computing
Client server (simpler)
(IBM, 2013)
Cloud Computing (more complex)
(Lemay, 2012)
Figure 1: Comparative complexity between client server and cloud computing infrastructure

6. Background: The studied private cloud
 Our case study was conducted on commercial
software running on the CC infrastructure of a
private cloud that mainly hosts and enables
access to a company’s email services.
VM host
Lan
DNS
Firewall
DMZ
WAAS Router
Verizon MPLS
Backbone
DMZ
Router WAAS
Firewall
Lan
AD
Unix Filer
Virtualization server
CAS Mbx/db
Figure 2: Studied private cloud

7. Performance Measurement Framework
 To achieve a performance measurement for this private
cloud, we faced 2 challenges:
1. How to determine the performance criteria?
We choose a limited number of characteristics and sub
characteristics from ISO 25010 (e.g. time behavior)
2. How to choose and link the 'measures' to the sub
characteristics and characteristics?
Choice of the 'measures' from the logs generate by the
nodes and apply to a specific characteristics (see
methodology)

8. Methodology
1. Data collection: automated from the performance logs
generated by the 12 nodes (Figure 2). In this presentation
only 2 were used (CAS, MBX/DB
2. Focus on the time behavior from ISO 25010 characteristics
and sub-characteristics and apply each 'measures' to the
pertinent sub characteristics
3. Data organization: physical location (North America,
Europe, Asia) and day time (business and non-business
hours)
4. Data analysis: statistical analysis results (averages,
variances, kurtosis, and skewness), and the results of a
visual examination of the time behavior graphs (see radial
graph, figure 4)

9. Data collection
 The data collected from the logs of two nodes are mainly
'low level derived measures' (*).
 The total private cloud is composed of approximately 80,000
client machines and 10,000 servers and network devices. In
this study we collected data from 10,000 servers and display
the data of 12 servers for the duration of 1 week – this
section of data is 600 MB and is represented on figure 6.
 With the total number of clients and servers, around
800,000 data points per minute are generated. To visualize
this situation, imagine a spreadsheet which grows by
800,000 lines of data every minute, with each line made up
of approximately 1,000 characters.
(*) There are 159 low level derived measures. Low level derived measures are the most atomic and
granular measures that are available in operational systems from this Cloud Computing Application.

10. Measurement of Time Behaviour
 With the data available in this case study, we were able to
assess the time behaviour quality characteristic via the
email service usage low level derived measures presented
partially (only 3 - see table) for the transmission function:

11. Measurement and Bautista framework
 Bautista suggest a number of derived measures based
on different characteristics.

12. Time behaviour and Bautista framework
 It is possible to reverse the previous table using Time behaviour characteristics:
Time
behaviour
Task function
Task executed
Task passed
etc.
Time function
Down time
Maximum
response time
etc.
Transmission
function
Transmission
error
Transmission
capacity
etc.

13. Data organisation
 Three physical locations (North America, Europe, and Asia),
 Processed over a 1 week period during business and non
business hours.
 A sub grouping of the data was created per host status
(machine level) to allow us to compare usage on a machine
by machine basis.
 Finally a data grouping is necessary to analyse the results.
For example, base statistics for one measure is divide in
business hours (ON) vs. non business hours (OFF). Will be
presented in the analysis of the data next.

14. Results analysis
 Analysis of the data (comparison)
 Visual interpretation (global)
 Analysis: single point measure and index creation

15. Analysis of the data
Web Service Bytes Sent/sec _Total
(On hours)
Web Service Bytes Sent/sec _Total
(Off hours)
Mean 43 KBps Mean 28 KBps
Standard Error 1.5 KBps Standard Error 979.83
Median 20 KBps Median 18 KBps
Mode #N/A Mode #N/A
Standard Deviation 84 KBps Standard Deviation 35 KBps
Sample Variance 7GBps Sample Variance 1 GBps
Kurtosis 27.77 Kurtosis 49.97
Skewness 5.01 Skewness 5.73
Range 821 989.44 Range 428426.46
Minimum 2KBps Minimum 3.1 KBps
Maximum 829 MBps Maximum 431MBps
Sum 124978002.8 Sum 37266022.34
Count 2857 Count 1309
Largest (5) 691MBps Largest (5) 343MBps
Smallest (5) 3 MBps Smallest (5) 3.2 KBps
Confidence Level
Confidence Level
(95.0%) 3 MBps
(95.0%) 1.9MBps
Example of the statistical data calculated (one characteristic, business VS off hours)

16. Analysis: Visual interpretation
LOW LEVEL DERIVED MEASURES (LLDM)
System Processor Queue Length
Web Service Bytes Sent/sec _Total
Web Service Connection
Attempts/sec _Total
Web Service Current Connections
_Total
Figure 4: Four LLDM, 8am to 6pm. Higher numbers mean more resources used.

17. Analysis: single point measure and index creation
DP Multifunction Gigabit
Server Adapter _51
Network Interface Bytes
Received/sec HP NC382i
DP Multifunction Gigabit
Server Adapter _52
Network Interface Bytes
Sent/sec HP NC382i DP
Multifunction Gigabit
Server Adapter _51
Network Interface Bytes
Sent/sec HP NC382i DP
Multifunction Gigabit
Server Adapter _52
Network Interface Bytes
Total/sec HP NC382i DP
Multifunction Gigabit
Server Adapter _51
Network Interface Bytes
Total/sec HP NC382i DP
Web Service Current
Connections _Total
Web Service Connection
Attempts/sec _Total
Web Service Bytes
Sent/sec _Total
System Processor Queue
Length
Figure 5: Time behavior at a single moment in time
Figure 5 is the measure of a single point in time; on this second, all these
LLDM had different values. If we calculate the area of this figure, we get to a
value that could be an indicator of the LLDM of that second.
This process was then replicated to all the data points, leading to the graph
on figure 6 (next)

18. 3.00E+06
Analysis: Index Creation
On this figure, we see peaks on the 11th and 13th, on different hours; these are
moments on the equivalent area of the radial figure would be bigger, indicating
a possible degraded performance for the whole system. The inverse is valid for
the end of the week, where the index is lower.
Figure 6: Evolution of the time behavior index
0.00E+00
5.00E+05
1.00E+06
1.50E+06
2.00E+06
2.50E+06
3.50E+06
Time
2/10/14 10:00 AM
2/10/14 10:00 AM
2/10/14 1:00 PM
2/10/14 1:00 PM
2/10/14 4:00 PM
2/10/14 4:00 PM
2/11/14 8:00 AM
2/11/14 8:00 AM
2/11/14 11:00 AM
2/11/14 11:00 AM
2/11/14 2:00 PM
2/11/14 2:00 PM
2/11/14 5:00 PM
2/11/14 5:00 PM
2/12/14 9:00 AM
2/12/14 9:00 AM
2/12/14 12:00 PM
2/12/14 12:00 PM
2/12/14 3:00 PM
2/12/14 3:00 PM
2/12/14 6:00 PM
2/12/14 6:00 PM
2/13/14 10:00 AM
2/13/14 10:00 AM
2/13/14 1:00 PM
2/13/14 1:00 PM
2/13/14 4:00 PM
2/13/14 4:00 PM
2/14/14 8:00 AM
2/14/14 8:00 AM
2/14/14 11:00 AM
2/14/14 11:00 AM
2/14/14 2:00 PM
2/14/14 2:00 PM
2/14/14 5:00 PM
2/14/14 5:00 PM
2/15/14 9:00 AM
2/15/14 9:00 AM
2/15/14 12:00 PM
2/15/14 12:00 PM
2/15/14 3:00 PM
2/15/14 3:00 PM
2/15/14 6:00 PM
2/15/14 6:00 PM
2/16/14 9:00 AM
2/16/14 9:00 AM
2/16/14 12:00 PM
2/16/14 12:00 PM
2/16/14 3:00 PM
2/16/14 3:00 PM
2/16/14 6:00 PM
Index

19. Conclusion and future research
 Demonstrated the possibility of using the framework to
measure quality characteristic
 Challenges regarding data collection, data processing
and data representation
 Future research: improved statistic techniques, larger
time frame, different quality measures, real time
processing.