Overview of Performance Evaluation Intro & Objective The Art of Performance Evaluation Professional Organizations, Journals, and conferences. Performance Projects Common Mistakes and How to Avoid Them Selection of Techniques and Metrics

1. AN OVERVIEW OF PERFORMANCE
EVALUATION & SIMULATION

2. OVERVIEW OF PERFORMANCE
EVALUATION
 Intro & Objective
 The Art of Performance Evaluation
 Professional Organizations, Journals,
and conferences.
 Performance Projects
 Common Mistakes and How to Avoid
Them
 Selection of Techniques and Metrics

3. WHY WE NEED TO SIMULATE?
3
 It may be too difficult, risky, or expensive
to observe a real, operational system
 Parts of the system may not be
observable (e.g., internals of a silicon
chip or biological system)

4. USES OF SIMULATIONS
 Analyze systems (performance, behavior)
before they are built
 Reduce the number of design errors
 Optimize design to improve the behavior
 Analyze operational systems
 Create virtual environments for training,
entertainment

5. APPLICATIONS OF SIMULATION
5
 System Analysis
 Telecommunication Networks (ATM, IP, TCP, UDP, WiFi …)
 Transportation systems (Traffic, Urban planning, Metro Planning, …)
 Electronic systems (e.g., microelectronics, computer systems)
 Battlefield simulations (blue army vs. red army)
 Ecological systems, Manufacturing systems, Logistics …
 Virtual Environments
 Physical phenomena (e.g. Trajectory of projectiles)
 training and entertainment (e.g., military, medicine, emergency
planning, flight simulation)

6. A FEW EXAMPLE APPLICATIONS
War gaming: test
strategies; training
Flight Simulator Transportation systems:
improved operations; urban
planning
Computer communication
network: protocol design
Parallel computer systems:
developing scalable software
Games

8. INTRO & OBJECTIVE
 Performance is a key criterion in the
design, procurement, and use of
computer systems.
 Performance  Cost
 Thus, computer systems professionals
need the basic knowledge of
performance evaluation techniques.

9. KEYWORDS
 System
 It is a collection of entities that act and interact together
toward the accomplishment of some logical end
(computer, network, communication systems, etc.)
 Simulation
 It is an experiment in a computer where the real system is
replaced by the execution of the program
 It is a program that mimics (imitate) the behaviour of the
real system

10.  Model
 It is a simplification of the reality
 A (usually miniature) representation of something; an
example for imitation or emulation
 A model can be Analytical (Queuing Theory) or by
Simulation.
 Performance Evaluation of a System means quantifying the
service delivered by the System
 Experimental, Analytical, or by simulation
Keywords

11. EXAMPLE
11
Real System (Motherboard)
Models of the System

12. EXAMPLE
12
Simulation Models of the System

13. EXAMPLE
13

14. EXAMPLE
14
Models of the System

15.  Why to use models?
 Implementation on real systems is very complex and costly,
 Experimentation on real systems may be dangerous (e.g.
chemical systems)
 If models adequately describes the reality, experimenting with
them can save money and time, and reduce the development
complexity
 When to use simulations?
 Analytic models may be very complex to evaluate, and may lead
to over implication of the real system
 Simulation can be a good alternative to evaluate the system
behavior very close to reality
Why using Models and Simulations?

16. INTRO & OBJECTIVE
 Objective:
1. Select appropriate evaluation
techniques, performance metrics and
workloads for a system.
2. Conduct performance measurements
correctly.
3. Use proper statistical techniques to
compare several alternatives.
4. Design measurement and simulation
experiments to provide the most
information with least effort.
5. Perform simulations correctly.

17. MODELING
 Model – used to describe almost any
attempt to specify a system under study.
Everyday connotation
– physical replica of a system.
 Scientific – a model is a name given to a
portrayal of interrelationships of parts of
a system in precise terms. The
portrayal can be interpreted in terms of
some system attributes and is
sufficiently detailed to permit study
under a variety of circumstances and to
enable the system’ s future behavior to
be predicted.

18. A TAXONOMY OF MODELS
 Predictability
 Deterministic – all data and relationships
are given in certainty. Efficiency of an
engine based on temperature, load and
fuel consumption.
 Stochastic - at least some of the
variables involved have a value which is
made to vary in an unpredictable or
random fashion. Example – financial
planning.
 Solvability
 Analytical – simple
 Simulation – complicated or an
appropriate equation cannot be found.

19. A TAXONOMY OF MODELS
 Variability
 Whether time is incorporated into the
model
 Static – specific time (financial)
 Dynamic – any time value (food cycle)
 Granularity
 Granularity of their treatment in time.
 Discrete events – clearly some events
(packet arrival)
 Continuous models – impossible to
distinguish between specific events taking
place (trajectory of a missile).

20. COMPUTER SIMULATION
20
 A Computer Simulation is a computer program that:
 attempts to simulate an abstract model of a particular
system.
 describes the behavior of a real (physical) system and its
evolution in time
 How it works?
 The behavior of the system is described by state variables
 The simulation program modifies the states variables to
emulate the evolution

21. PERFORMANCE EVALUATION
21

22. PERFORMANCE METRICS
22
 The Performance Metric is a measurable quantity that
precisely captures what we want to measure (response time,
throughput, delay, etc.).
 For example, In computer systems, we might evaluate
 The response time of a processor to execute a given
task.
 The execution time of two programs in a multi-processor
machine.
 In Network systems, we might evaluate
 The (maximum/average) delay experienced by a voice
packet to reach the destination
 The throughput of the network
 The required bandwidth to avoid congestion

23. WHAT DOES AFFECT THE
PERFORMANCE?
23
 The performance of a system is dramatically affected by the Workload
 The Workload: it characterises the quantity and the nature of the system
inputs
 In the context of Web Servers, system inputs are http requests (GET
or POST requests). The workload characterises
 the intensity of the requests: how many requests are received by
the web server. High intensities deteriorate the performance.
 The nature of the requests: the request can be simple GET
request or a request that require the access of a remote
database. The performance will be different for different request
types.
 Benchmarks: used to generate loads that is intended to mimic a
typical user behaviour.

24. HOW TO PROCEED?
I hear and forget. I see and I remember. I do and I
understand – Chinese Proverb

25. PERFORMANCE PROJECTS
 The best way to learn simulation is to apply the
concepts to a real-system
 The project should encompass:
 Select a computer sub-system : a network
congestion control, security, database, operating
systems.
 Perform some measurements.
 Analyze the collected data.
 Simulate AND Analytically model the subsystem
 Predict its performance
 Validate the Model.

26. PROFESSIONAL ORGANIZATIONS, JOURNALS
AND CONFERENCES
 ACM Sigmetrics : Association of Computing
Machinery’s.
 IEEE Computer Society – The Institute of Electrical and
Electronic Engineers (IEEE) Computer Society.
 IASTED – The International Association of Science and
Technology for Development

27. COMMON MISTAKES AND HOW TO AVOID THEM
1. No Goals
2. Biased Goals
3. Unsystematic Approach
4. Analysis without understanding The Problem
5. Incorrect Performance Metrics
6. Unrepresentative Workloads
7. Wrong Evaluation Techniques
8. Overlooking Important Parameters
9. Ignoring Significant Factors

28. COMMON MISTAKES AND HOW TO AVOID THEM
10. Inappropriate Experimental Design
11. Inappropriate Level of Detail
12. No Analysis
13. Erroneous Analysis
14. No Sensitivity Analysis
15. Ignoring Errors in Input
16. Improper Treatment of Outliers
17. Assuming No Change in the Future
18. Ignoring Variability

29. COMMON MISTAKES AND HOW TO AVOID THEM
19. Too Complex Analysis
20. Improper Presentation of Results
21. Ignoring Social Aspects
22. Omitting Assumptions and Limitations.

30. A SYSTEMATIC APPROACH
 State Goals and Define the System
 List Services and Outcomes
 Select Metrics
 List Parameters
 Select Factors to Study
 Select Evaluation Technique
 Select Workload
 Design Experiments
 Analyze and Interpret Data
 Present Results