(Transcripts) Reza FARRAHI MOGHADDAM, Fereydoun FARRAHI MOGHADDAM, Vahid ASGHARI, Mohamed CHERIET Presented in Network of Future (NoF’12), Nov 21st-23rd, 2012, Tunis, Tunisia

1. This is a report on research directions and progress of Behavior Analysis Technical
Group (BATG) of Synchromedia Lab, ETS.
BATG has a focus on cloud-based Telco solutions.
This report provides our vision with respect to behavior analysis in computing
systems, and also the elements of our proposed framework to address this need.
1

2. We start with discussing the necessity of behavior analysis (BA), especially its
application in failure prediction.
Then, we discuss the high-level structure and design of the proposed BA
framework.
Next, the three sub-paradigms that constitute the main framework will be presented
with examples.
After that, some of main features of the framework, such as its ability to upgrade a
service, and also its intrinsic scalability along various dimensions will be discussed.
Finally, the conclusions and also future prospects will be provided.
2

3. Why Behavior Analysis (BA)? There are many applications that can be improvised
using a BA solution. We discuss two of them here which are more related to our
applications.
First application is Failure Prediction. Failures are parts of all systems. Components
will low failure rates are expensive and also have a big environmental footprint. At
the same time, most of Service Layer Objects (SLOs) in the SLAs are at the service
layer i.e., the highest system level. Therefore, there is a big opportunity for BA in
order to intervene on component and sub-system failures (faults), and isolate them
before surfacing to the service layer. Failure prediction brings this to a higher level
in which the intervention can be started even before the occurrence of a fault (at the
sub-system level). We will discuss this more later on.
The second emerging application of BA in the computing systems is Profiling of
Actors. This application is more relevant to ecosystems of computing systems. In
these ecosystems, for example a telecom business, the actors ranging from the end
users (clients) to service providers and big computing/network/content (CNC)
providers (such as operators) have a high degree of freedom, and therefore they
cannot be considered as components of a system. Therefore, an ecosystemic
picture is required. BA could be used in profiling of actors in these ecosystems in
order to achieve smarter management of scarce resources, build reputations for
actors that can be used in their interactions, and also spot and isolate those actors
whose behavior is a threat to the others, among other applications.
3

4. As said before, failure are always part of a system. Usually, component-level
failures are considered as faults. However, in this work, because of high complexity
of systems and high degree of ambiguity in definition of borders, we may call all
faults and failures as failures, and then they are categorized based on their
corresponding level in the system and also their relation or impact on the SLAs.
There are many failure sources. In recent years, thanks to advanced manufacturing,
there is a big shift in weight from hardware-sourced failures toward software-
sourced and also human-sourced failures.
However, at the same time, there is a rebounding effect because of various
attempts to exploit operation in previously-not-allowed regimes of hardware. One
major example is as follows:
An interesting source of failure which is very close to our objective of making green
and low-footprint computing systems is an emerging approach in system operation,
which we call Cyclic ElastoPlastic Operation (CEPO). It seems that CEPO can bring
a huge opportunity for energy and cost saving, and also GHG emission reduction.
However, at the same time, CEPO introduces higher rates of failure in the systems.
BA can play a central role in incorporation of the CEPO approach in computing
systems and especially datacenters by providing an opportunity to mitigate side
effects of CEPO. Using CEPO, for example, datacenters could operate at higher
temperature that reduces the cooling costs and their Power usage effectiveness
(PUE) (lower PUE is better).
4

5. Cyclic elastoplastic operation (CEPO) is well-known in Mechanical and Civil
engineering. It enables structures to operate safely in their plastic regimes, which
are traditionally considered as regimes of no return. The plastic regimes are beyond
the yield point, which can be considered as the bearable stress level beyond which
the stress is not linearly related to the imposed strain.
Working in plastic regimes has the benefit of increasing the load capacity of
structures, and therefore reducing a lot of capital and operational costs, and also
environmental footprints.
The challenge is that the operation in plastic regimes cannot be guaranteed
because the system may eventually reach the plastic collapse point. Estimating an
accurate collapse point is very difficult, and at the same time, micro-level collapse
points can be reached even when operating far from the macro-level collapse point.
Therefore, the duration of stay in the plastic regime is very important and should be
limited.
CEPO provides an answer to these problems by cyclic displacement of the system
state between elastic and plastic regimes. It not only controls the stay interval in the
plastic regime, it can also prevent permanent deformations in the structure.
5

6. The same concept of Cyclic ElastoPlastic Operation (CEPO) can be applied to computing
systems. We consider two examples.
First, consider a data center and its cooling system. Traditional recommendations put a
very restricted allowable range for the inlet air temperature. However, it has been observed
that with each 1°C increase in the temperature, we can expect 2 to 4 percent energy
saving. This can simply lead to an upgrade in the PUE of a data center from 1.5 to 1.2 just
by increasing the inlet air temperature by 5 degrees. At the same time, the fans and their
speed should be controlled in order to avoid any rebound effect.
The drawback of this technique is an increase in the failure rate. As reported by American
Society of Heating, Refrigerating and Air Conditioning Engineers (ASHRAE), figure below
taken from The Green Grid White Paper #50, there is an increase of 50% in the failure rate
if the inlet temperature is increased from 20 degrees to 33 degrees.
However, CEPO can bring a solution here. With limited stay intervals in the plastic regime
(high inlet air temperature) and cyclic operation, not only the failure rates could be
significantly reduced, the life expectancy of servers can be protected.
BA can play a big role in CEPO application as it can provide the optimal cycling timings
based on predicted failures. Although it should consider an optimal takeoff-landing
procedure in order to avoid triggering failures because of variations in the temperature.
6

7. Here, the high-level structure of the proposed behavior Analysis framework is
provided.
Because of high complexity of computing systems, an Ensemble-of-Experts
approach is considered. In this ensemble, three sub-paradigms analyze the system,
and provide insights on its behavior and trends. Then, a decision making is made
depending on the desired degree of dependability and reliability of the system.
Three sub-paradigms are:
Probabilistic Behavior Analysis
Simulated Probabilistic Behavior Analysis
Behavior-Time Profile Modeling and Analysis
The detailed design of the framework is provided in the following slides.
It is worth noting that the framework can work in two different pictures depending on
the target: System or Ecosystem pictures.
7

8. The systemic picture is provided here.
The framework constitutes of three main units and a collection of opportunistic
agents.
The three main units are: BAU, BSU and CRU
Behavior Analyzer Unit (BAU) is the core of the framework, and the three sub-
paradigms, data analyzers, and models are located and developed there. As
machine learning technique are used to develop the behavior models, BAU
requests specific behaviors from Behavior Stimulator Unit (BSU) to be injected into
the system. These behaviors could also be failure behaviors. The decision making
and also cognitive responses to the predicted or observed abnormal behaviors is
performed in the Cognitive Responder Unit (CRU).
As can be seen from the diagram, the framework consider the system in many
layers from hardware (including the network and access parts), to middleware,
virtualware and high-level applications. The state of components at each layer is
opportunistically collected by the BA agents of the framework. In this way, a large
amount of not-necessary big data is filtered by the agents, and the core part of
states is uploaded to BAU.
8

9. As said before, the framework also works in a ecosystemic picture to handle
behavior analysis of ecosystems of actors. The role of the behavior analysis
framework is more on profiling the actors, not the state of their components. The BA
framework can be used in its systemic picture to analyze the components.
In an ecosystem, the end users officially interact with high-level service providers,
while they actually interact with computing providers (including
computing/networking/content (CNC) providers) that host the higher-level service
providers. The framework not only creates dynamic profiles of the actors, it identify
changes in role of actors in the form of transferring from one class to another one.
The three units of the framework are: Actor Simulation Unit (ASU), Behavior
Analyzer Unit ( BAU), and Cognitive Advisory Unit (CAU)
ASU creates imaginary (fake) actors in the end user and also service provider
classes upon the request from the BAU in order to create desired use cases in
order to profile other actors.
BAU collects information from the CNC providers, and gives the results to CAU
CAU provides advisory suggestions to the computing providers based on
Ensemble-of-Experts analysis, and could possibly advise the service providers.
There is no guarantee that its advises are accepted.
9

10. Once more, we want to emphases on the multi-layer structure of the framework. It
considers layers from pure hardware (including both computing and networking
components) to middleware and virtualware, and finally to highest level of
applications.
The components in each layer, although will be analyzed separately in a multi-level
analysis of that layer, are imaginary linked to the components on other layers via
their physical and non-physical location information. This opens a new approach to
location intelligence in computing systems.
10

11. The first sub-paradigm considered in the BA framework is based on statistical
inference. Assuming that the PoA estimation is core of calculations, each layer is
analyzed in a multi-level graph representation and analysis.
The PoA of a sub-graph can be calculated based on its Cumulative Distribution
Function (CDF) of failure on an time interval [0, t0]. If we assume all components
are at their fully-maintained state at t = 0, the interval can be represented by t0.
Also, in order to identify similar CDFs, a Differential Density Function (DDF) is
introduced. Using the scaling parameter, s, many similar CDFs can be represented
by only one DDF, and this can reduce the complexity of calculations.
11

12. Here, an example of a 2-component system is provided. The CDF of the system is
calculated based on the CDFs of its components. Also, the DDF is provided. If two
components are the same, the CDF and DDF of the system are simplified.
12

13. In order to achieve a better representation of empirical data and experiments, a new
distribution function, Tanh distribution function is introduced.
The formula of CDF and DDF of tanh distribution are provided. Also, the profiles of
CDFs and DDFs of 1-component and 2-component systems using this distribution
are plotted.
13

14. To validate the performance of the Tanh distribution, a real database, lanl05
database, is used. The database has been retrieved from the Failure Trace Archive
(FTA). The union interpretation of the data is used in the calculations. The database
consists of 19874 availability records over 9 years. Two Goodness of Fitness (GoF)
tests, Kolmogorov-Smirnov and Anderson-Darling, were used. The Tanh distribution
showed a high p-value for the rejection hypothesis, even higher than those of the
Weibull distribution. The better goodness of fitness can also be seen from the plot.
The absolute errors to the empirical values in percentage are also shown that again
show the better performance of the proposed Tanh distribution.
14

15. When the complexity of the sub-graphs is very high, and also when the target
availability is not the full availability, statistical inference using simulation can
achieve the required results faster.
Also, the simulated results can be used to validate the analytical results.
The core of the simulation paradigm is the Monte-Carlo (MC) method.
In each run of the MC method, for each component, the fault occurrence time is
calculated based on its CDF. Then, according to the required availability (full, partial,
etc), the failure time of the sub-graph or sub-system is calculated. Using these
statistics, the CDF of the sub-system can be calculated.
1000 runs have been used to build the statistics in each case.
15

16. An example, in which the simulation paradigm is used to calculate the CDFs of 1-
compoment and two-component sub-graphs. The analytical plots are also provided
for the purpose of comparison.
16

17. An example, in which the CDF of a complex 5-component sub-graph is easily
calculated using the simulation paradigm. Also, the DDF of this 5-component sub-
graph is presented on the right-hand side.
17

18. In our approach to time-profile analysis, time profiles of various characteristics of
components at various levels and layers are collected.
In order to reduce the volume of collected “big data” , opportunistic agents are
considered as collectors-on-site in order to “sample” the most important part of data,
and then upload it to BAU.
In BAU, various machine learning methods, especially SVMs, are considered to
learn and model the behavior.
18

19. The third sub-paradigm, as said before, focuses on temporal cause-and-effect
relations in the systems.
Although many high-level faults are actually effects of low-level causing faults, there
are other types of cause-and-effect faults that are not initialed by a low-level fault. In
other words, cumulation, modulation and amplification of “normal” behaviors and
actions at low-level can lead to “faulty” behaviors and events at higher levels of a
complex system.
Therefore, temporal analysis of behavior profiles is of great importance.
As the result, many of failures are more related to the business and operation of the
system, and therefore it is expected to have more patterns and cycles in the failure
behavior, because business operations usually follow some sort of cycling.
The case of faulty systems without faulty components can be connected to context-
sensitive analysis. Again, context can identified easier when temporal behavior is
considered.
And finally, there is a need to handle gradual events (including fault events). Highly-
complex systems require abstraction at a computable level of details. Therefore,
even the lowest-level components may have a considerably high level of complexity,
and their state space cannot be approximated with a few (discrete) number of
states. For example, IC components themselves could constitute of millions of
microelectronic sub-components.
19

20. A simple example of the application of this sub-paradigm is shown in this slide. The
system is composed of two servers, from each one two profiles (CPU Load and
Allocated Memory) are collected.
As can be seen a gradual fault event of memory leakage starts from 9:10 in the
second system. The BAU detects this event at 9:45, and the cognitive response of
CRU in the form of adding an additional server prevents a SLA violation at 10:00.
20

21. Here, we discuss the promise of the BA framework in upgrading services. This is a
key aspect in enabling emerging technologies, which may have been ignored or will
be ignored in sensitive applications with high grade requirements, such as telco
grades. For example, the telco grade of availability requires 5nines (less than 5:15
minutes downtime a year) grade. Many of these emerging technologies, such as
cloud computing, may not have the required grade level in their basic operation.
However, behavior analysis could upgrade them even without major changes in
their architecture and basic operation.
Also, it is worth pointing out that there are two major approach categories of
business models. The first is leased-based businesses, such as Infrastructure as a
Service (IaaS). In these businesses, SLOs can be expressed in terms of availability
of resources. Therefore, Probability of Availability (PoA) could be a good target
variable in the analysis.
In the second category, task completion is the target, and therefore Probability of
Completion (PoC) should be considered.
Here, as an example, the impact of BA intelligence on the grade of a system is
shown. A 4nines system is upgraded to the 5nines grade thanks to predicting 90%
of failures before their occurrence. This could be achieved without any direct
investment in upgrading the system components.
Glossary: Mean Time Between Failures (MTBF), Mean Time To Repair (MTTR)
21

22. A simple example of a computing system is shown on left side. As can be seen, in
order to have a uniform approach to all aspects of the computing systems, the
network links are also considered as components (shown as green ovals).
As discussed CRU uses cognitive approaches in responding to failure events. One
approach is to respond before a failure occurred, or even before a fault occurred.
The second one, i.e. responding even before a fault, can be achieved by
recruiting/retiring resources, in such a way that they have some period of time to
recover (via self-healing or intervention). An example, of this technique is shown on
right side where the connectivity of two servers is assured all the time by two
alternating modes. This also helps in footprint reduction because some of
components are not in circuit all the time. The BA framework play the key role here
to determine the timing of alternation and recruit/retire cycles.
22

23. The behavior analysis framework can scale on three different dimensions. The
horizontal scaling powered and absorbed by vertical scaling via various levels of
sub-graph and sub-system abstraction. This helps in feasible analysis of highly
complex systems.
On another dimension, the framework can easily scale along the platform
dimension. An example, of scaling from cloud platform to sky platform is shown on
the right-hand side. The platform scalings can be hierarchical or federal. In either
cases, a Recapitulator receives the behavior state and profile of a low-level system
and recapitulates it in low-volume, low-dimensional data which will be hierarchically
transmitted to the higher-level BA units (in this examples, the sky-level units), or
federally shared among the systems at the same level. A skybus is considered in
this example to facilitate the exchange of the behavior data.
23

24. A generic behavior analysis framework introduced:
Multi-expert: three sub-paradigms
Multi-layers
Multi-level
Scalable: horizontal, vertical, platform
New Tanh distribution. Applied to a read database.
Future prospects:
Testing the sub-paradigms at scale.
CRU modeling.
Integration with real systems.
The time-profile analysis sub-paradigm is one field for new methods and models.
More sophisticated distributions will be considered.
24

25. The BATG members
The Synchromedia website
The sponsors: NSERC
25