Feedback Driven Development

08.09.16 | Dept. of CS – Software Technology Group | Technical University of Darmstadt | Harini Gunabalan |
Harini Gunabalan
Master Thesis, Distributed Software systems
Feedback Driven Development of
Cloud Applications
1
Execution Analytics Coordinated Adaptation Feedback

Motivation
2

Motivation
Cloud
3

Motivation
Cloud
Developer
Operator
End Users
4

Motivation
Cloud
Developer
Operator
End Users
App App
Deploy app
5

Motivation
Cloud
Developer
Operator
End Users
App App
Deploy app
VMVM VM VM
Provision resources
6

Motivation
Cloud
Developer
Operator
End Users
App App
Deploy app
VMVM VM VM
Provision resources
Use App
7

Cloud
Developer
Operator
End Users
App App
Deploy app
VMVM VM VM
Provision resources
Use App
8
Not
working!
Motivation

Motivation
Cloud
Developer
Operator
End Users
App App
Deploy app
VMVM VM VM
Provision resources
Use App
Where is the problem?Not
working!
9

Cloud
Developer
Operator
End Users
App App
Deploy app
VMVM VM VM
Provision resources
Use App
Where is the problem?
Maybe, I
should
improve
code?
Not
working!
10
Motivation

Cloud
VMVM VM VM
Provision resources
App
End Users
App
Use App Developer
Operator
Maybe, I
should
improve
code?
Is the
resource
provisioning
not sufficient?
Deploy app
Where is the problem?Not
working!
11
Motivation

Problem Statement
12

Problem Statement – APM Information
13

Problem Statement - FDD
14
Automating Resource
provisioning Using the
Run-time information
Identifying Source
Code Issues
What is Feedback Driven Development?

Problem Statement – Auto-scaling
15
Automating Resource
provisioning Using the Run-
time information
Problem:
Automate Resource
Provisioning
Solution:
Auto-scaling
What is Auto-scaling?
On-demand scale out and
scale in depending on
Load.

System Design
16

Architecture
17
• 3 components
Ø Cloud
monitoring
Ø Scaling-
service
Ø Data modeling
• Platform level
autoscaling

Implementation
22

System Footprint
23

Component 1: Cloud Monitoring
24
Key metrics monitored
• Average response time (CF Logs)
• Number of incoming requests (CF
Logs)
• Number of running app instances
(CF API)
• CPU utilization (CF API)
• Memory utilization (CF API)
• Disk utilization (CF API)

• Model data to identify correlation between metrics
• Most real world data are Multi-dimensional.
• Multiple-input and Multiple-output (MIMO) models.
Ø State-space Model
Ø Polynomial Models
Ø ARX
Ø ARMAX
Component 3: Data Modeling
26

State Space Models
• Represented by differential equations and state variables.
• Output can be predicted for any future time provided the input, output,
and a minimum set of state variables xi(t), are known.
dx/dt = Ax + Bu
y = C x + Du
27

Polynomial models - ARX & ARMAX
ARX:
• Auto regression - current
output depends on the past
input and output values.
• Considering auto regression
and the current inputs, ARX
model can be mathematically
described as:
A(z) y(t) = B(z)u(t - n) + e(t)
28
ARMAX:
• Unlike ARX, ARMAX considers
stochastic dynamics.
• Better for systems with extra
disturbances
• Includes both AR(p) and MA(q)
models.
• ARMAX is represented mathematically
as
A(z) y(t) = B(z)u(t - n) + c(z)e(t)

1. Set Output = [Response
Time]
2. Set Input = [ Number of
incoming requests, No of
instances, CPU%,
memory, and disk
utilization]
Data Modeling in MATLAB System
Identification Toolbox: Import Data
29

MATLAB System Identification Toolbox –
Dataset splitting
30

• Model estimation
• State-space Model
• Polynomial Models
• ARX
• ARMAX
• Validate the
model
Model Estimation and Validation
31

Estimating State Space model
32

Estimating ARX and ARMAX models
33

Evaluation
34
Image Source: https://www.usu.edu/ccampis/evaluation/

Deploy Guestbook Application to Cloud
35

Generate Load on Guestbook app
36

Scaling occurs
37
RequestspersecondNo.ofinstances

Configuration File Settings
38

Comparison with and without the Auto-scaler
39
Requests
persecond
Average
response
time
Requests
persecond
Average
response
time
Without Auto-scaler,
Maximum response
time is 750 ms
With Auto-scaler,
Maximum response
time is 100 ms

Model output
The Model estimation output for State space, ARX and ARMAX models.
40

Insert page title
Headline here
§ Lorem ipsum dolor sit amet, consectetur
adipiscing elit. Maecenas pellentesque laoreet
urna molestie placerat ipsum.
Headline here
§ Lorem ipsum dolor sit amet, consectetur
adipiscing elit. Maecenas pellentesque laoreet
urna molestie placerat ipsum.
Video
Demo

Conculsion
43
Automated Resource
provisioning Using the run-
time metrics
Solved!
ü Monitoring Service
ü Auto-scaler
ü Correlation Model to
make the auto-scaler
smart!

• Metric parameter tuning
• Metric combination & Custom Metrics
• Determining metric thresholds
• Testing with different applications types - memory or database
intensive.
• Improved modeling, collection of larger sample datasets
Open Challenges and Future work

References
Ø M. Httermann,DevOps for developers.Apress, 2012.
Ø J. Cito, P. Leitner,H. C. Gall, A. Dadashi,A. Keller,and A. Roth, “Runtime metric meets developer:building better cloud
applications using feedback,” in 2015 ACM International Symposium on New Ideas, New Paradigms,and Reflections on
Programming and Software,pp. 14–27,ACM, 2015.
Ø “Amazon web services.” https://aws.amazon.com/, Accessed:08-Aug-2016.
Ø “New relic.” https://newrelic.com/, Accessed: 08-Aug-2016.
Ø J. Cito, P. Leitner,T. Fritz, and H. C. Gall, “The making of cloud applications:An empirical study on software development
for the cloud,” in Proceedings ofthe 2015 10th Joint Meeting on Foundations of Software Engineering,pp.393–403,ACM,
2015.
Ø L. Ljung,System identification.Springer,1998.
Ø “Sap hana cloud platform.” https://hcp.sap.com/index.html, Accessed: 08-Aug-2016.
Ø S. R. Seelam,P. Dettori, P. Westerink, and B. B. Yang,“Polyglotapplication auto scaling service for platform as a service
cloud,” in Cloud Engineering (IC2E),2015 IEEE International Conference on,pp. 84–91,IEEE, 2015.
Ø J. Humble and D. Farley, Continuous delivery:reliable software releases through build,test, and deploymentautomation.
Pearson Education,2010.
Ø http://docs.pivotal.io/pivotalcf/1-7/customizing/autoscale-configuration.html
Ø Cloud wave project- http://cloudwave-fp7.eu/
45

Thank You
46
www.cloudwave-fp7.eu
Prof. Dr. –Ing. Mira Mezini
Dr. –Ing. Guido Salvaneschi
Dr. Gerald Junkermann
Aryan Dadashi
Jürgen Cito

Appendix: Polynomial models - ARX
• The ARX model to evaluate the output is based on auto-regression.
• Auto regressive model is a model whose current output depends on the past
input and output values. The generic notion to denote auto-regressive model of
order p, AR(p) for a variable X is:
where c and i are constants and e(t) is the noise. Considering auto regression and
the inputs, ARX model can be mathematically described as:
A(z) y(t) = B(z)u(t - n) + e(t)
where y(t) is the output, u(t) is the input, and e(t) is the noise/error measured in the
output. A(z) and B(z) are polynomials of the specified order with respect to the
backward shift operator z-1.
47

Appendix: Polynomial models - ARMAX
Unlike the ARX model, in ARMAX, the stochastic dynamics are considered. ARMAX models
are better for systems with more disturbances. In general, the moving average model of
order q, MA(q) is represented in the below notation:
where e(t-i) is the noise/error. The notation for the autoregressive moving average(ARMA)
model is as below:
This model includes both AR(p) and MA(q) models. Based on these the following
mathematical equation for the ARMAX model can be written as:
A(z) y(t) = B(z)u(t - n) + c(z)e(t)
where, y(t) is the output, u(t)is the input, and e(t) is the noise. A(z), B(z) and C(z) are
polynomials of specified orders with respect to the backward shift operator z-1
48

Appendix: CASpott
Feedback Handler
- Queries the raw logs
- Prepares the raw logs for
further processing by
mapping them to the API
format of the Statistical
Analysis Engine
Statistical Analysis Engine
- Analyzes and aggregates
the collected information
coming from Feedback
Handler based on
predefined metrics
Static Code Analysis Engine
- Analyzes the code and
realizes the connection
between the feedback and
the code

Feedback Driven Development

Recommended

Recommended

More Related Content

Similar to Feedback Driven Development

Similar to Feedback Driven Development (20)

Recently uploaded

Recently uploaded (20)

Feedback Driven Development