Towards agile formal methods The main goal of this work is to overcome the aforementioned limitations by enabling automated decision gates in performance testing of microservices that allow requirements traceability. We seek to achieve this goal by endowing common agile practices used in microservice performance testing, with the ability to automatically learn and then formally verify a performance model of the System Under Test (SUT) to achieve strong assurances of quality. Even if the separation between agile and formal methods increased over the years, we support the claim that formal methods are at a stage where they can be effectively incorporated into agile methods to give them rigorous engineering foundations and make them systematic and effective with strong guarantees.

1. Performance assessment of microservices with strong guarantees
Towards agile formal methods
Matteo Camilli, Assistant Professor “Junior” (RTDA)
Faculty of Computer Science,
Free University of Bozen-Bolzano, Italy
matteo.camilli@unibz.it
https://matteocamilli.github.io
1

2. Matteo Camilli - matteo.camilli@unibz.it
Outline
2
• Introduction
• DevOps practices for microservices
• QA: performance & scalability
• Our approach
• Automated test-based learning of performance models
• Automated formal veri
fi
cation of performance requirements
• Some results from controlled experiments
• Take away message

3. Matteo Camilli - matteo.camilli@unibz.it
Introduction
3
• The microservice architectural style
• Distributed application where all its modules are microservices
• Microservice: cohesive, independent process interacting via messages (HTTP API)
SOA Microservices
Unix
design
principles
✓ modularity
✓ maintainability
✓ easier deployment of latest build
…

4. Matteo Camilli - matteo.camilli@unibz.it
• The DevOps setting
• Frequent releases in short cycles
• Changes —> easy to introduce
• E
ff
ects of changes —> hard to understand
• Decisions
• e.g., go/no-go, speci
fi
c deployment con
fi
g
• Important for success/failure!
• We’d like to make them with
“strong guarantees”
Introduction (2)
4
requirements
performance
testing

5. Matteo Camilli - matteo.camilli@unibz.it
• The DevOps setting
• Frequent releases in short cycles
• Changes —> easy to introduce
• E
ff
ects of changes —> hard to understand
• Decisions
• e.g., go/no-go, speci
fi
c deployment con
fi
g
• Important for success/failure!
• We’d like to make them with
“strong guarantees”
Introduction (2)
4
requirements
performance
testing
How should I drive the testing process?

6. Matteo Camilli - matteo.camilli@unibz.it
• The DevOps setting
• Frequent releases in short cycles
• Changes —> easy to introduce
• E
ff
ects of changes —> hard to understand
• Decisions
• e.g., go/no-go, speci
fi
c deployment con
fi
g
• Important for success/failure!
• We’d like to make them with
“strong guarantees”
Introduction (2)
4
requirements
performance
testing
How do I specify
requirements?
How should I drive the testing process?

7. Matteo Camilli - matteo.camilli@unibz.it
• The DevOps setting
• Frequent releases in short cycles
• Changes —> easy to introduce
• E
ff
ects of changes —> hard to understand
• Decisions
• e.g., go/no-go, speci
fi
c deployment con
fi
g
• Important for success/failure!
• We’d like to make them with
“strong guarantees”
Introduction (2)
4
requirements
performance
testing
How do I specify
requirements?
How should I drive the testing process?
How to make
the “right” decisions?

8. Matteo Camilli - matteo.camilli@unibz.it
• Analysis of the operational setting
• Usage pro
fi
le —> behavior of the users in terms of possible sequences of interactions Actors <—> App
Our approach — how to drive testing
5
Timestamp Session id Request
t1 1 home
t2 1 login
t3 2 home
…
Valid requests RESTful API Log of sessions (HTTP requests) Graph of traces (DTMC model)
(E.g., Disco https://
fl
uxicon.com/disco/)
Process Mining Graph structure
with probability
on edges
SockShop demo
API

9. Matteo Camilli - matteo.camilli@unibz.it
• Analysis of the operational setting
• Workload intensity —> number of concurrent users accessing the App
Our approach — how to drive testing (2)
6
Empirical distribution
of the workload intensity
Discrete distribution
Data binning

10. Matteo Camilli - matteo.camilli@unibz.it
• Analysis of the operational setting
• Workload intensity —> number of concurrent users accessing the App
Our approach — how to drive testing (2)
6
Empirical distribution
of the workload intensity
Discrete distribution
Data binning
Very likely
workload
unlikely
workload

11. Matteo Camilli - matteo.camilli@unibz.it
• Elicitation of the factors
• DTMC model represents the behavior of the users;
• the empirical distribution for each workload intensity ;
• the set of alternative deployment con
fi
gurations, each one de
fi
ning (in our case)
• amount of RAM,
• CPU share,
• replicas per microservice
• Testing sessions
• One session for each
• Each session shall generate synthetic users according to
ℳ
f(λ1), . . . , f(λk) λi ∈ Λ
𝒞
⟨λ, c⟩ ∈ Λ ×
𝒞
ℳ
Our approach — how to drive testing (3)
7
(in our in-vitro experiments)

12. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to drive testing (4)
8

13. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to drive testing (4)
8
• For each testing session
• Docker deploys the SUT according to
• LOCUST generates users according to
and
• Inference module automatically augment
with time information to obtain
(CTMC model)
c
λ
ℳ
ℳ
𝒳

14. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to drive testing (4)
8
• For each testing session
• Docker deploys the SUT according to
• LOCUST generates users according to
and
• Inference module automatically augment
with time information to obtain
(CTMC model)
c
λ
ℳ
ℳ
𝒳
• Bayesian inference
• For each edge i,j
• We collect #times , total time
• Gamma posterior —>
• CTMC rate
Ni Ti
νi = Ni/Ti
rij = pijνi

15. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to drive testing (4)
8
• For each testing session
• Docker deploys the SUT according to
• LOCUST generates users according to
and
• Inference module automatically augment
with time information to obtain
(CTMC model)
c
λ
ℳ
ℳ
𝒳
• Bayesian inference
• For each edge i,j
• We collect #times , total time
• Gamma posterior —>
• CTMC rate
Ni Ti
νi = Ni/Ti
rij = pijνi
• Bayes factor
• Termination criterion (inferred parameters
are statistically stable)

16. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to make the “right” decisions
9
Critical!
lower probability,
Higher likelihood

17. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to make the “right” decisions
9
• Computation of the cumulative probability distributions
• For each service Zs(x) = P(RTs < x)
Critical!
lower probability,
Higher likelihood

18. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to make the “right” decisions
9
• Computation of the cumulative probability distributions
• For each service Zs(x) = P(RTs < x)
• Con
fi
guration score
• Given a workload we’d like to maximize the
area under the curve weighted by the likelihood
•
λ
As Zs
φ(λ) =
∑
s
As
𝒮
(s)
Critical!
lower probability,
Higher likelihood

19. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to make the “right” decisions
9
• Computation of the cumulative probability distributions
• For each service Zs(x) = P(RTs < x)
• Con
fi
guration score
• Given a workload we’d like to maximize the
area under the curve weighted by the likelihood
•
λ
As Zs
φ(λ) =
∑
s
As
𝒮
(s)
• Total score
• Then maximize the score over multiple sessions
weighed by the likelihood of a given workload
•
Φ(Λ) =
∑
λ
φλ f(λ)
Critical!
lower probability,
Higher likelihood

20. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to make the “right” decisions (2)
10

21. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to make the “right” decisions (2)
10
4 replicas are likely
to decrease performance

22. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to make the “right” decisions (2)
10
4 replicas are likely
to decrease performance
Good options (we could then chose based on resource consumption)

23. Matteo Camilli - matteo.camilli@unibz.it
Our approach — how to make the “right” decisions (3)
11
• Formal veri
fi
cation of performance requirements
• Expressed using Continuous Stochastic Logic (CSL)
• Fully automated by o
ff
-the-shelf model checking tools (e.g., PRISM https://www.prismmodelchecker.org/)
CSL requirement examples
Automated verification report
E.g., config 7 does not meet
all the reqs R1-R5

24. Matteo Camilli - matteo.camilli@unibz.it
Take away message
12
• Make the “right” decisions with strong guarantees is important!
• We shown how to achieve this goal by integrating formal approaches into DevOps practices
• Automated execution load testing sessions
• Automated building of formal models (CTMCs)
• Automated computation of the con
fi
guration score
• Automated veri
fi
cation of domain-dependent requirements
• Our implementation is open source
• PPTAM load testing tool https://github.com/pptam/pptam-tool
• Test-based learning and veri
fi
cation https://github.com/matteocamilli/pptam-ctmc-experiments