In this analysis, we establish an economic and mathematical (queueing theory) framework to calculate how much it would cost to build the dev/test lab of your dreams. One in which developers never wait for testing resources, and every commit is tested on replicas of the production environment. Then, we contrast several different options to get there and evaluate all of them from an economic perspective. After the presentation, in order to illustrate that this is not only theory, I also gave a quick demonstration of how we at Ravello use our own technology to develop our own application. Each engineer can spin up as many instances of the production replica app as needed on demand for dev/test. We showed how we have integrated Ravello with Jenkins so that on every commit, we spin up the production replica application and run integration tests in parallel. If you have any questions, feel free to reach out. We are more than happy to discuss how this may be relevant to your development process. www.ravellosystems.com

1. The
economic
(in)feasibility
of
on-­‐premise
dev/test
The
demand
and
supply
mismatch
of
development
and
tes4ng
resources
Navin
R.
Thadani,
@navinthadani
Ravello
Systems,
Inc.
Ravello
Systems,
Inc.
|
GlueCon
2013

2. On-­‐premise
development
&
test
sucks
Sprint
1
Sprint
2
Release
Candidate
Product
late
to
market
and
poor
quality
Data
center/private
cloud
capacity
Project
Demand
Resources
Resources
Sprint
1
Sprint
2
Release
Candidate
Ravello
Systems,
Inc.
|
GlueCon
2013

3. #1
Lack
of
internal
capacity
#2
Difficult
to
replicate/
provision
environments
quickly
Ravello
Systems,
Inc.
|
GlueCon
2013

4. A
simple
applicaSon
router
firewall
load
balancer
be1
OS
db2
OS
fe2
OS
fe1
OS
be2
OS
db1
OS
Front
end
Back
end
db
4er
q
OS
Ravello
Systems,
Inc.
|
GlueCon
2013

5. A
simple
development
process
Unit
test
environments
Integra4on
test
environments
System
test
environments
for
every
commit
for
every
commit
batched
Ravello
Systems,
Inc.
|
GlueCon
2013

6. Fundamental
quesSon
How
much
capacity
do
you
need
to
opSmally
develop
and
test
your
applica4on?
Ravello
Systems,
Inc.
|
GlueCon
2013

7. AssumpSon
As
far
as
possible,
you
want
to
test
on
replicas
of
the
produc4on
environment
Ravello
Systems,
Inc.
|
GlueCon
2013

8. A
queuing
theory
perspecSve
C2
C3
Test
environments
C6
C7
C8
C1
Arrival
rate
=
λ
commits/hour
C5
Service
rate
=
μ
tests/hour
M-­‐D-­‐s
queue
Average
Sme
spent
in
the
queue
Probability
of
waiSng
more
than
5
minutes
Overall
resource
uSlizaSon
Ravello
Systems,
Inc.
|
GlueCon
2013

9. Dev/test
efficiency
fronSer
Number
of
tesSng
environments
Cost
Cost
of
tes4ng
environments
Cost
of
“wai4ng”
4me
Total
cost
X
TE1
Ravello
Systems,
Inc.
|
GlueCon
2013

10. 8
person
development
team
5,000
commits
(in
12
months)
300,000
lines
of
Java
Ravello
Systems,
Inc.
|
GlueCon
2013

11. Some
more
numbers
Unit
tests
Integra4on
tests
System
test
environments
λ
4/hr
μ
8/hr
μ
5/hr
Ravello
Systems,
Inc.
|
GlueCon
2013

12. Average
wait
Sme
1 2 3 4 5 6 7 8 9 10
1 2 0 0 0 0 0 0 0 0 0
2 4 0 0 0 0 0 0 0 0 0
3 9 1 0 0 0 0 0 0 0 0
4 24 1 0 0 0 0 0 0 0 0
5 oo 2 0 0 0 0 0 0 0 0
Number
of
integraSon
tesSng
environments
Commit
rate
P0 =1/
λ
µ( )
n
n!
+
λ
µ( )
S
S!
1
1− λ / Sµ
"
#
$
%
&
'
n=0
S−1
∑
)
*
+
+
+
+
,
-
.
.
.
.
Lq =
λ / µ( )
S
λ / Sµ( )
S! 1-λ/Sµ( )
2
P0
Wq =
Lq
λ
Or
simple
simulaSon
h_ps://github.com/geertj
Dust
off
your
text
books
;)
Ravello
Systems,
Inc.
|
GlueCon
2013

13. Dev/test
efficiency
fronSer
(λ=4/hr)
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1
2
3
4
5
6
7
8
9
10
11
12
OpSmal
point
• 2
x
central
UT
• 2
x
IT
• 1
x
ST
• 2
x
manual
QA
Number
of
tes4ng
environments
Cost
Cost
of
“wai4ng”
Capex
+
opex
Total
cost
fron4er
Ravello
Systems,
Inc.
|
GlueCon
2013

14. Reality
bites
(badly)
Commits
per
hour
0
5
10
15
20
25
30
10/2/12
11/2/12
12/2/12
1/2/13
2/2/13
3/2/13
4/2/13
Average
4
commits/
hour
Peak
27
commits/
hour
Ravello
Systems,
Inc.
|
GlueCon
2013

15. Dev/test
efficiency
fronSer
Number
of
tesSng
environments
Cost
Cost
of
tes4ng
environments
Cost
of
“wai4ng”
4me
For
commit
rate
L1
For
commit
rate
L2
TE1
TE2
X1
X2
Ravello
Systems,
Inc.
|
GlueCon
2013

16. Average
wait
Sme
Number
of
integraSon
tesSng
environments
1 2 3 4 5 6 7 8 9 10
1 2 0 0 0 0 0 0 0 0 0
2 4 0 0 0 0 0 0 0 0 0
3 9 1 0 0 0 0 0 0 0 0
4 24 1 0 0 0 0 0 0 0 0
5 100 2 0 0 0 0 0 0 0 0
6 100 4 1 0 0 0 0 0 0 0
7 100 6 1 0 0 0 0 0 0 0
8 100 11 1 0 0 0 0 0 0 0
9 100 26 2 0 0 0 0 0 0 0
10 100 100 3 1 0 0 0 0 0 0
11 100 100 4 1 0 0 0 0 0 0
12 100 100 7 1 0 0 0 0 0 0
13 100 100 12 2 0 0 0 0 0 0
14 100 100 26 2 1 0 0 0 0 0
15 100 100 100 3 1 0 0 0 0 0
16 100 100 100 5 1 0 0 0 0 0
17 100 100 100 7 1 0 0 0 0 0
18 100 100 100 12 2 1 0 0 0 0
19 100 100 100 27 3 1 0 0 0 0
20 100 100 100 100 3 1 0 0 0 0
21 100 100 100 100 5 1 0 0 0 0
22 100 100 100 100 7 2 1 0 0 0
23 100 100 100 100 13 2 1 0 0 0
24 100 100 100 100 26 3 1 0 0 0
25 100 100 100 100 100 4 1 0 0 0
26 100 100 100 100 100 5 1 0 0 0
27 100 100 100 100 100 8 2 1 0 0
28 100 100 100 100 100 13 2 1 0 0
29 100 100 100 100 100 28 3 1 0 0
30 100 100 100 100 100 100 4 1 0 0
Commit
rate
Ravello
Systems,
Inc.
|
GlueCon
2013

17. Dev/test
efficiency
fronSer
(λ=27/hr)
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1
2
3
4
5
6
7
8
9
10
11
12
OpSmal
point
• 5
x
central
UT
• 8
x
IT
• 1
x
ST
• 3
x
manual
QA
Number
of
tes4ng
environments
Cost
Cost
of
“wai4ng”
Capex
+
Opex
Total
cost
fron4er
Ravello
Systems,
Inc.
|
GlueCon
2013

18. $100K
–
substandard
lab
WaiSng,
inefficiency,
resource
contenSon
…
$250K
–
good
lab
service
Efficient
development
Hidden
cost
>>$1m
(inefficiency)
Average
uSlizaSon
(<5%)
Ravello
Systems,
Inc.
|
GlueCon
2013

19. A
new
fronSer?
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1
2
3
4
5
6
7
8
9
10
11
12
Public
Cloud
25x
lower
cost
Internal
data
center
Number
of
tes4ng
environments
Ravello
Systems,
Inc.
|
GlueCon
2013

20. Reality
check
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1
2
3
4
5
6
7
8
9
10
11
12
Change
applicaSon
to
run
on-­‐premise
(producSon)
and
cloud
(development
and
test)
?
Public
Cloud
Internal
data
center
Number
of
tes4ng
environments
Ravello
Systems,
Inc.
|
GlueCon
2013

21. Your
opSons
1x
>>
2.5x
0.1x
<0.1x
Wai4ng
4me
Capex
+
Opex
Modify
your
app
to
run
in
the
cloud
and
internal
data
center
+
automa4on
Internal
sub
opSmal
lab
Internal
opSmal
lab
“Migrate”
to
public
cloud
How
we
have
solved
it

Ravello
Systems,
Inc.
|
GlueCon
2013

22. DemonstraSon
Ravello
Systems,
Inc.
|
GlueCon
2013

23. Ravello
Systems,
Inc.
|
GlueCon
2013