The document discusses an approach called semi-purification for automatically generating unit test cases from source code. Semi-purification replaces dependencies like global variables and database calls in the source code with function parameters. This allows existing automated test case generation tools to be used by treating the semi-purified code as if it were pure. Challenges discussed include handling shared subroutines, loops, and concurrency. The goal is to increase test coverage for complex, distributed systems with frequent changes like those used at CERN.

1. Considering Execution Environment Resilience:
A White-Box Approach
Stefan Klikovits1,2, David PY Lawrence1,3,
Manuel Gonzalez-Berges2, Didier Buchs1
1Université de Genève, Carouge, Switzerland
2CERN, Geneva, Switzerland
3Honeywell International Sarl., Rolle, Switzerland
Tuesday 8th September, 2015

2. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
What is this all about?
How to
generate test cases with little user interaction
on a large scale
unit test level
2 / 21

3. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Welcome to CERN
3 / 21

4. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Welcome to CERN
LHC, experiments, infrastructure (e.g. power grid)
large-scale, widespread, complex systems
many types of hard- and software
> 100 subsystems,
10,000s of devices,
1,000,000s of parameters
thousands of physicists/engineers/workers
high employee turnover
3 / 21

5. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Welcome to CERN
LHC, experiments, infrastructure (e.g. power grid)
large-scale, widespread, complex systems
many types of hard- and software
> 100 subsystems,
10,000s of devices,
1,000,000s of parameters
thousands of physicists/engineers/workers
high employee turnover
high reliability and resilience expectations
3 / 21

6. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
How do we supervise it?
framework built on top for HEP domain
proprietary scripting language: Control (CTRL)
4 / 21

7. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
So where is the problem?
until recently no automated unit test execution
frequent changes in execution environment
(mostly) manual verification
big expenses (time) on QA side
5 / 21

8. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
So where is the problem?
until recently no automated unit test execution
frequent changes in execution environment
(mostly) manual verification
big expenses (time) on QA side
⇒ increase test coverage
5 / 21

9. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Testing
1 f ( x ){
2 i f GLOBAL_VAR:
3 return dbGet ( x )
4 e l s e :
5 return −1
6 }
f(x)
1 test_f (){
2 dbSet ( " t e s t " ,5) // prepare
3 GLOBAL_VAR = True
4 x = f ( " t e s t " ) // act
5 a s s e r t ( x == 5) // a s s e r t
6 }
Test case for f(x)
6 / 21

10. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
What do we want?
code test cases
Iterative Test Case System (ITEC)
regression testing
consider dependencies
automatic test case generation (ATCG)
build on existing research & tools
generate unit & component tests
7 / 21

11. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Automated Test Case Generation
source code based
black-box (function signature) vs.
white-box (function body)
8 / 21

12. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Semi-purification
replace dependencies with parameters
1 f ( x ){
2 i f GLOBAL_VAR :
3 return dbGet(x)
4 e l s e :
5 return −1
6 }
A non-pure function
1 f_sp ( x , a ,b){
2 i f a :
3 return b
4 e l s e :
5 return −1
6 }
Semi-purified f(x)
a
a
a
a
9 / 21

13. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Semi-purification
replace dependencies with parameters
1 f ( x ){
2 i f GLOBAL_VAR :
3 return dbGet(x)
4 e l s e :
5 return −1
6 }
A non-pure function
1 f_sp ( x , a ,b){
2 i f a :
3 return b
4 e l s e :
5 return −1
6 }
Semi-purified f(x)
1 test_f_sp (){
2 x = f ( " t e s t " , True , 5 ) // act
3 a s s e r t ( x == 5) // a s s e r t
4 }
Test case
9 / 21

14. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Semi-purification (cont.)
replace dependencies with parameters
1 functionA ( x ){
2 a = functionB ( x )
3 return a
4 }
5
6 functionB ( x ){
7 b = GLOBAL_VAR
8 b++
9 return b
10 }
Function with SRC
1 functionA_sp ( x , y){
2 a = functionB ( x , y)
3 return a
4 }
5
6 functionB_sp ( x , y){
7 b = y
8 b++
9 return b
10 }
Semi-purified w. SRC
10 / 21

15. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Semi-purification (cont.)
replace dependencies with parameters
1 functionA ( x ){
2 a = functionB ( x )
3 return a
4 }
5
6 functionB ( x ){
7 b = GLOBAL_VAR
8 b++
9 return b
10 }
Function with SRC
1 functionA_sp ( x , y){
2 a = functionB ( x , y)
3 return a
4 }
5
6 functionB_sp ( x , y){
7 b = y
8 b++
9 return b
10 }
Semi-purified w. SRC
10 / 21

16. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Semi-purification: Concept
code contains dependencies
global variables, data base values, subroutine calls,
other resources
manual way: test doubles (mocks, stubs, fakes, . . . )
[ME06]
remove dependencies
based on localization [SW03, SK13]
input parameters instead of dependencies
use any ATCG (black- and white-box)
11 / 21

17. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
“Shortcut”
f (P){D} f (P ∪ PD ){}
TI, TS TI ∪ TID
SP
SP−1
ATCG
Figure: Test case generation schema
P . . . Parameters
D . . . Dependencies
TI . . . Test Input
TS . . . Test Setup Routine
12 / 21

18. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Identified Technical Challenges
Shared Subroutines
Loops
Concurrency
13 / 21

19. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Shared subroutine dependencies
1 var SPEED_VAR = 1
2 adjustSpeed ( ){
3 x = getTheSpeed ( )
4 i f x < 10 :
5 doubleTheSpeed ( )
6 }
CUT
1 getTheSpeed ( ){
2 return SPEED_VAR
3 }
Subroutine 1
1 doubleTheSpeed ( ){
2 speed = SPEED_VAR
3 SPEED_VAR = speed ∗2
4 }
Subroutine 2
14 / 21

20. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Shared subroutine dependencies
1 var SPEED_VAR = 1
2 adjustSpeed ( ){
3 x = getTheSpeed ( )
4 i f x < 10 :
5 doubleTheSpeed ( )
6 }
CUT
1 getTheSpeed ( ){
2 return SPEED_VAR
3 }
Subroutine 1
1 doubleTheSpeed ( ){
2 speed = SPEED_VAR
3 SPEED_VAR = speed ∗2
4 }
Subroutine 2
14 / 21

21. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Shared subroutine dependencies
1
2 adjustSpeed (a){
3 x = getTheSpeed (a)
4 i f x < 10 :
5 doubleTheSpeed ( )
6 }
CUT
1 getTheSpeed (a){
2 return a // SPEED_VAR
3 }
Subroutine 1
1 doubleTheSpeed ( ){
2 speed = SPEED_VAR
3 SPEED_VAR = speed ∗2
4 }
Subroutine 2
14 / 21

22. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Shared subroutine dependencies
1
2 adjustSpeed (a, b){
3 x = getTheSpeed (a)
4 i f x < 10 :
5 doubleTheSpeed (b)
6 }
CUT
1 getTheSpeed (a){
2 return a // SPEED_VAR
3 }
Subroutine 1
1 doubleTheSpeed (b){
2 speed = b // SPEED_VAR
3 b = speed ∗2
4 }
Subroutine 2
14 / 21

23. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Shared subroutine dependencies
1
2 adjustSpeed (a, b){
3 x = getTheSpeed (a)
4 i f x < 10 :
5 doubleTheSpeed (b)
6 }
CUT
1 getTheSpeed (a){
2 return a // SPEED_VAR
3 }
Subroutine 1
1 doubleTheSpeed (b){
2 speed = b // SPEED_VAR
3 b = speed ∗2
4 }
Subroutine 2
14 / 21

24. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Shared subroutine dependencies
1
2 adjustSpeed (a, b){
3 x = getTheSpeed (a)
4 i f x < 10 :
5 doubleTheSpeed (b)
6 }
CUT
1 getTheSpeed (a){
2 return a // SPEED_VAR
3 }
Subroutine 1
1 doubleTheSpeed (b){
2 speed = b // SPEED_VAR
3 b = speed ∗2
4 }
Subroutine 2
14 / 21

25. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Shared subroutine dependencies
1
2 adjustSpeed (a, b){
3 x = getTheSpeed (a)
4 i f x < 10 :
5 doubleTheSpeed (b)
6 }
CUT
1 getTheSpeed (a){
2 return a // SPEED_VAR
3 }
Subroutine 1
1 doubleTheSpeed (b){
2 speed = b // SPEED_VAR
3 b = speed ∗2
4 }
Subroutine 2
BUT: a and b replace the same dependency
14 / 21

26. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Loops
1 s l e e p U n t i l R e a d y ( ){
2
3 while dbGet(notReadyDP) :
4 s l e e p (5) // s l e e p f o r 5 seconds
5
6 }
A semi-purified loop
15 / 21

27. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Loops
1 s l e e p U n t i l R e a d y (a){ // a =bool
2
3 while a : // replaces dbGet(notReadyDP)
4 s l e e p (5) // s l e e p f o r 5 seconds
5
6 }
A semi-purified loop
Test Cases:
a: False ⇒ loop not executed
a: True ⇒ endless loop
15 / 21

28. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Loops
1 s l e e p U n t i l R e a d y (a){ [bool]
2 i = 0
3 while a[i] : // replaces dbGet(notReadyDP)
4 s l e e p (5) // s l e e p f o r 5 seconds
5 i++
6 }
A semi-purified loop
Test Cases:
a: [False]⇒ loop not executed
a: [True, True, . . . , False] ⇒ loop execution
Open questions:
how long should the list be?
how to modify correctly?
test modified code or w. threads?
15 / 21

29. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
Concurrency
manager
Ctrl UI
manager
Device
Drivers
Dist
manager
DB
manager
EV
manager
Figure: WinCC OA’s manager concept
16 / 21

30. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
SP: Technical Challenges (cont.)
access to ALL data points
100+ sub-systems
discover dirty read/write scenarios (adjustSpeed())
16 / 21

31. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
ITEC implementation
IDE SP engine
CTRL
test gen.
test
driver
code SPdata
test cases
results
ITEC
ATCG
SP code
test inputs
Figure: ITEC workflow
17 / 21

32. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
ITEC implementation
IDE SP engine
CTRL
test gen.
test
driver
code SPdata
test cases
results
ITEC
ATCG
source code
translator
test input
translator
SP CTRL
CTRL
test inputs
SP tool code
test inputs
TI generator
Figure: ITEC workflow
17 / 21

33. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Conclusion
Semi-purification
remove dependencies
facilitate unit tests generation
use black-box techniques on all code
18 / 21

34. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
Future works
What’s next?
connect to test framework & roll-out
intrinsic domain information
compare different ATCG
research into technical challenges
19 / 21

35. Considering
Execution
Environment
Resilience:
A White-Box
Approach
S.Klikovits,
D. Lawrence,
M. Gonzalez-
Berges,
D. Buchs
References
[ME06] Meszaros, G.:
XUnit Test Patterns: Refactoring Test Code, Chapter 23.
Test Double Patterns, pages 521–590.
Prentice Hall PTR, Upper Saddle River, NJ, USA, (2006)
[SC03] Sward, R. E., Chamillard, A. T.:
Re-engineering Global Variables in Ada.
In: Proc. 2004 ACM SIGAda international conference on
Ada, pp. 29–34, ACM, New York, (2003).
[SK13] Sankaranarayanan, H., Kulkarni, P.:
Source-to-Source Refactoring and Elimination of Global
Variables in C Programs.
In: Journal of Software Engineering and Applications,
Vol. 6 No. 5, pp. 264–273, (2013).
20 / 21

36. Considering Execution Environment Resilience:
A White-Box Approach
Stefan Klikovits1,2, David PY Lawrence1,3,
Manuel Gonzalez-Berges2, Didier Buchs1
1Université de Genève, Carouge, Switzerland
2CERN, Geneva, Switzerland
3Honeywell International Sarl., Rolle, Switzerland
Tuesday 8th September, 2015