The document discusses automated boundary value testing and analysis. It defines program boundaries as places where behavior is supposed to change or actually changes. It explores using diversity to automatically test boundary cases by comparing multiple executions and observing differences. Challenges include defining appropriate diversity metrics and handling different data types and sizes.

1. Using Diversity for Automated Boundary
Value Testing
Felix Dobslaw
#TestVikings
Crest open Workshop (CoW) 2019

2. 2/77
Boundary Value Analysis/Testing

3. 3/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent

4. 4/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent

5. 5/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent

6. 6/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent
What are program boundaries?

7. 7/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent
What are program boundaries?
Take 1: where behavior is supposed to change (specification centric)

8. 8/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent
What are program boundaries?
Take 1: where behavior is supposed to change (specification centric)
Take 2: where behavior changes (function centric)

9. 9/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent
What are program boundaries?
Take 1: where behavior is supposed to change (specification centric)
Take 2: where behavior changes (function centric)
Tackle Take 1: Explicit specification, check edge cases. Requires manual work.

10. 10/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent
What are program boundaries?
Take 1: where behavior is supposed to change (specification centric)
Take 2: where behavior changes (function centric)
Tackle Take 1: Explicit specification, check edge cases. Requires manual work.
Tackle Take 2: Explore program boundaries, extract actual edge cases. Can be automated.

11. 11/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent
What are program boundaries?
Take 1: where behavior is supposed to change (specification centric)
Take 2: where behavior changes (function centric)
Tackle Take 1: Explicit specification, check edge cases. Requires manual work.
Tackle Take 2: Explore program boundaries, extract actual edge cases. Can be automated.
BVA vs. BVT?

12. 12/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent
What are program boundaries?
Take 1: where behavior is supposed to change (specification centric)
Take 2: where behavior changes (function centric)
Tackle Take 1: Explicit specification, check edge cases. Requires manual work.
Can be used in combination.
Tackle Take 2: Explore program boundaries, extract actual edge cases. Can be automated.
BVA vs. BVT?

13. 13/77
Boundary Value Analysis/Testing
Why? Find bugs, anomalies, deviation from intent
What are program boundaries?
Take 1: where behavior is supposed to change (specification centric)
Take 2: where behavior changes (function centric)
Tackle Take 1: Explicit specification, check edge cases. Requires manual work.
Can be used in combination.
Tackle Take 2: Explore program boundaries, extract actual edge cases. Can be automated.
BVA vs. BVT?

14. Why Automated Boundary Value Testing?
14/77

15. Why Automated Boundary Value Testing?
$ sudo -u# -1 id -u
15/77

16. Why Automated Boundary Value Testing?
$ sudo -u# -1 id -u
It is at the boundaries where interesting things
happen...
16/77

17. 17/77

18. 18/77

19. 19/77

20. 20/77
aBVA/aBVT literature

21. 21/77
aBVA/aBVT literature

22. 22/77
aBVA/aBVT literature

23. 23/77
aBVA/aBVT literature

24. 24/77

25. 25/77
Specification:
inputs
outputs
requirements
constraints

26. 26/77
program
Specification:
inputs
outputs
requirements
constraints

27. 27/77
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints

28. 28/77
Vision
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints augmented
process

29. 29/77
Absolute vs. Relative Assessment

30. 30/77
SUTx y
Absolute vs. Relative Assessment

31. 31/77
SUTx y
Assert y == yexp
Absolute vs. Relative Assessment

32. 32/77
SUTx y
Assert y == yexp
Absolute vs. Relative Assessment

33. 33/77
SUTx y
x1 x2
y1 y2
SUT SUT
Assert y == yexp
Absolute vs. Relative Assessment

34. 34/77
SUTx y
x1 x2
y1 y2
?
SUT SUT
Assert y == yexp
Absolute vs. Relative Assessment

35. 35/77
Foundation: Diversity

Challenge: Describe relation between inputs and outputs for
arbitrary data types.
Automated BVT/BVA?
x1 x2
SUT
y1 y2
?

36. 36/77
Foundation: Diversity

Challenge: Describe relation between inputs and outputs for
arbitrary data types.
Automated BVT/BVA?
x1 x2
SUT
y1 y2
?

37. 37/77
Foundation: Diversity

Challenge: Describe relation between inputs and outputs for
arbitrary data types.

Kolmogorov Complexity has a potential
Automated BVT/BVA?
x1 x2
SUT
y1 y2
?

38. 38/77
Foundation: Diversity

Challenge: Describe relation between inputs and outputs for
arbitrary data types.

Kolmogorov Complexity has a potential

Applicable for all data types
Automated BVT/BVA?
x1 x2
SUT
y1 y2
?

39. 39/77
Foundation: Diversity

Challenge: Describe relation between inputs and outputs for
arbitrary data types.

Kolmogorov Complexity has a potential

Applicable for all data types

“Compression trick” makes it practical

CC ~ KC
Automated BVT/BVA?
x1 x2
SUT
y1 y2
?

40. 40/77
Foundation: Diversity

Challenge: Describe relation between inputs and outputs for
arbitrary data types.

Kolmogorov Complexity has a potential

Applicable for all data types

“Compression trick” makes it practical

CC ~ KC

Normalized Information Distance (NID)
Automated BVT/BVA?
x1 x2
SUT
y1 y2
?

41. 41/77
Diversity Corridor

42. 42/77
Diversity Corridor

43. 43/77
Diversity Corridor

44. 44/77
Diversity of What?
SUT
SUT=Software Under Test

45. 45/77
Diversity of What?
SUTInput
SUT=Software Under Test

46. 46/77
Diversity of What?
SUTInput Output
SUT=Software Under Test

47. 47/77
Diversity of What?
SUTInput Output
State
SUT=Software Under Test

48. 48/77
Diversity of What?
SUTInput Output
State
SUT=Software Under Test

49. 49/77
Diversity of What?
SUTInput Output
State
SUT=Software Under Test

50. 50/77
Diversity of What?
SUTInput Output
State
SUT=Software Under Test

51. 51/77
Diversity of What?
SUTInput Output
State
x
SUT=Software Under Test

52. 52/77
Diversity of What?
SUTInput Output
State
x
Diversity(Execution1, Execution2) → A x B x C...
SUT=Software Under Test

53. 53/77
Derivatives
● Slope of the tangent line to the graph at x
How sensitive to change is f in x?

54. 54/77
Derivatives
● Slope of the tangent line to the graph at x
How sensitive to change is f in x?
secant

55. 55/77
Derivatives
● Slope of the tangent line to the graph at x
How sensitive to change is f in x?
secant

56. 56/77
Input Mutation
How do I get bmin then?

May require exploring the SUT’s behavior on a number of “close
values”

e.g. via Search-based Software Engineering with Mutators

57. 57/77
Boundary Without Oracle
month
day of month
year

58. 58/77
aBVA 4 ML?
SUTInput Output

59. 59/77
aBVA 4 ML?
anything anything
SUTInput Output
Domain:

60. 60/77
aBVA 4 ML?
anything anything
SUTInput Output
Domain:

61. 61/77
aBVA 4 ML?
SUTInput Output
Domain:

62. 62/77
aBVA 4 ML?
Input Output
Domain:

63. 63/77
aBVA 4 ML?
MUT=Model Under Test
Input Output
Domain:
MUT
(ANN)

64. 64/77
aBVA 4 ML?
[0,1]^n U {0,1}^n [0,1]^n | {0,1}^n
MUT=Model Under Test
Input Output
Domain:
MUT
(ANN)

65. 65/77
aBVA 4 ML?
[0,1]^n U {0,1}^n [0,1]^n | {0,1}^ndistance + direction
MUT=Model Under Test
Input Output
Domain:
MUT
(ANN)

66. 66/77
This is where we are...

67. 67/77
This is where we are...
● All diversity measures may not be practical

68. 68/77
This is where we are...
● All diversity measures may not be practical
● deterministic vs stochastic

69. 69/77
This is where we are...
● All diversity measures may not be practical
● deterministic vs stochastic
● Discrimination on boundary

70. 70/77
This is where we are...
● All diversity measures may not be practical
● deterministic vs stochastic
● Discrimination on boundary
● single metric may not discriminate well
● output diversity upper hand (?)

71. 71/77
This is where we are...
● All diversity measures may not be practical
● deterministic vs stochastic
● Discrimination on boundary
● single metric may not discriminate well
● output diversity upper hand (?)
● distance metric has an impact, NCD possibly
too simplistic (to be continued...)

72. 72/77
Challenges

73. 73/77
Challenges

74. 74/77
Challenges
size

75. 75/77
Challenges
sizedirection

76. 76/77
Challenges
sizedirection
ruggedness

77. 77/77
TestVikings

“We come over land and sea to break your code”
Find us at: https://testvikings.github.io/
I don’t tweet but post on LinkedIn (Felix Dobslaw)