Speech on Software Boundary Exploration at Swedish Association for Software Testing (SAST) meetup 2021.

1. 1/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Exploring the Boundaries of our Software
Assist. Prof. Felix Dobslaw
25/5/2021 10:50-11:40 CEST
https://www.flickr.com/photos/a_siegel/2356136219

2. 2/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Exploring the Boundaries of our Software
Assist. Prof. Felix Dobslaw
25/5/2021 10:50-11:40 CEST
G
https://www.flickr.com/photos/a_siegel/2356136219
Based on collaboration with
Robert Feldt,
Francisco Gomes de Oliveira Neto

3. 3/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Outline
●
Why Boundaries, why exploration?
●
Boundary Value…. Testing/Analysis/Exploration
– Terminology
– Process
●
Outlook

4. 4/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Research Questions

5. 5/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Research Questions
●
How can we automatically find areas of relevance
for testing in our software?
– “Boundaries”

6. 6/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Research Questions
●
How can we automatically find areas of relevance
for testing in our software?
– “Boundaries”
●
How can we practically support testers in their
working routine?
– Better fault finding and efficiency

7. 7/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
What do we mean by test
automation?

8. 8/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
What do we mean by test
automation?
Not classical CI/CD
running human created test cases automatically

9. 9/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
What do we mean by test
automation?
Not classical CI/CD
running human created test cases automatically
But
running machine created test cases automatically

10. https://time.com/3785942/blue-marble/

11. https://time.com/3785942/blue-marble/
Defining Boundaries is hard

12. https://time.com/3785942/blue-marble/
Defining Boundaries is hard

13. https://time.com/3785942/blue-marble/
Defining Boundaries is hard
Test Automation Augmentation

14. https://time.com/3785942/blue-marble/
Defining Boundaries is hard
Test Automation Augmentation Diversity

15. 15/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
inputs
outputs
requirements
constraints

16. 16/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Specification:
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed

17. 17/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Specification:
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model

18. 18/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model

19. 19/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model

20. 20/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model

21. 21/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model

22. 22/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model
Goal
vs.
System

23. 23/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model
Goal
vs.
System

24. 24/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model
Goal
vs.
System

25. 25/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model
Goal
vs.
System

26. 26/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
validate
Specification
inputs
outputs
requirements
constraints
Entirely
implicit
Greatly
detailed
Waterfall
Model
Goal
vs.
System
Correct software depends on
correct boundaries. Boundary Value
Exploration reveals discrepancies
between desired and actual behavior.

27. 27/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)

28. 28/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Boundary Value
Testing

29. 29/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Boundary Value
Testing
BVT: “Execution of specific input pairs in
order to ensure that an actual boundary
is also expected.”

30. 30/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Boundary Value
Analysis
Boundary Value
Testing
BVT: “Execution of specific input pairs in
order to ensure that an actual boundary
is also expected.”

31. 31/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
BVA: “Analysis of artifacts of the software
development process to clarify the expected
and actual boundaries of a software.”
Boundary Value
Analysis
Boundary Value
Testing
BVT: “Execution of specific input pairs in
order to ensure that an actual boundary
is also expected.”

32. 32/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Boundary Value
Analysis
Boundary Value
Testing
Boundary Value
Exploration
Candidates

33. 33/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Boundary Value
Analysis
Boundary Value
Testing
Boundary Value
Exploration
Candidates

34. 34/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Boundary Value
Analysis
Boundary Value
Testing
Boundary Value
Exploration
Candidates
BVE: “A collection of techniques that select
or help select inputs to detect and identify
boundary candidates.”

35. 35/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Boundary Value
Analysis
Boundary Value
Testing
Boundary Value
Exploration
Candidates
BVE: “A collection of techniques that select
or help select inputs to detect and identify
boundary candidates.”

36. 36/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
byte-count

37. 37/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
byte-count
bytecount(int) → human readable string

38. 38/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
byte-count
bytecount(int) → human readable string

39. 39/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
byte-count
bytecount(int) → human readable string

40. 40/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints
BVE Process Example

41. 41/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints

42. 42/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
mining

43. 43/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
mining
bc(999999) → “1000.0 kB”
bc(1000000) → “1.0 MB”

44. 44/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
mining
bc(999949) → “999.9 kB”
bc(999950) → “1.0 MB”
“clear case”
bc(999999) → “1000.0 kB”
bc(1000000) → “1.0 MB”

45. 45/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
mining
bc(99999…...16) → BoundsError(“kMGTPE”, 9)
bc(99999…...17) → BoundsError(“kMGTPE”, 10)
bc(999949) → “999.9 kB”
bc(999950) → “1.0 MB”
“clear case”
not so clear case...
bc(999999) → “1000.0 kB”
bc(1000000) → “1.0 MB”

46. 46/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
mining
bc(99999…...16) → BoundsError(“kMGTPE”, 9)
bc(99999…...17) → BoundsError(“kMGTPE”, 10)
bc(999949) → “999.9 kB”
bc(999950) → “1.0 MB”
“clear case”
not so clear case...
bc(999999) → “1000.0 kB”
bc(1000000) → “1.0 MB”
Specification Pull Request:

47. 47/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
mining
bc(99999…...16) → BoundsError(“kMGTPE”, 9)
bc(99999…...17) → BoundsError(“kMGTPE”, 10)
Standardized
Specification Syntax:
bc(999949) → “999.9 kB”
bc(999950) → “1.0 MB”
“clear case”
not so clear case...
bc(999999) → “1000.0 kB”
bc(1000000) → “1.0 MB”
Specification Pull Request:

48. 48/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Specification Pull Request Update:
Bc(1_000_000...000) → last valid
Bc(1_000_000...001) → ArgumentError(“out of bounce”)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
mining
bc(99999…...16) → BoundsError(“kMGTPE”, 9)
bc(99999…...17) → BoundsError(“kMGTPE”, 10)
Standardized
Specification Syntax:
bc(999949) → “999.9 kB”
bc(999950) → “1.0 MB”
“clear case”
testing → quality assurance
not so clear case...
bc(999999) → “1000.0 kB”
bc(1000000) → “1.0 MB”
Specification Pull Request:

49. 49/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Specification Pull Request Update:
Bc(1_000_000...000) → last valid
Bc(1_000_000...001) → ArgumentError(“out of bounce”)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
implementation
boundary
mining
bc(99999…...16) → BoundsError(“kMGTPE”, 9)
bc(99999…...17) → BoundsError(“kMGTPE”, 10)
Standardized
Specification Syntax:
bc(999949) → “999.9 kB”
bc(999950) → “1.0 MB”
“clear case”
testing → quality assurance
not so clear case...
bc(999999) → “1000.0 kB”
bc(1000000) → “1.0 MB”
Specification Pull Request:

50. 50/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Specification Pull Request Update:
Bc(1_000_000...000) → last valid
Bc(1_000_000...001) → ArgumentError(“out of bounce”)
program
Specification:
tests
inputs
outputs
requirements
constraints
boundary
implementation
boundary
mining
bc(99999…...16) → BoundsError(“kMGTPE”, 9)
bc(99999…...17) → BoundsError(“kMGTPE”, 10)
Standardized
Specification Syntax:
bc(999949) → “999.9 kB”
bc(999950) → “1.0 MB”
“clear case”
testing → quality assurance
not so clear case...
bc(999999) → “1000.0 kB”
bc(1000000) → “1.0 MB”
Specification Pull Request:

51. 51/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
How to automatically mine
boundaries?
●
Without oracle, or only partial information, only
having the SUT at hand.

52. Absolute vs. Relative Assessment
R. Feldt and F. Dobslaw.
"Towards automated boundary value testing with program
derivatives and search." International Symposium on Search
Based Software Engineering. Springer, Cham, 2019.

53. SUT
x y
Absolute vs. Relative Assessment
R. Feldt and F. Dobslaw.
"Towards automated boundary value testing with program
derivatives and search." International Symposium on Search
Based Software Engineering. Springer, Cham, 2019.

54. SUT
x y
Assert y == yexp
Absolute vs. Relative Assessment
R. Feldt and F. Dobslaw.
"Towards automated boundary value testing with program
derivatives and search." International Symposium on Search
Based Software Engineering. Springer, Cham, 2019.

55. SUT
x y
Assert y == yexp
Absolute vs. Relative Assessment
R. Feldt and F. Dobslaw.
"Towards automated boundary value testing with program
derivatives and search." International Symposium on Search
Based Software Engineering. Springer, Cham, 2019.

56. SUT
x y
x1 x2
y1 y2
SUT SUT
Assert y == yexp
Absolute vs. Relative Assessment
R. Feldt and F. Dobslaw.
"Towards automated boundary value testing with program
derivatives and search." International Symposium on Search
Based Software Engineering. Springer, Cham, 2019.

57. SUT
x y
x1 x2
y1 y2
?
SUT SUT
Assert y == yexp
Absolute vs. Relative Assessment
R. Feldt and F. Dobslaw.
"Towards automated boundary value testing with program
derivatives and search." International Symposium on Search
Based Software Engineering. Springer, Cham, 2019.

58. SUT
x y
x1 x2
y1 y2
?
SUT SUT
Assert y == yexp
Absolute vs. Relative Assessment
Distance Metric Examples: Jaccard, Hamming, Euclidean…
Very basic one: StringLength
R. Feldt and F. Dobslaw.
"Towards automated boundary value testing with program
derivatives and search." International Symposium on Search
Based Software Engineering. Springer, Cham, 2019.

59. SUT
x y
x1 x2
y1 y2
?
SUT SUT
Assert y == yexp
Absolute vs. Relative Assessment
Distance Metric Examples: Jaccard, Hamming, Euclidean…
Very basic one: StringLength
Example:
30 and 31 are neighbors in the input space for field month:
Date(2021, 30, 4) → “30/4/2021”
StringLength is 9
Date(2021, 31, 4) → “ERROR, month field out of bounds.”
StringLength is 33
Big difference, this seems to be a boundary candidate!
But is it an interesting one?...
R. Feldt and F. Dobslaw.
"Towards automated boundary value testing with program
derivatives and search." International Symposium on Search
Based Software Engineering. Springer, Cham, 2019.

60. 60/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
F. Dobslaw, F. G. de Oliveira Neto and R. Feldt,
"Boundary Value Exploration for Software Analysis,"
2020 IEEE International Conference on Software Testing, Verification
and Validation Workshops (ICSTW), 2020, pp. 346-353,
doi: 10.1109/ICSTW50294.2020.00062.

61. 61/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Interactive Boundary Exploration
F. Dobslaw, F. G. de Oliveira Neto and R. Feldt,
"Boundary Value Exploration for Software Analysis,"
2020 IEEE International Conference on Software Testing, Verification
and Validation Workshops (ICSTW), 2020, pp. 346-353,
doi: 10.1109/ICSTW50294.2020.00062.

62. 62/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
- Go to www.menti.com
- Enter the code
6546 248
- Answer 1 question.
Thanks!
Questions, Comments?

63. 63/63 --- Exploring the Boundaries of our Software, SAST 2021, Felix Dobslaw (felix.dobslaw@miun.se)
Please partake in an eye-tracking
study!
https://studenter.miun.se/~evth1400/EyeGazeStudy/?group=3
Link