Shift-Left Testing: QA in a DevOps World by David Laulusa

1. Shift-Left Testing
QA in a DevOps World
David Laulusa
@dlaulusa
Feb 25, 2020

2. About Me
• Brigham Young University, BS/EE
• Princeton University, MS/EE
• 20+ yrs in industry
• Development/QA
• Startups, Large Enterprises, Permanent/Contract
• Telecommute
• DevOpsDays Columbus

3. Laulusa

4. Laulusa

5. Laulusa

6. Laulusa

7. Laulusa

8. Laulusa

10. Agenda
Inject quality earlier in the process

11. You cannot inspect quality
into a product.
- Harold F. Dodge

12. Inspection does not improve the quality,
nor guarantee quality. Inspection is too
late. The quality, good or bad, is already
in the product.
- William Deming

13. Quality is not an act.
It is a habit.
- Aristotle

14. Agenda
• Testing Principles
• Testing Partners
• Test

22. Continuous Delivery is the NEW Agile
Satisfy the customer through early and
continuous delivery of valuable software
- Agile Manifesto

23. SMALL BATCHES

26. F1
Design
Dev
Test
Deploy
F2
Design
Dev
Test
Deploy
B1
Design
Dev
Test
Deploy
B2
Design
Dev
Test
Deploy
F3
Design
Dev
Test
Deploy
B3
Design
Dev
Test
Deploy

29. Shift-Left Testing
Static Unit Integration End-to-End Acceptance

30. Bugs are cheaper when caught young.
- Larry Smith

32. TESTING PRINCIPLES

33. Testing a System
System
Expected Outputs
Inputs

34. Input Domain
• The input domain contains all possible inputs.
• Choose a finite set of values.
• Include valid, invalid and special values
• Include values that represent “normal use” (domain-specific knowledge)
• Test selection is about sampling the input space in a cost-effective manner.

35. EQUIVALENCE CLASSES

36. Equivalence Classes
• All members of an equivalence class contribute to the fault detection
in the same way.
• The combination of all equivalence classes must cover the entire
domain.
• Equivalence classes must not overlap.
• Select one value from each Equivalence Class.
• Choosing more than one value does not increase fault detection.

37. Range
• One class with values inside the range, and two with values outside
the range.
• If the range were 10 <= x <=20.
1.Equivalence Class 1: all numbers between 10 and 20, inclusive
2.Equivalence Class 2: all numbers less than 10
3.Equivalence Class 3: all numbers greater than 20
4.You could also do letters and special characters.

38. String
• At least one containing all legal strings and one containing all illegal
strings.
• For example, let fname: string be a variable to denote a first name.
1. Valid name {Sue}
2. Special characters {ε}
3. Number {7}
4. Too long {Too many characters so it would not be able to fit into
the database}

39. Enumeration
• Each value in a separate class.
• And then any other value.
• For example, consider auto_color ∈ {red, green , blue}.
• The following classes are generated, {{red}, {green}, {blue}, {blah}}

40. Array
• One class containing all legal arrays, one containing only the empty
array, and one containing arrays larger than the expected size.
• For example, consider int[] aName = new int [3].
• The following classes are generated:
1. {[]}
2. {[-10, 20]}
3. {[-9, 0, 12, 15]}

41. BOUNDARY VALUE
ANALYSIS

42. Boundary-Value Analysis
Programmers often make mistakes in processing values at and near
the boundaries of equivalence classes.
• min-1, min, min+1
• max-1, max, max+1
• 0, null, empty string

43. COMBINATORIAL TESTING

44. Combinatorial Testing
• Many defects only occur when a combination of inputs or events
occur that interact with each other.
• However, if you were to test every combination of inputs it may take
years especially with today’s complex systems.
• How do you choose which combination of inputs?
• Combinatorial testing is a technique for reducing the number of test
cases without drastically compromising functional coverage, in order
to get more ‘bang for the buck’.

45. Testing a System
X (1,2) Y (Q,R) Z (5,6)
System

46. Combinatorial Testing Where T = 2,
Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC2 1 R 5
TC3 1 Q 6
TC4 1 R 6
TC5 2 Q 5
TC6 2 R 5
TC7 2 Q 6
TC8 2 R 6

47. Test Case X Y Z
TC1 1 Q 5
TC2 1 R 5
TC3 1 Q 6
TC4 1 R 6
TC5 2 Q 5
TC6 2 R 5
TC7 2 Q 6
TC8 2 R 6
Combinatorial Testing Where T = 2,
Pairwise Testing

48. Test Case X Y Z
TC1 1 Q 5
TC2 1 R 5
TC3 1 Q 6
TC4 1 R 6
TC5 2 Q 5
TC6 2 R 5
TC7 2 Q 6
TC8 2 R 6
Combinatorial Testing Where T = 2,
Pairwise Testing

49. Combinatorial Testing Where T = 2,
Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC2 1 R 5
TC3 1 Q 6
TC4 1 R 6
TC5 2 Q 5
TC6 2 R 5
TC7 2 Q 6
TC8 2 R 6

50. Combinatorial Testing Where T = 2,
Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC4 1 R 6
TC6 2 R 5
TC7 2 Q 6
TC8 2 R 6

51. Combinatorial Testing Where T = 2,
Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC4 1 R 6
TC6 2 R 5
TC7 2 Q 6

52. Combinatorial Testing Where T = 2,
Pairwise Testing
Test Case X Y Z
TC1 1 Q 5
TC4 1 R 6
TC6 2 R 5
TC7 2 Q 6
• Going from 8 -> 4 isn’t that big a deal
• Reducing # tests 50% is not that big a deal either
• Reducing by hand is tedious

53. Android Options
Parameter Values
HardKeyboardHidden No, Undefined, Yes
KeyboardHidden No, Undefined, Yes
Keyboard 12Key, NoKeys, Qwerty, Undefined
NavigationHidden No, Undefined, Yes
Navigation Dpad, NoNav, Trackball, Undefined, Wheel
Orientation Landscape, Portrait, Square, Undefined
ScreenLayout_Long Mask, No, Undefined, Yes
ScreenLayout_Size Large, Mask, Normal, Small, Undefined
TouchScreen Finger, NoTouch, Stylus, Undefined
3*3*4*3*5*4*4*5*4 = 172,800 combinations

54. Combinatorial Testing to the Rescue
T # Tests % of Total (172k) % Defects Found
2 29 0.02% 76%
3 137 0.08% 95%
4 625 0.4% 97%
5 2532 1.5% 99%
6 9168 5.3% 100%
• The savings as a percentage of exhaustive testing depends on the
number of parameters and the # of values for the parameters.
• With larger systems the savings can be enormous

55. “There’s an App for that”
• AllPairs (perl)
• PICT (Microsoft)
• ACTS (NIST)

56. When Not to Use Combinatorial Testing
• When you really do have to test all combinations
• This assumes a uniform distribution across all
combinations.
• Some combinations are more important than others.
• Some combinations are impossible.

57. T-Way - Effective in Finding Those Pesky Bugs
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6
# Interacting Parameters Leading to Failure
Browser Bugs (194) Server Bugs (171)

58. Fly With Us – We found 90% of the defects!

59. DEPENDENCY INJECTION

60. Dependency Injection
In software engineering, dependency injection is a software design pattern
that implements inversion of control for resolving dependencies. A
dependency is an object that can be used (a service). An injection is the
passing of a dependency to a dependent object (a client) that would use it.

61. sub method {
my $self = shift;
my $obj = MyProject::Class->new();
my $x = $obj->get_something();
. . .
return($x);
}

62. sub method {
my $self = shift;
my $obj = shift;
my $x = $obj->get_something();
. . .
return($x);
}

63. PARTNER WITH
DEVELOPMENT

64. Grooming the Backlog
• Make sure that there are sane completion requirements
• Is it testable?

67. Why Do We Unit Test?
static OSStatus
SSLVerifySignedServerKeyExchange(SSLContext *ctx, bool isRsa, SSLBuffer
signedParams, uint8_t *signature, UInt16 signatureLen) {
OSStatus err;
...
if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
goto fail;
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;
goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
goto fail;
...
fail:
SSLFreeBuffer(&signedHashes);
SSLFreeBuffer(&hashCtx);
return err;
}

68. Why Do We Unit Test?
• It gives you confidence that your code works as you expect it to work.
• Easily refactor your code to make it cleaner and more efficient.
• Saves time because unit tests help prevent regressions from being introduced and
released.
• Once a bug is found, you can write a unit test for it, you can fix the bug, and the bug can
never make it to production again because the unit tests will catch it in the future. (shift left
testing)
• Implicit documentation

69. How Do We Unit Test?
• Unit tests test units in isolation, in a dedicated, controlled environment.
• Each unit test creates in own environment by instantiating only the classes
needed to execute one test, putting them in a known state, then it invokes the
method to be tested and verify the outcome.
• This verification is done by assertions on the behavior of the method (as opposed
to its implementation).
• Performing the verification on the behavior and not on the implementation is
important as this allows modifying the implementation without breaking the unit
tests, and therefore using the unit tests as a safety net for the modification.

70. What Do We Unit Test?
• Generally a method.
• Need to write tests for the public methods.
• Could write tests for private methods, but it means you might have
to re-write these tests when you re-factor.
• No need to write tests for trivial setter/getters.

71. Anatomy of a Unit Test
• Set up all conditions for testing.
• Call the method (or Trigger) being tested.
• Verify that the results are correct.
• Clean up modified records.

72. Unit Tests
• Make each test orthogonal (i.e., independent) to all the others
• Don’t make unnecessary assertions
• Test only one code unit at a time
• Mock out all external services and state
• Avoid unnecessary preconditions
• Unit tests should ideally take less than 5 min to run. If it approaches 10 min, then
we can break them up and run sets in parallel.
• Unit tests will run each time code is checked in.
• Builds cannot progress unless all unit tests pass.

73. A Test is not a unit test if:
• It talks to the database
• It communicates across the network
• It touches the file system
• It can't run at the same time as any of your other unit tests
• You have to do special things to your environment (such as editing
config files) to run it.

75. TEST-DRIVEN
DEVELOPMENT

76. Test-Driven Development
• write a failing test
• write just enough code to make it pass
• refactor the code to clean it up

77. TDD
• TDD is a design process, not a testing process.
• Unit tests help to shape your design.
• TDD is a robust way of designing software components (“units”) interactively so
that their behavior is specified through unit tests.
• TDD helps you to deliver software components that individually behave according
to your design.
• A suite of good unit tests is immensely valuable: it documents your design, and
makes it easier to refactor and expand your code while retaining a clear overview
of each component’s behavior.

79. REGRESSION TESTING

83. CODE COVERAGE

84. Code Coverage
• Code coverage is a measure used to describe the degree to which
the source code of a program is executed when a particular test
suite runs.
• A program with high code coverage, measured as a percentage,
has had more of its source code executed during testing which
suggests it has a lower chance of containing undetected software
bugs compared to a program with low code coverage.

85. Code Coverage
• Function coverage
• Statement coverage
• Branch coverage
• Conditional coverage

86. Code Coverage
sub foo {
my($x, $y) = @_;
if ($x eq ‘string’ && $y < 10) {
return 1;
}
else {
return 0;
}
}

87. Code Coverage
100%?

89. PROFILE YOUR CODE

90. Profiling Your Code
Profiling is a form of dynamic program analysis that can measure
things like:
• memory usage
• time complexity of a program
• usage of particular instructions
• frequency and duration of function calls

91. Profiling Your Code
• optimization
• algorithm
• loops
• expensive function calls, including system calls
• lazy building

92. PARTNER WITH
PRODUCT OWNER

93. USER ACCEPTANCE
TESTING

95. Acceptance Testing - Example
Test Specification:
• Free shipping a month before Christmas

96. Test Code
@Test
public void
orderLessThenAMonthBeforeChristmasShouldGiveZeroShipping() {
Calendar purchaseDate = GregorianCalendar.getInstance();
purchaseDate.set(Calendar.YEAR, 2014);
purchaseDate.set(Calendar.MONTH, Calendar.NOVEMBER);
purchaseDate.set(Calendar.DAY_OF_MONTH, 24);
int christmasEve = 24;
int expectedShipping = 0;
String expectedCurrency = “USD";
Date purchase = purchaseDate.getTime();
Shipping shipping = new Shipping(purchase, christmasEve);
int actualCost = shipping.getCost();
assertThat(actualCost, is(expectedShipping));
}

97. Cucumber - Layers
Execute Specification (Gherkin)
Step Definitions - Glue (47
languages)
System Under Test

98. Cucumber - Specification
Scenario: Free shipping a month before Christmas
Given a customer that Christmas is celebrated 24 of December
When a customer buys a book on 2014-12-10
Then the shipping cost should be $0

99. Cucumber - Step Definitions
@When("^a customer buys a book on (.*)$")
public void
buyBook(@Format("yyyy-MM-dd") Date purchaseDate)
throws Throwable {
this.purchaseDate = purchaseDate;
}

101. MUTATION TESTING

102. Mutation Testing
• White Box Testing
• Mutate (change) certain statements in the source code and check if
the test cases are able to find the errors.
• This is called killing the mutant. Test suites are measured by the
percentage of mutants that they kill.
• New tests can be designed to kill additional mutants.

104. Other Types of Testing
• Security Testing
• Performance Testing
• Compliance Testing
• Resiliency Testing

105. EXPLORATORY TESTING

108. PARTNER WITH
EXTERNAL
STAKEHOLDERS

109. INTER-SYSTEM TESTING

111. PARTNER WITH
OPERATIONS

113. Testing in Production
• Logging
• Server Monitoring
• Application Performance Monitoring
• Health Check

115. TEST

116. Burger World Burger:
How Many?:
Here: To Go:
Lettuce:
Tomato:
Pickle:
Onions:
Mayo:
Ketchup:
Mustard:
Instructions:
Order Cancel

117. QUESTIONS?

118. THANKS
David Laulusa
@dlaulusa