Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck

Kevin Brockhoff
Deliver Faster with
BDD/TDD
Designing Automated Tests
That Don’t Suck

–Uncle Bob
“The only way to go fast is to go well.”

Accelerate
❖ Software Delivery Performance Metrics That Matter
❖ Deployment Frequency
❖ Lead Time for Changes
❖ Mean Time To Restore (MTTR)
❖ Change Failure Rate
Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
By Nicole Forsgren PhD, Jez Humble, and Gene Kim

Fast vs Quality
Annual State of DevOps Report sponsored by Puppet Labs

–Jerry Weinberg
“A system is never ﬁnished being developed until
it ceases to be used.”

–John Woods
“Always code as if the guy who ends up
maintaining your code will be a violent
psychopath who knows where you live.”

–Michael Feathers
“Legacy code is code without tests.”

Deliver Valuable Software Faster
Problem How TDD/BDD Can Help
Long QA Test Cycles Identify incorrect behavior before release to QA
Rework
Identify incorrect behavior during initial
development so can fix when most familiar with the
code
Disproportionate time spent studying code before
making small change
Confidence tests will identify if change will break
other behavior
Slow and difficult client development Verify API usability and tests as documentation
Slow and difficult to extend and/or add features
Testability strongly correlated with flexibility and
extensibility

–Chinese proverb
“To be uncertain is to be uncomfortable, but to be
certain is to be ridiculous.”

Testing Pyramid
Testing pyramid concept introduced in
Succeeding with Agile by Mike Cohn

End-to-End Tests
❖ Developers like it because it offloads
most, if not all, of the testing to others.
❖ Managers and decision-makers like it
because tests that simulate real user
scenarios can help them easily
determine how a failing test would
impact the user.
❖ Testers like it because they often
worry about missing a bug or writing
a test that does not verify real-world
behavior; writing tests from the user's
perspective often avoids both
problems and gives the tester a greater
sense of accomplishment.
Days Left Pass % Notes
1 5%
Everything is broken! Signing in to the service is
broken. Almost all tests sign in a user, so almost
all tests failed.
0 4%
A partner team we rely on deployed a bad build to
their testing environment yesterday.
-1 54%
A dev broke the save scenario yesterday (or the
day before?). Half the tests save a document at
some point in time. Devs spent most of the day
determining if it's a frontend bug or a backend
-2 54%
It's a frontend bug, devs spent half of today
figuring out where.
-3 54%
A bad fix was checked in yesterday. The mistake
was pretty easy to spot, though, and a correct fix
was checked in today.
-4 1%
Hardware failures occurred in the lab for our
testing environment.
-5 84%
Many small bugs hiding behind the big bugs (e.g.,
sign-in broken, save broken). Still working on the
small bugs.
-6 87%
We should be above 90%, but are not for some
reason.
-7 89.54%
(Rounds up to 90%, close enough.) No fixes were
checked in yesterday, so the tests must have
been flaky yesterday.
In Theory In Practice
Source: Mike Wacker on Google Testing Blog

–Fred Brooks
“The hardest part of building a software system is
deciding precisely what to build... No other part of
the work so cripples the resulting system if done
wrong. No other part is more difﬁcult to rectify
later”

Java Unit Test Frameworks
Strong Points Migrating from JUnit 4
JUnit 5
• Leverages Java 8 lambdas and
functional interfaces
• Modularity, powerful
extensions mechanism
• No support for Runners and
Rules
• Spring tests based on
ThreadLocal usage break
TestNG
• Flexible grouping and running
of tests
• Different ordering to
assertion parameters than
JUnit
Spock
• Data tables with multi-line
strings
• No need for mocking
framework
• Getting used to Groovy-
based DSL
• Additional test source
directory

Other Java Test Frameworks
Type Highlights
Mockito Mock Interface injection, Data control, Verification
Spring Test Dependency Inject Spring config doc, Transaction rollback
Arquillian JEE Server Stub Embedded JEE server functionality
Cucumber BDD Gherkin-based communication with SME
Wiremock Service Virtual Lightweight service stubs
Spring Cloud
Contract
Consumer Driven Uses Wiremock under the hood
Hoverfly Service Virtual Full system service virtualization
ArchUnit Architecture Consistent -ilities w/o bureaucracy

–Ben Franklin
“The bitterness of poor quality remains long after
the sweetness of low price is forgotten.”

Test Source Layout
❖ Maven
❖ All in src/test/java or src/test/groovy
❖ Use surefire and failsafe include / exclude to control test execution (Unit = *Test, Integration = *IT)
❖ Define non-default profile for integration tests
❖ Use jacoco-maven-plugin to merge test coverage
❖ Gradle
❖ Utilize gradle-testsets-plugin and don’t link into default build
❖ Separate directories (Unit = src/test/java, Integration = src/integrationTest/java)
❖ Use Jacoco plugin to merge test coverage
❖ Sonar
❖ Automatically merges all test coverage in one metrics as of v6.x

–Aristotle
“Quality is not an act, it is a habit.”

When to Unit Test
- Questionable
- Code obvious at ﬁrst glance
- Coordinating logic
✓ Absolutely Mandatory
✓ Algorithmic logic
✓ Complex code with many
dependencies
Vikas Hazrati on Knoldus blog

What to Test
Category Test Category Test
DTO
Serialization, equals,
hashCode
DAO/Repository
Integration tests against
actual datastore
Converters, Parsers,
Serializers, Validators
Source data from test data
parameters or files on class
path
Utility Methods Every branch, edge cases
Routers, Coordinators
Decision logic if any
extracted into methods
Business Rules
BDD specifications, All pairs
testing
JavaEE EJBs, JCAs Use Arquillian REST Controllers
Handling of optional
parameters, Use REST
framework-provided mocks
APIs
Integration tests verifying
wiring and configuration of
framework AOP
UIs Basic wiring, algorithms
Docker Images
Wiring, startup, health
check
Third-party Libraries
Integration of generated
code, Behavior assumptions

Data Transfer Object
public class ExampleEntityPKTest {
private final Logger logger = LoggerFactory.getLogger(ExampleEntityPKTest.class);
@Test
public void shouldSerializeAndDeserializeRetainingEquality() {
String employeeId = "99377";
String postingId = "DEC18";
ExampleEntityPK original = new ExampleEntityPK();
original.setPostingId(postingId);
original.setEmployeeId(employeeId);
ExampleEntityPK copy = (ExampleEntityPK)
SerializationUtils.deserialize(SerializationUtils.serialize(original));
assertEquals(original, copy);
assertEquals(original.hashCode(), copy.hashCode());
assertEquals(postingId, copy.getPostingId());
assertEquals(employeeId, copy.getEmployeeId());
}
@Test
public void shouldTreatObjectsWithDifferentValuesAsNotEqual() {
ExampleEntityPK pk1 = new ExampleEntityPK("99377", "JUL19TEST");
assertEquals(pk1, pk1);
assertFalse(pk1.equals(pk2));
assertFalse(pk1.hashCode() == pk2.hashCode());
assertFalse(pk1.equals(null));
assertFalse(pk1.equals(pk1.getPostingId()));
}
@Test
public void shouldSortByPostingIdAndThenEmployeeId() {
ExampleEntityPK pk2 = new ExampleEntityPK(“11", "JUL19TEST");
assertTrue(pk1.compareTo(pk2) > 0);
}
}

Data Access Object
@RunWith(SpringRunner.class)
@ContextConfiguration(classes = { DatabaseConfig.class })
@TestPropertySource("/test-application.properties")
public class EmployeeRepositoryITest {
@Autowired
private EmployeeRepository repository;
@Autowired
private DataSource dataSource;
@Test
public void shouldRetrievePageOfEmployees() {
PageRequest pageable = PageRequest.of(0, 32, Sort.Direction.ASC, "employeeId");
Page<Employee> results = repository.findAll(pageable);
assertFalse(results.getContent().isEmpty());
}
@Test
public void shouldRetrieveEmployeeByEmployeeId() {
PageRequest pageable = PageRequest.of(4, 4, Sort.Direction.ASC, "employeeId");
Page<Employee> results = repository.findAll(pageable);
assertFalse(results.getContent().isEmpty());
for (Employee em : results.getContent()) {
Optional<Employee> result = repository.findByEmployeeId(em.getEmployeeId());
assertTrue(result.isPresent());
}
}
@Test
public void shouldRetrieveValuesFromPersonnel() {
Optional<Employee> result = repository.findById(retrieveValidEmployeeId());
assertTrue(result.isPresent());
assertNotNull(result.get().getSeniorityDate());
}
private Long retrieveValidEmployeeId() {
Long result = null;
try (Connection conn = dataSource();
PreparedStatement ps = conn.prepareStatement("SELECT MAX(employee_id) FROM employee")) {
try (ResultSet rs = ps.executeQuery()) {
rs.next();
result = rs.getLong(1);
}
} catch (SQLException cause) {
throw new IllegalStateException(cause);
}
return result;
}
}

Converters
public class EmployeeConverterTest {
@Test
public void shouldConvertFullyPopulatedEntity() {
EmployeeConverter converter = new EmployeeConverter();
EmployeeEntity source = loadEmployeeEntity(“employee-fully-populated.json”);
Employee target = converter.convert(source);
assertEquals(source.getEmployeeId(), target.getEmployeeId());
assertEquals(source.getName(), target.getEmployeeName());
}
@Test
public void shouldConvertNullEntity() {
EmployeeConverter converter = new EmployeeConverter();
EmployeeEntity source = null;
Employee target = converter.convert(source);
assertNull(target);
}
private EmployeeEntity loadEmployeeEntity(String fileName) {
EmployeeEntity entity = null;
ObjectMapper objectMapper = new ObjectMapper();
try (InputStream stream = getClass().getResourceAsStream(fileName)) {
entity = objectMapper.readValue(stream, EmployeeEntity.class);
} catch (IOException cause) {
throw new IllegalStateException(cause);
}
return entity;
}
}

Utility Methods
public class IdentifierUtilsTest {
private final Logger logger = LoggerFactory.getLogger(IdentifierUtilsTest.class);
@Test
public void shouldExtractEmployeeIdFromValidIds() {
String[] pathIds = { "947334.APR19", "947334.APR%202019%20X", };
for (String pathId : pathIds) {
String employeeId = IdentifierUtils.extractEmployeeIdFromPathId(pathId);
logger.info("{}", employeeId);
assertTrue(StringUtils.hasText(employeeId));
}
}
@Test
public void shouldExtractPostingIdFromValidIds() {
String[] pathIds = { "947334.APR19", "947334.APR%202019%20X", "947334.APR19.x", };
String postingId = IdentifierUtils.extractPostingIdFromPathId(pathId);
logger.info("{}", postingId);
assertTrue(StringUtils.hasText(postingId));
}
}
@Test
public void shouldThrowExceptionForEmployeeIdFromInvalidIds() {
String[] pathIds = { "947334", "947334.", "947334-APR19.", };
try {
String employeeId = IdentifierUtils.extractEmployeeIdFromPathId(pathId);
assertTrue(StringUtils.hasText(employeeId));
fail("should have thrown exception");
} catch (IllegalArgumentException exception) {
logger.info("{}", exception.getMessage());
}
}
}
@Test
public void shouldThrowExceptionForPostingIdFromInvalidIds() {
String[] pathIds = { "947334", "947334.", "947334-APR19.", };
try {
String postingId = IdentifierUtils.extractPostingIdFromPathId(pathId);
assertTrue(StringUtils.hasText(postingId));
fail("should have thrown exception");
} catch (IllegalArgumentException exception) {
logger.info("{}", exception.getMessage());
}
}
}
}

Business Rules
public class RpnCalculatorStepdefs implements En {
private RpnCalculator calc;
public RpnCalculatorStepdefs() {
Given("a calculator I just turned on", () -> {
calc = new RpnCalculator();
});
When("I add {int} and {int}", (Integer arg1, Integer arg2) -> {
calc.push(arg1);
calc.push(arg2);
calc.push("+");
});
Given("I press (.+)", (String what) -> calc.push(what));
Then("the result is {double}", (Integer expected) -> assertEquals(expected, calc.value()));
Then("the result is {int}", (Integer expected) -> assertEquals(expected.doubleValue(), calc.value()));
Before(new String[]{"not @foo"}, (Scenario scenario) -> {
scenario.write("Runs before scenarios *not* tagged with @foo");
});
After((Scenario scenario) -> {
// result.write("HELLLLOO");
});
Given("the previous entries:", (DataTable dataTable) -> {
List<Entry> entries = dataTable.asList(Entry.class);
for (Entry entry : entries) {
calc.push(entry.first);
calc.push(entry.second);
calc.push(entry.operation);
}
});
}
static final class Entry {
private final Integer first;
private final Integer second;
private final String operation;
Entry(Integer first, Integer second, String operation) {
this.first = first;
this.second = second;
this.operation = operation;
}
}
}

All-Pairs Testing
❖ For each pair of input parameters, tests all possible
discrete combinations of those parameters
❖ The most common bugs in a program are generally
triggered by either a single input parameter or an
interaction between pairs of parameters
❖ Software available to generate optimum test data

EJB
@RunWith(Arquillian.class)
public class MemberRegistrationTest {
@Deployment
public static Archive<?> createTestArchive() {
return ShrinkWrap.create(WebArchive.class, "test.war")
.addClasses(Member.class, MemberRegistration.class, Resources.class)
.addAsResource("META-INF/test-persistence.xml", "META-INF/persistence.xml")
.addAsWebInfResource(EmptyAsset.INSTANCE, "beans.xml")
// Deploy our test datasource
.addAsWebInfResource("test-ds.xml", "test-ds.xml");
}
@Inject
MemberRegistration memberRegistration;
@Inject
Logger log;
@Test
public void testRegister() throws Exception {
Member newMember = new Member();
newMember.setName("Jane Doe");
newMember.setEmail("jane@mailinator.com");
newMember.setPhoneNumber("2125551234");
memberRegistration.register(newMember);
assertNotNull(newMember.getId());
log.info(newMember.getName() + " was persisted with id " + newMember.getId());
}
}

Concurrency
public class MicrometerTestApplicationTests {
private Logger logger = LoggerFactory.getLogger(MicrometerTestApplicationTests.class);
@Autowired
private WebTestClient webClient;
@Test
public void threadPoolResponseDistribution() throws BrokenBarrierException, InterruptedException {
int clients = 8;
int runs = 2048;
CyclicBarrier barrier = new CyclicBarrier(clients + 1);
ExecutorService executorService = Executors.newCachedThreadPool();
for (int i = 0; i < clients; i++) {
executorService.submit(new ThreadPoolTestRunner(barrier, runs));
}
barrier.await();
barrier.await();
webClient.get().uri("/actuator/mappings").exchange().expectStatus().isOk();
webClient.get().uri("/actuator/prometheus").exchange().expectStatus().isOk();
FluxExchangeResult<String> stats = webClient.get().uri("/stats/tpool").exchange().returnResult(String.class);
logger.info("Thread Pool Test Result: {}", new String(stats.getResponseBodyContent()));
}
FluxExchangeResult<String> stats = webClient.get().uri("/stats/hystrix").exchange().returnResult(String.class);
logger.info("Hystrix Test Result: {}", new String(stats.getResponseBodyContent()));
}
class ThreadPoolTestRunner implements Runnable {
private CyclicBarrier barrier;
private int runs;
ThreadPoolTestRunner(CyclicBarrier barrier, int runs) {
this.barrier = barrier;
this.runs = runs;
}
@Override
public void run() {
try {
barrier.await();
for (int i = 0; i < runs; i++) {
webClient.get().uri("/tpool/{id}", UUID.randomUUID().toString()).exchange().expectStatus().isOk();
}
barrier.await();
} catch (InterruptedException | BrokenBarrierException ignore) {
fail("improperly configured test");
}
}
}

–Andrew Hunt and Dave Thomas
“It’s not at all important to get it right the ﬁrst
time. It’s vitally important to get it right the last
time.”

Testability Aphorisms
❖ Static getInstance involving a remote connection is EVIL
❖ PowerMock provides powerful functionality which should
NEVER be used
❖ PowerMock required == Redesign required
❖ God objects need the Single Responsibility Principle
applied
❖ In method section comments should be sub-method names
❖ Minimize leaky abstractions

–powermock.github.io
“Please note that PowerMock is mainly intended
for people with expert knowledge in unit testing.
Putting it in the hands of junior developers may
cause more harm than good.”

–George Santayana
“Those who cannot remember the past are
condemned to repeat it.”

Test Automation Anti-Patterns
1. Mushrooming
• Stage 1: Small and Localized
• Stage 2: Generalization
• Stage 3: Stafﬁng
• Stage 4: Development of Non-Core Features
• Stage 5: Overload
2. The Competition
• Variety 1: Different teams use different frameworks
• Variety 2: Different teams feed requirements to enterprise effort and one team gets favored
• Variety 3: Mushrooming at stage 5 so team starts new solution at stage 1
3. The Night Time Fallacy
• Test Automation Truism: Machines create work for more testers
• The Tragedy of the Commons: Five teams each assume 12 hours to test time: 5 teams * 12 hours = 60 hours
4. Going for the Numbers
• Point and click creation to up numbers but poorly engineered for resiliency
5. The Sorcerer’s Apprentice Syndrome
• Mimicking human actions sound straight forward, but is sometimes hard to accomplish programmatically and is frequently
inefﬁcient.
The Pathologies of Failed Test Automation Projects - Michael Stahl, July 2013

Test Data Strategies
❖ Use actual production data
❖ PII, PCI, HIPAA data must be masked
❖ Use queries to get valid test values
❖ Use automatic rollback for data changing tests
❖ Delete and refresh on a regular basis
❖ Generate fake data
❖ Domain speciﬁc generators
❖ Generate per test
❖ Import reference data
❖ Periodic cleanup

–Kaner, Bach, Pettichord
“Testers don’t like to break things; they like to
dispel the illusion that things work.”

Future of Enterprise QA
❖ Embedded in delivery team
❖ Gherkin speciﬁcation development
❖ Exploratory testing
❖ Risk-based test prioritization
❖ Independent group
❖ AI-assisted intelligence
❖ Customer experience optimization and insights
❖ Robotic process automation

–Jerry Weinberg
“You can be a great tester if you have
programming skills. You can also be a great tester
if you have no programming skills at all. And, you
can be a lousy tester with or without programming
skills. A great tester will learn what skills she needs
to continue to be great, in her own style.”

Testing in Production Methodologies
❖ A/B Testing (aka Online Controlled Experimentation)
❖ Some percent of users of a website or service are unbeknownst to them given an alternate
experience (a new version of the service). Data is then collected on how users act in the old versus
new service, which can be analyzed to determine whether the new proposed change is good or not.
❖ Ramped Deployment
❖ Using Exposure Control, a new deployment of a website or service is gradually rolled out. First to a
small percentage of users, and then ultimately to all of them. At each stage the software is
monitored, and if any critical issues are uncovered that cannot be remedied, the deployment is
rolled back.
❖ Shadowing
❖ The system under test is deployed and uses real production data in real-time, but the results are not
exposed to the end user.
Source: https://blogs.msdn.microsoft.com/seliot/2011/06/07/testing-in-production-tip-it-really-happensexamples-from-facebook-amazon-google-and-microsoft/

–Isaac Asimov
“The most exciting phrase to hear in science, the
one that heralds discoveries, is not ‘Eureka!’ but
‘Now that’s funny…”

Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck

Similar to Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck (20)

Recently uploaded

Recently uploaded (20)

Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck