Mutation testing involves making small changes to code to test whether tests can detect those changes. It is more effective than just measuring test coverage at finding flaws in tests. Tools like Pitest and Descartes can perform mutation testing. Mutations may change conditionals, remove lines, or replace method bodies. Mutation testing is slower but can find cases where tests rely on implementation details. It is best run locally during development and as part of PRs, potentially failing PRs with too low mutation scores. This helps improve test quality over time.

1. Mutation testing
Too good to be true?
@pkubowicz
Piotr Kubowicz
nexocode

2. ➔ Java/JVM since 2007
➔ now: backend Kotlin
$ whois piotr-kubowicz

4. ThoughtWorks
Our teams have used Pitest for a while now, and we
recommend its use in Java projects to measure the
health of the test suite.
Technology Radar, Oct 2020

5. Hype for Mutation Testing

6. What is it?

7. fun canPayFrom(account: Account, amount: Int): Boolean =
account.balance > amount

8. fun canPayFrom(account: Account, amount: Int): Boolean =
account.balance > amount
@Test
fun `can pay more`() {
assertThat(canPayFrom(Account(10), 1)).isTrue()
}
@Test
fun `cannot pay less`() {
assertThat(canPayFrom(Account(10), 20)).isFalse()
}
100% coverage ✓

9. has coverage != well tested

10. Mutation
fun canPayFrom(account: Account, amount: Int): Boolean =
account.balance > amount
fun canPayFrom(account: Account, amount: Int): Boolean =
account.balance >= amount
tests pass ⚠
☢

11. Example mutations
➔ conditional boundary: > becomes >=
➔ a && b becomes a && !b
➔ remove line
➔ return null/false/0

12. Tools
➔ Java/JVM: Pitest (2013)
➔ JS, .NET: Stryker
➔ LLVM/C/C++: Mull
➔ Ruby: Mutant
➔ PHP: Infection

13. Pitest

14. Run steps
1. Collect coverage
2. Determine what to mutate and how
3. Execute each mutation
a. load modified class → execute covering tests
b. killed mutation = some test failed; otherwise survived

15. Testing approach vs. mutation testing
➔ test 1:1 code, isolated ☢👍
➔ test high-level API, use DB ☢👎

16. 10-30x slower

17. ⬅
⬅

18. Gregor
➔ built-in
Descartes
➔ external lib

19. Gregor
➔ built-in
➔ mutates expressions, lines
or whole blocks
Descartes
➔ external lib
➔ Extreme Mutation
Testing: replaces whole
method body

20. return account.balance > amount
Gregor
return account.balance >= amount
Descartes
return true

21. Gregor
➔ doesn't understand Kotlin
Descartes
➔ does not mutate getters
➔ respects nullability

22. Detecting bad tests

23. Use case
Send reminders about invoices not paid in time
➔ don't notify about PAID, CANCELLED
➔ include invoice ID

24. Really bad test
whenever(invoiceRepository.findAllByStatusNotIn(any())).thenReturn(
Flux.just(Invoice("i1", 151, WEEK_AGO)) // Flux == reactive list
)
overdueInvoiceNotificationJob.sendNotifications()
verify(invoiceRepository).findAllByStatusNotIn(any())
verify(notificationSender).sendNotification(any())

25. whenever(invoiceRepository.findAllByStatusNotIn(any()
)).thenReturn(
Flux.just(Invoice("i1", 151, WEEK_AGO))
)
overdueInvoiceNotificationJob.sendNotifications()
verify(invoiceRepository).findAllByStatusNotIn(any())
verify(notificationSender).sendNotification(any())
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification(
"${invoice.status} is overdue!"
)
}

26. whenever(invoiceRepository.findAllByStatusNotIn(any()
)).thenReturn(
Flux.just(Invoice("i1", 151, WEEK_AGO))
)
overdueInvoiceNotificationJob.sendNotifications()
verify(invoiceRepository).findAllByStatusNotIn(any())
verify(notificationSender).sendNotification(any())
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification(
"${invoice.status} is overdue!"
)
}

27. whenever(invoiceRepository.findAllByStatusNotIn(any()
)).thenReturn(
Flux.just(Invoice("i1", 151, WEEK_AGO))
)
overdueInvoiceNotificationJob.sendNotifications()
verify(invoiceRepository).findAllByStatusNotIn(any())
verify(notificationSender).sendNotification(any())
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification(
"${invoice.status} is overdue!"
)
}

28. whenever(invoiceRepository.findAllByStatusNotIn(any()
)).thenReturn(
Flux.just(Invoice("i1", 151, WEEK_AGO))
)
overdueInvoiceNotificationJob.sendNotifications()
verify(invoiceRepository).findAllByStatusNotIn(any())
verify(notificationSender).sendNotification(any())
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification(
"${invoice.status} is overdue!"
)
}

29. whenever(invoiceRepository.findAllByStatusNotIn(any()
)).thenReturn(
Flux.just(Invoice("i1", 151, WEEK_AGO))
)
overdueInvoiceNotificationJob.sendNotifications()
verify(invoiceRepository).findAllByStatusNotIn(any())
verify(notificationSender).sendNotification(any())
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification(
"${invoice.status} is overdue!"
)
}

30. 4, 3, 2, 1, 0…

31. Pitest default
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification("${invoice.status} is overdue!")
}

32. Gregor, mutators=STRONGER
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification("${invoice.status} is overdue!")
}

33. Descartes
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification("${invoice.status} is overdue!")
}
⬅

34. Gregor, mutators=ALL
fun sendNotifications() {
invoiceRepository.findAllByStatusNotIn(listOf(PAID))
.filter { isOverdue(it, clock) }
.next()
.doOnNext { sendNotification(it) }
.subscribe()
}
private fun sendNotification(invoice: Invoice) {
notificationSender.sendNotification("${invoice.status} is overdue!")
}
⬅

35. Blind spots
➔ bad values in a constant list
◆ listOf(PAID, CANCELLED)
➔ bad field or constant
◆ dto.timeFinished = entity.timeStarted
➔ fluent APIs (reactive programming)
◆ .next()

39. Not useful mutations

40. Comparing with our tools
➔ community size
➔ language support (Kotlin)
➔ context-based mutations
➔ code review integration

41. How to apply

42. 10–30x
80%

43. When to run
➔ locally during development

44. When to run
➔ locally during development slow
➔ report in PR

45. When to run
➔ locally during development slow
➔ report in PR false positives and blind spots
➔ fail PR if too low

46. When to run
➔ locally during development slow
➔ report in PR false positives and blind spots
➔ fail PR if too low wasting effort
➔ very low test quality?

48. Thank you!
Piotr Kubowicz
/in/pkubowicz | @pkubowicz
github.com/pkubowicz/pitest-gradle-sandbox
pitest.org | stryker-mutator.io
testing.googleblog.com/2021/04/mutation-testing.html