This document discusses negative testing strategies for automating tests. It begins by defining negative testing as testing invalid or unexpected inputs to show how a system does not work. The document then provides examples of how writing negative tests can improve quality, coverage, stability and make applications more robust. Specific strategies are outlined, such as discussing negative acceptance criteria with users, separating tests into functional groups, and using feature flags to enable and disable features for different test types. The conclusion emphasizes changing to a more positive mindset about negative testing to achieve better outcomes.

2. How I automate negative tests
An anecdote of effective practices in actual project implementations

3. 198
8
200
5
200
9
201
4
201
6
201
7

5. • One of the market leaders in low
code platforms
• Help organizations to build and
improve apps in lightening speed
(months turn into weeks)
• In-house cloud support (and support
for major cloud vendors)
• Based out of Rotterdam, Boston,
and London
Mendix

6. Disclaimer
“Quality is value to some person”
Gerald (Jerry) M. Weinberg
“Testing is questioning a product in order to
evaluate it”
James Bach

7. • What is negative testing?
• The light-bulb moment
• Why negative testing is needed?
• How I write (better) negative tests: inspired by true events
• Recap: helpful practices
• Conclusion
• Q&A
Agenda

8. “testing the system by giving invalid data” – an accepted answer in
stackoverflow
“Tests aimed at showing that a component or system does not work.
Negative testing is related to the tester's attitude rather than a specific
test approach or test design technique, e.g., testing with invalid input
values or exceptions.” - ISTQB Glossary
Some negative test scenarios for testing a pen –
• Change medium to write-on
• Replace the refill with an empty one
• Put the pen in liquid and verify
What is Negative testing?

9. • It is different from positive testing
• Fragile and not straightforward
• Too improbable to some
• Sometimes the term is used as a guide word heuristic
• There is nothing negative in negative testing
The light-bulb moment

10. • Prevents application from crashing
• Finds defects – improved quality
• Increases testing coverage
• More stable and reliable application
• Ensures application handles both good and bad data well
Why negative testing is needed?
Good Input Process Good Output Bad Input Process Ok Output
☺ 😉

11. How I write (better) negative tests:
inspired by true events

12. • Ask end user examples for negative scenarios
• Makes application robust
• Sparks productive discussions within the team
• May result in new business rule(s) and/or recruitments
• Also talk in terms of What the product should not do...
Discuss negative acceptance criteria too

13. Send email with
repayment link
Flow diagram of online donation system

14. • Ask end user examples for negative scenarios
• Makes application robust
• Sparks productive discussions within the team
• May result in new business rule(s) and/or recruitments
• Also talk in terms of What the product should not do...
Discuss negative acceptance criteria too

15. Should
not show
exceptio
n
Flow diagram of online donation system

16. • The classic question applies here
• Depends on risk and frequency of use
• If costly to maintain – consider ROI
• Technically challenging – implement feature differently
• Write test friendly code – less fragile tests
What and what not to automate

17. • Should have clear test objective
• Should verify an user behavior
• Group tests into functional slices –
➢Separate test to check routing to different banks
➢Seperate tests to check payment method decisioning
• Group tests based on type –
➢granular functional tests
➢integration tests
➢E2E/chain tests
Separate tests with clear test objective

18. Flow diagram of online donation system

19. • Should have clear test objective
• Should verify an user behavior
• Group tests into functional slices –
➢Separate test to check routing to different banks
➢Seperate tests to check payment method decisioning
• Group tests based on type –
➢granular functional tests
➢integration tests
➢E2E/chain tests
Separate tests with clear test objective

20. Flow diagram of online donation system

21. • Should have clear test objective
• Should verify an user behavior
• Group tests into functional slices –
➢Separate test to check routing to different banks
➢Seperate tests to check payment method decisioning
• Group tests based on type –
➢granular functional tests
➢integration tests
➢E2E/chain tests
Separate tests with clear test objective

22. Flow diagram of online donation system

23. Scenario: Check cancel payment functionality in an online donation application
Set up:
➢Execute a head-less test automation script
➢Trigger a database script to populate the data
Actual test: Test only the payment cancelling feature
Test set up can be a hack
Fill in donor
details
Choose mode of
payment
Cancel payment

24. Negative testing exposes loose ends
Positive scenario of successful payment for an online donation system:
• Comments box is an optional field
• So, often not filled
Negative scenario of spamming the application:
• The same comments box is a key component
• The team may agree to change the behavior
– only available to authenticated user

25. Negative testing exposes loose ends
Positive scenario of successful payment for an online donation system:
• Comments box is an optional field
• So, often not filled
Negative scenario of spamming the application:
• The same comments box is a key component
• The team may agree to change the behavior
– only available to authenticated user

26. • Negative tests can result in false positives
False positives
Positive test like –
auth_user.should
have_comments(“. t-helper-
textarea”)
• Will pass if the comments section
is available
• Will fail if the comments section is
removed/changed
Negative test like –
guest_user.should_not
have_comments(“. t-helper-
textarea”)
• Will pass if the comments section is
removed
• Will pass even if class is renamed to
t-helper-comments in DOM
Positive Test Negative Test False positive

27. • Balance with DRY (Don’t Repeat Yourself) opposite
• Write adjacent helper methods
Possible solutions to false positives

28. Scenario: User is able to add and delete contribution cause
Technical assertion
Functionally un-automatable!

29. Flow diagram of online donation system (new

30. Scenario: User is able to add and delete contribution cause
Technical assertion
Functionally un-automatable!

31. Scenario: User is able to add and delete donation cause
Customized assertion
Customized assertions

32. Scenario: Check deletion of a particular object (split shapes) in a modeling tool
Test should not test “Test Data”
Say NO to these kind of tests Say YES to these kind of tests

33. Scenario: Validate length of donor’s first name
Implementation impacts negative test
automation
Explicit Tests
No Tests

34. • The examples are phrased out as what customers don’t want
• Convert examples to positive behavior
• A sample scenario can be –
Given we’ve gone live with the front page
When the president of USA resigns on the same day
Then the site shouldn’t go down under the weight of people reading that ne
• Negative scenarios are powerful - often related to interesting stories
Negative non-functional scenarios in BDD

35. • “Postive test bias” exists in TDD approach
Negative testing in TDD
Number of
Test Cases
28.67%
71.33%
Positive TCs Negative TCs
Number of
Defects
Found
71.15%
28.85%
Positive TCs Negative TCs
Source:
Title: Effects of Negative Testing on TDD: An Industrial
Experiment
Author: Adnan Causevic, Rakesh Shukla, Sasikumar
Punnekkat

36. Feature flag in negative testing
Usage of feature flag:
✓Early access
✓Run A/B tests
✓Newbie vs Power user
Example
A Feature Flag (or Feature Toggle) is the ability to easily turn on/off features
of an application.

37. • A test to check the donor and donation details are available in different tabs
only with feature flag
• A test to check the donor and donation details are not available in tabbed view
without feature flag
• Separate test(s) for detailed checks of the tabbed views
Feature flag is live

38. • A test to check the donor and donation details are available in different tabs
only with feature flag
• A test to check the donor and donation details are not available in tabbed view
without feature flag
• Separate test(s) for detailed checks of the tabbed views
Feature flag is removed

39. ✓ Discuss both positive and negative A/C
✓ Write test friendly code - less fragile tests
✓ Separate tests with clear test objective
✓ Focus on test, set up can be hack
✓ Negative testing exposes loose ends
✓ DRY opposite and adjacent helper
methods
✓ Customized assertions
✓ Test functionality not test data
✓ Implementation impacts
✓ Give positive twist to negative
non-functional BDD scenarios
✓ Positive test bias in TDD
✓ Feature flags in negative testin
Recap: helpful practices

40. × Testing strategy is based on “positive” testing
× Teams map tests to requirements
× Customers face issues related to untested conditions
× Customers used product in an unusual way
✓Testing a product against requirements is necessary but not enough
✓It’s more of a mindset change – everyone in team should be involved
✓More negative testing will lead to more positive outcomes
Time to get positive about negative testing

41. Thanks
tuhinmitra88@gmail.com Questions? @tuhinSub