Lightning talk I gave at the London PyData Meetup in October 2015. In it I describe an approach to generate huge amounts of tests by automatically generating tests based on test data. The code is available at: https://github.com/bolsote/ukpcre

1. TESTING WITHOUT
WRITING TESTS
HAVING FUN WITH POSTCODES AND LIES
VICTOR MUNOZ

2. WHAT'S IN A
POSTCODE?

3. A typical British postcode has a clearly defined
structure...
...but many possible variations
Writing a correct regular expression is feasible, but tricky
So, so many cases to test against

4. W1 6LS
N1C 4AQ
N17 6LA
SW8 1UQ
WC2H 7LT
HU12 0AR

5. THE BAD, THE WRONG
AND THE UGLY

6. There is a list of official rules
Even a (wrong) regexp in an official document

7. THE WRONG
^([Gg][Ii][Rr] 0[Aa]{2}) |
(
(
([A-Za-z][0-9]{1,2}) |
(
([A-Za-z][A-Ha-hJ-Yj-y][0-9]{1,2}) |
(
([AZa-z][0-9][A-Za-z]) |
([A-Za-z][A-Ha-hJ-Yj-y][0-9]?[A-Za-z])
)
)
) [0-9][A-Za-z]{2})$
No, this is not LISP. Promise.

8. THE BAD
^(
([gG][iI][rR] {0,}0[aA]{2}) |
(
(
([a-pr-uwyzA-PR-UWYZ][a-hk-yA-HK-Y]?[0-9][0-9]?) |
(
([a-pr-uwyzA-PR-UWYZ][0-9][a-hjkstuwA-HJKSTUW]) |
([a-pr-uwyzA-PR-UWYZ][a-hk-yA-HK-Y][0-9][abehmnprv-yABEHMNPRV-Y])
)
) {0,}[0-9][abd-hjlnp-uw-zABD-HJLNP-UW-Z]{2}
)
)$

9. THE UGLY
^(?P<postcode>
(?P<girobank>girs*0aa) | # Girobank.
(
(?P<first>
((?![qvx])[a-z][a-hk-y]?[0-9][0-9]?) | # First part.
((?![qvx])[a-z][0-9][a-hjkpstuw]) | # First part.
((?![qvx])[a-z][a-hk-y][0-9][abehmnprvwxy]) # First part.
)s*(?P<second>[0-9](?![cikmov])[a-z]{2})? # Second part.
)
)$
...but it's my ugly.

10. HOW DO I TEST THAT?

11. Carefully
Create lots of cases
Cover all possibilities of success and failure
TL; DR: Hell

12. CAN I HAVE SOME
MAGIC?

13. Since you ask nicely...
Let's use pytest to generate tests from the test dataset
Note for testing nerds: Yes, this is just a semi-magic form
of table-driven tests.

14. TESTING STRATEGY (I)
1. Check if the postcode matches
2. Check it matches the right thing
3. Check the first part is matched correctly
4. Check the second part is matched correctly

15. class ValidPostcodeTests:
def test_match(self, data, postcode, first, second):
assert pattern.match(data)
def test_groups(self, data, postcode, first, second):
assert pattern.match(data).group('postcode') == postcode
assert pattern.match(data).group('first') == first
assert pattern.match(data).group('second') == second

16. TESTING STRATEGY(II)
For invalid postcodes, just check it doesn't match

17. class TestInvalidPostcode:
def test_match(self, data, postcode, first, second):
assert pattern.match(data) is None

18. TESTING STRATEGY (III)
Test for variations as well:
Lowercase postcodes
Postcodes without space separator
Partial postcodes
Any combination of those

19. transforms = {
"lowercase": partial(
_transform_factory,
lambda k: k.lower() if k else k
),
"nospace": partial(
_transform_factory,
lambda k: k.replace(' ', '') if k else k
),
}

20. Once you have all the combinations, generate tests.
class DataGenerator:
def generate_data(self):
for item in self.dataset:
for transform in self._get_transform_combinations():
yield self._transform_data(item, *transform)
def generate_ids(self):
for item in self.dataset:
for transform in self._get_transform_combinations():
yield "{}-{}".format(
item[0].replace(" ", "_"),
"_".join(transform) or "original"
)

21. Finally, hook that into pytest
def pytest_generate_tests(metafunc):
if metafunc.cls:
gen = DataGenerator(metafunc.cls.postcodes)
metafunc.parametrize(
'data,postcode,first,second',
list(gen.generate_data()),
ids=list(gen.generate_ids()),
scope='class'
)

22. Your test cases should look like this:
class TestFullPostcodes(ValidPostcodeTests):
postcodes = (
("W1 6LS", "W1 6LS", "W1", "6LS",),
("N1C 4UQ", "N1C 4UQ", "N1C", "4UQ",),
("N1P 4AA", "N1P 4AA", "N1P", "4AA",),
("N17 6LA", "N17 6LA", "N17", "6LA",),
("SW8 1UQ", "SW8 1UQ", "SW8", "1UQ",),
("CW3 9SS", "CW3 9SS", "CW3", "9SS",),
("SE5 0EG", "SE5 0EG", "SE5", "0EG",),
("WC2H 7LT", "WC2H 7LT", "WC2H", "7LT",),
("WC1N 2PL", "WC1N 2PL", "WC1N", "2PL",),
("HU12 0AR", "HU12 0AR", "HU12", "0AR"),
)

23. I WANT TO COVER
EVERYTHING

24. OS CODEPOINT
1.7M+ postcodes
Covers pretty much everything

25. First, extract the useful bits in a useful format
def get_postcodes(datadir):
for fn in listdir(datadir):
with open(datadir + fn) as f:
for row in reader(f):
yield row[0]
if __name__ == '__main__':
datadir = 'tests/data/Data/CSV/'
postcodes = list(get_postcodes(datadir))
with open("tests/OSpostcodes.db", "wb") as f:
pickle.dump(postcodes, f, pickle.HIGHEST_PROTOCOL)

26. Then, generate millions of tests
def pytest_generate_tests(metafunc):
if pytest.config.getoption("osdb"):
with open("tests/OSpostcodes.db", "rb") as f:
metafunc.parametrize("postcode", pickle.load(f))
else:
metafunc.parametrize("postcode", pytest.skip([]))
def test_OS_postcodes(postcode):
assert pattern.match(postcode)

27. QUESTIONS?
COMMENTS?

28. victorm@marshland.es
@vmberti
github.com/bolsote/ukpcre