Given to Global Dev Study, a talk about Robust Python and User-Defined Types

1. Robust Python
Pat Viafore
July 2021

2. About Me

3. About Me
Started my career working on lightning detection systems

4. About Me
Worked on telecommunication systems for 9+ years

5. About Me
Now working for Canonical, building Ubuntu images for the cloud

6. What do these have in common?

7. About Me
Wrote code in 2007 -- was operational for at least 10 years

8. About Me
Wrote code in 2009 -- still operational; worked with code from 2001 (20 years old)

9. About Me
Working on a codebase that started in 2007 (14 years old)

10. I've worked on code that lives a long
time

11. Code will live a long time

12. Code will outlive your time working on it

13. Code will become legacy

14. Legacy Code
A codebase where you do not have direct communication
with the original authors

15. Why does this matter?

16. YOU

17. YOU
Tasks completed in
a month

18. Fast-forward a few years........

19. Future
Maintainer

20. Future
Maintainer
Tasks completed in
a month
?

21. Future
Maintainer
Tasks completed in
a month

22. Future
Maintainer
Tasks completed in
a month

23. You have a duty to deliver value in a timely
manner

24. You have a duty to make it easy for future
collaborators to deliver value in a timely
manner

25. Make it easy for your future maintainers

26. Do you want future maintainers to thank you
for your foresight ........

27. .......or curse your name?

28. Robustness

29. Robustness
A robust codebase is resilient and error-free in spite of constant
change

30. Your code will change

31. How do you make it easy for someone
working in your codebase?
What about someone you'll never meet?

32. The goal is to speed up future developers

33. They will not know your code as well as you
do

34. Detect errors through tooling, make your code
understandable, and make it hard to introduce
faults

35. Communicating
Intent

36. Ask yourself why you make the decisions in
code?

37. Why?
● For Loop vs. While Loop
● Lists vs. Sets
● Dataclasses vs. Classes
● Dictionaries vs. Classes

38. text = "This is some generic text"
index = 0
while index < len(text):
print(text[index])
index += 1

39. for character in text:
print(character)

40. The abstractions you choose communicate to
the future

41. Robust Python

42. Typechecking User Defined Types
Extensibility
Building a Safety
Net

43. Typechecking User Defined Types
Extensibility
Building a Safety
Net

44. User-Defined Types in Robust Python
● Enumerations
● Dataclasses
● Classes
● Subtyping
● API Design
● Protocols
● Runtime Validation with Pydantic

45. User-Defined Types in Robust Python
● Enumerations
● Dataclasses
● Classes
● Subtyping
● API Design
● Protocols
● Runtime Validation with Pydantic

46. User-defined types are a way for you to define
your own vocabulary

47. The abstractions you choose communicate to
the future

48. def print_receipt(
order: Order,
restaurant: tuple[str, int,
str]):
total = (order.subtotal *
(1 + tax[restaurant[2]]))

49. def print_receipt(
order: Order,
restaurant: tuple[str, int,
str]):
total = (order.subtotal *
(1 + tax[restaurant[2]]))

50. def print_receipt(
order: Order,
restaurant: Restaurant):
total = (order.subtotal *
(1 + tax[restaurant.city]))
print(Receipt(restaurant.name,

51. User-defined types unify mental models

52. User-defined types make it easier to reason
about a codebase

53. User-defined types make it harder to make
errors

54. I want to focus on the "why" we use user-
defined types, not "how" to create them

55. Enumerations

57. MOTHER_SAUCES = ("Béchamel",
"Velouté",
"Espagnole",
"Tomato",
"Hollandaise")

58. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce(MOTHER_SAUCE[0],
["Onion"])

59. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
# What was MOTHER_SAUCE[0] again?
create_daughter_sauce(MOTHER_SAUCE[0],
["Onion"])

60. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce(MOTHER_SAUCE[0],
["Onion"])

61. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce("Bechamel",
["Onion"])

62. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce("BBQ Sauce",
["Onion"])

63. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce("BBQ Sauce",
["Onion"])
Wrong

64. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce("Bechamel",
["Onion"])

65. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce("Béchamel",
["Onion"])

66. Strings can be anything

67. Restrict your choices with enumerations

68. from enum import Enum
class MotherSauce(Enum):
BÉCHAMEL = "Béchamel"
VELOUTÉ = "Velouté"
ESPAGNOLE = "Espagnole"
TOMATO = "Tomato"
HOLLANDAISE = "Hollandaise"
MotherSauce.BÉCHAMEL

69. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce(MOTHER_SAUCE[0],
["Onion"])

70. def create_daughter_sauce(
mother_sauce: str,
extra_ingredients: list[str]):
# ...
create_daughter_sauce(MOTHER_SAUCE[0],
["Onion"])

71. def create_daughter_sauce(
mother_sauce: MotherSauce,
extra_ingredients: list[str]):
# ...
create_daughter_sauce(MotherSauce.BÉCHAMEL,
["Onion"])

72. Catch mistakes with static analysis

73. Use enumerations to prevent collaborators
from using incorrect values

74. Use enumerations to simplify choices

75. Prevent bugs

76. What about composite data?

77. Data classes

78. Data classes represent a relationship between
data

79. Author's Name
Recipe
Ingredient List
Author's Life
Story
# of servings
Recipe Name

80. Online Recipe
Author's Name
Recipe
Ingredient List
Author's Life
Story
# of servings
Recipe Name

81. @dataclass
class OnlineRecipe:
name: str
author_name: str
author_life_story: str
number_of_servings: int
ingredients: list[Ingredient]
recipe: str

82. recipe = OnlineRecipe(
"Pasta With Sausage",
"Pat Viafore",
"When I was 15, I remember ......",
6,
["Rigatoni", ..., "Basil", "Sausage"],
"First, brown the sausage ...."
)

83. recipe.name
>>> "Pasta With Sausage"
recipe.number_of_servings
>>> 6

84. Data classes represent heterogeneous
data

85. Heterogeneous data
● Heterogeneous data is data that may be multiple different
types (such as str, int, list[Ingredient], etc.)
● Typically not iterated over -- you access a single field at a
time

86. # DO NOT DO THIS
recipe = {
"name": "Pasta With Sausage",
"author": "Pat Viafore",
"story": "When I was 15, I remember
....",
"number_of_servings": 6,
"ingredients": ["Rigatoni", ..., "Basil"],
"recipe": "First, brown the sausage ...."

87. # is life story the right key name?
do_something(recipe["life_story"])
# What type is recipe?
def do_something_else(recipe: dict):
# .... snip ....

88. Any time a developer has to trawl through the
codebase to answer a question about data, it
wastes time and increases frustration

89. This will create mistakes and incorrect
assumptions, leading to bugs

90. recipe: OnlineRecipe = create_recipe()
# type checker will catch problems
do_something(recipe.life_story)
def do_something_else(recipe: OnlineRecipe):
# .... snip ....

91. Use data classes to group data together and
reduce errors when accessing

92. You communicate intent and prevent future
developers from making errors

93. Data classes aren't appropriate for all
heterogeneous data

94. Invariants

95. Invariants
● Fundamental truths throughout your codebase
● Developers will depend on these truths and build
assumptions on them
● These are not universal truths in every possible system,
just your system

98. Invariants
● Sauce will never be put on top of other toppings
(cheese is a topping in this scenario).
● Toppings may go above or below cheese.
● Pizza will have at most only one sauce.
● Dough radius can be only whole numbers.
● The radius of dough may be only between 15 and 30 cm

99. @dataclass
class Pizza:
radius_in_cm: int
toppings: list[str]

100. pizza = Pizza(15, ["Tomato Sauce",
"Mozzarella",
"Pepperoni"])
# THIS IS BAD!
pizza.radius_in_cm = 1000
pizza.toppings.append("Alfredo Sauce")

101. Classes

102. @dataclass
class Pizza:
radius_in_cm: int
toppings: list[str]

103. class Pizza:
def __init__(self, radius_in_cm: int,
toppings: list[str])
assert 15 <= radius_in_cm <= 30
sauces = [t for t in toppings
if is_sauce(t)]
assert len(sauces) <= 1
self.__radius_in_cm = radius_in_cm
sauce = sauces[:1]
self.__toppings = sauce +
[t for t in toppings if not is_sauce(t)]

104. class Pizza:
def __init__(self, radius_in_cm: int,
toppings: list[str])
assert 15 <= radius_in_cm <= 30
sauces = [t for t in toppings
if is_sauce(t)]
assert len(sauces) <= 1
self.__radius_in_cm = radius_in_cm
sauce = sauces[:1]
self.__toppings = sauce +
[t for t in toppings if not is_sauce(t)]
INVARIAN
T
CHECKING

105. # Now an exception
pizza = Pizza(1000, ["Tomato Sauce",
"Mozzarella",
"Pepperoni"])

106. class Pizza:
def __init__(self, radius_in_cm: int,
toppings: list[str])
assert 15 <= radius_in_cm <= 30
sauces = [t for t in toppings
if is_sauce(t)]
assert len(sauces) <= 1
self.__radius_in_cm = radius_in_cm
sauce = sauces[:1]
self.__toppings = sauce +
[t for t in toppings if not is_sauce(t)]

107. class Pizza:
def __init__(self, radius_in_cm: int,
toppings: list[str])
assert 15 <= radius_in_cm <= 30
sauces = [t for t in toppings
if is_sauce(t)]
assert len(sauces) <= 1
self.__radius_in_cm = radius_in_cm
sauce = sauces[:1]
self.__toppings = sauce +
[t for t in toppings if not is_sauce(t)]
"Private"
Members

108. # Linters will catch this error
# Also a runtime error
pizza.__radius_in_cm = 1000
pizza.__toppings.append("Alfredo Sauce")

109. Classes create invariants that developers
cannot easily modify

110. Classes must always preserve these
invariants

111. class Pizza:
# ... snip ...
def add_topping(self, topping: str):
if is_sauce(topping) and self.has_sauce():
raise TooManySaucesError()
if is_sauce(topping):
self.__toppings.insert(0, topping)
else:
self.__toppings.append(topping)

112. Give future collaborators solid classes to
reason upon

113. Classes allow you to group inter-related data
and preserve invariants across their lifetime

114. Creating User-Defined Types

115. Sub-typing

116. Sub-typing is a relationship between
types

117. A sub-type has all the same behaviors as a
super-type (it may also customize some
behaviors)

118. Inheritance

119. class Rectangle:
def __init__(self, height: int, width: int):
self.__height = height
self.__width = width
def set_width(self, width: int):
self.__width = width
def set_height(self, height: int):
self.__height = height
# ... snip getters ...

120. class Square(Rectangle):
def __init__(self, side_length: int):
self.set_height(side_length)
def set_height(self, side_length: int):
self.__height = side_length
self.__width = side_length
def set_width(self, side_length: int):
self.__height = side_length
self.__width = side_length

121. What I've just shown you has a very subtle
error.

124. FOOD_TRUCK_AREA_SIZES = [
Rectangle(1, 20),
Rectangle(5, 5),
Rectangle(20, 30)
]

125. Is a square a rectangle?

127. Yes, a square is a rectangle (geometrically
speaking)

128. Is a square substitutable for a rectangle?

130. FOOD_TRUCK_AREA_SIZES = [
Rectangle(1, 20),
Rectangle(5, 5),
Rectangle(20, 30)
]

131. FOOD_TRUCK_AREA_SIZES = [
Rectangle(1, 20),
Square(5),
Rectangle(20, 30)
]

132. What can go wrong?

133. def double_food_truck_area_widths():
for ft_shape in FOOD_TRUCK_AREA_SIZES:
old_size = ft_shape.get_width()
ft_shape.set_width(old_size * 2)

134. def double_food_truck_area_widths():
for ft_shape in FOOD_TRUCK_AREA_SIZES:
old_size = ft_shape.get_width()
# What happens when this is a square?
ft_shape.set_width(old_size * 2)

135. def double_food_truck_area_widths():
for food_truck_shape in
FOOD_TRUCK_AREA_SIZES:
old_size = food_truck_shape.get_width()
food_truck_shape.set_width(old_size *
2)

136. def double_food_truck_area_widths():
for ft_shape in FOOD_TRUCK_AREA_SIZES:
old_size = ft_shape.get_width()
old_height = ft_shape.get_height()
ft_shape.set_width(old_size * 2)
# Is this a reasonable assert?
assert ft_shape.get_height() ==
old_height

137. Developers will write code based on the
constraints of the superclass

138. Do not let subclasses violate those constraints

139. Someone changing a super-class should not
need to know about all possible sub-classes

140. Liskov Substitution Principle

141. Substitutability
● Do not strengthen pre-conditions
● Do not weaken post-conditions
● Do not raise new types of exceptions
○ Looking at you, NotImplementedError
● Overridden functions almost always should call super()

142. Inheritance

143. Is-A

144. Is-A

145. Can-Substitute-For-A

146. Types of sub-typing
● Inheritance
● Duck Typing
● Protocols
● Plug-ins
● etc.

147. How you subtype will influence how easy it is
to make errors as code changes

148. The choices you make in code may reduce
future errors

149. The choices you make in code communicate
to the future

150. The abstractions you make in code
communicate to the future

151. Software Development is both Archaeology
and Time Travel

152. Tips for writing Robust Python
● Don't rely on developer's memory or their ability to find all
usage in a codebase
● Communicate intent through deliberate decisions in your
codebase
● Make it hard for developers to do the wrong thing
● Make them succeed by default
● Use tooling to catch errors when they do happen

153. You have a duty to deliver value in a timely
manner

154. You have a duty to make it easy for future
collaborators to deliver value in a timely
manner

155. After all, you'll step into a project one day that
is legacy

156. How do you want that codebase to look?

157. Robust Python

158. https://learning.oreilly.com/get-learning/?code=ROBUSTP21

159. Contact
Twitter: @PatViaforever
Blog: https://patviafore.com
Contracting/Consulting through Kudzera, LLC
https://kudzera.com
E-mail: pat@kudzera.com