In the same way that we need to make assertions about how code functions, we need to make assertions about data, and unit testing is a promising framework. In this talk, we'll explore what is unique about unit testing data, and see how Two Sigma's open source library Marbles addresses these unique challenges in several real-world scenarios.

1. UNIT TESTING DATAUNIT TESTING DATA
WITH MARBLESWITH MARBLES
JANE ADAMS & LEIF WALSHJANE ADAMS & LEIF WALSH

2. ONCE UPON A TIME...ONCE UPON A TIME...

3. ONCE UPON A TIME...ONCE UPON A TIME...
CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS.

4. ONCE UPON A TIME...ONCE UPON A TIME...
CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS.
THEY WERETHEY WERE ALOTALOTOLDER THAN THEIR PARENTS.OLDER THAN THEIR PARENTS.

5. ONCE UPON A TIME...ONCE UPON A TIME...
CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS.
THEY WERETHEY WERE ALOTALOTOLDER THAN THEIR PARENTS.OLDER THAN THEIR PARENTS.
THEY WERE A LOT OLDER THANTHEY WERE A LOT OLDER THAN EVERYONEEVERYONE..

6. EVERYONE THAT WORKS WITH DATA HAS STORIES LIKEEVERYONE THAT WORKS WITH DATA HAS STORIES LIKE
THISTHIS

7. HI, I'M JANEHI, I'M JANE

8. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS?

9. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS?
1. Children are born after their parents

10. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS?
1. Children are born after their parents
2. People can't live forever

11. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?

12. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?
Values are correct

13. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?
Values are correct
We're not missing any data

14. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?
Values are correct
We're not missing any data
Records are unique

15. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?
Values are correct
We're not missing any data
Records are unique
Measurements are precise

16. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?
Values are correct
We're not missing any data
Records are unique
Measurements are precise
(this is a non-exhaustive list)

17. WHY DOES THIS MATTER?WHY DOES THIS MATTER?

18. WE DON'T JUST HAVE DATA TO HAVE IT.WE DON'T JUST HAVE DATA TO HAVE IT.

19. WE DON'T JUST HAVE DATA TO HAVE IT.WE DON'T JUST HAVE DATA TO HAVE IT.
WE USE DATA TO MAKE DECISIONS.WE USE DATA TO MAKE DECISIONS.

20. WE SHOULD BE EXPLICIT ABOUT OUR ASSUMPTIONS.WE SHOULD BE EXPLICIT ABOUT OUR ASSUMPTIONS.

21. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?

22. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?
Data are always changing

23. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?
Data are always changing
Some changes are loud while others are silent

24. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?
Data are always changing
Some changes are loud while others are silent
Manually checking data is inconsistent and error-prone

25. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?
Data are always changing
Some changes are loud while others are silent
Manually checking data is inconsistent and error-prone
We're working with alotof data

26. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?
Data are always changing
Some changes are loud while others are silent
Manually checking data is inconsistent and error-prone
We're working with alotof data
We're working with a lot of differentkindsof data

27. I'M LEIFI'M LEIF

28. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO?

29. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO?
Encode our assumptions in testable form

30. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO?
Encode our assumptions in testable form
Test those assumptions on incoming data

31. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO?
Encode our assumptions in testable form
Test those assumptions on incoming data
Report when our assumptions don't hold

32. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO?
Encode our assumptions in testable form
Test those assumptions on incoming data
Report when our assumptions don't hold
Report allof the assumptions that don't hold

33. "Whatifwewroteunittestsfordata
likewewriteunittestsforcode?"

34. unittestunittest

35. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM?

36. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM?
Encode our assumptions in testable form

37. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM?
Test those assumptions on incoming data
Encode our assumptions in testable form

38. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM?
Report when our assumptions don't hold
Encode our assumptions in testable form
Test those assumptions on incoming data

39. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM?
Report allof the assumptions that don't hold
Encode our assumptions in testable form
Test those assumptions on incoming data
Report when our assumptions don't hold

40. tripduration 0 days 00:18:09
starttime 2018-08-01 00:00:09.341000-04:00
stoptime 2018-08-01 00:18:18.889000-04:00
start station id 31
start station name Seaport Hotel - Congress St at Seaport Ln
start station latitude 42.3488
start station longitude -71.0417
end station id 190
end station name Nashua Street at Red Auerbach Way
end station latitude 42.3657
end station longitude -71.0643
bikeid 1026
usertype Subscriber
birth year 1969
gender 0
Hubway Bike Share Dataset

41. How long was the trip?
tripduration 0 days 00:18:09
starttime 2018-08-01 00:00:09.341000-04:00
stoptime 2018-08-01 00:18:18.889000-04:00
start station id 31
start station name Seaport Hotel - Congress St at Seaport Ln
start station latitude 42.3488
start station longitude -71.0417
end station id 190
end station name Nashua Street at Red Auerbach Way
end station latitude 42.3657
end station longitude -71.0643
bikeid 1026
usertype Subscriber
birth year 1969
gender 0
Hubway Bike Share Dataset

42. How far was the trip?
tripduration 0 days 00:18:09
starttime 2018-08-01 00:00:09.341000-04:00
stoptime 2018-08-01 00:18:18.889000-04:00
start station id 31
start station name Seaport Hotel - Congress St at Seaport Ln
start station latitude 42.3488
start station longitude -71.0417
end station id 190
end station name Nashua Street at Red Auerbach Way
end station latitude 42.3657
end station longitude -71.0643
bikeid 1026
usertype Subscriber
birth year 1969
gender 0
Hubway Bike Share Dataset

43. Internal metadata
tripduration 0 days 00:18:09
starttime 2018-08-01 00:00:09.341000-04:00
stoptime 2018-08-01 00:18:18.889000-04:00
start station id 31
start station name Seaport Hotel - Congress St at Seaport Ln
start station latitude 42.3488
start station longitude -71.0417
end station id 190
end station name Nashua Street at Red Auerbach Way
end station latitude 42.3657
end station longitude -71.0643
bikeid 1026
usertype Subscriber
birth year 1969
gender 0
Hubway Bike Share Dataset

44. Who took the trip?
tripduration 0 days 00:18:09
starttime 2018-08-01 00:00:09.341000-04:00
stoptime 2018-08-01 00:18:18.889000-04:00
start station id 31
start station name Seaport Hotel - Congress St at Seaport Ln
start station latitude 42.3488
start station longitude -71.0417
end station id 190
end station name Nashua Street at Red Auerbach Way
end station latitude 42.3657
end station longitude -71.0643
bikeid 1026
usertype Subscriber
birth year 1969
gender 0
Hubway Bike Share Dataset

45. ???
tripduration 0 days 00:18:09
starttime 2018-08-01 00:00:09.341000-04:00
stoptime 2018-08-01 00:18:18.889000-04:00
start station id 31
start station name Seaport Hotel - Congress St at Seaport Ln
start station latitude 42.3488
start station longitude -71.0417
end station id 190
end station name Nashua Street at Red Auerbach Way
end station latitude 42.3657
end station longitude -71.0643
bikeid 1026
usertype Subscriber
birth year 1969
gender 0
Hubway Bike Share Dataset

46. class TripDistanceTestCase(unittest.TestCase):
def setUp(self):
self.data = ...
def tearDown(self):
delattr(self, 'data')
def test_for_long_trips(self):
thresholds = [
('marathon', 42195),
('10km', 10000)
]
for severity, threshold in thresholds:
with self.subTest(severity=severity):
long_trips = self.data[
self.data['distance_meters'] > threshold]
self.assertTrue(long_trips.empty)

47. Load the data
class TripDistanceTestCase(unittest.TestCase):
def setUp(self):
self.data = ...
def tearDown(self):
delattr(self, 'data')
def test_for_long_trips(self):
thresholds = [
('marathon', 42195),
('10km', 10000)
]
for severity, threshold in thresholds:
with self.subTest(severity=severity):
long_trips = self.data[
self.data['distance_meters'] > threshold]
self.assertTrue(long_trips.empty)

48. Pick some thresholds
class TripDistanceTestCase(unittest.TestCase):
def setUp(self):
self.data = ...
def tearDown(self):
delattr(self, 'data')
def test_for_long_trips(self):
thresholds = [
('marathon', 42195),
('10km', 10000)
]
for severity, threshold in thresholds:
with self.subTest(severity=severity):
long_trips = self.data[
self.data['distance_meters'] > threshold]
self.assertTrue(long_trips.empty)

49. For each threshold
class TripDistanceTestCase(unittest.TestCase):
def setUp(self):
self.data = ...
def tearDown(self):
delattr(self, 'data')
def test_for_long_trips(self):
thresholds = [
('marathon', 42195),
('10km', 10000)
]
for severity, threshold in thresholds:
with self.subTest(severity=severity):
long_trips = self.data[
self.data['distance_meters'] > threshold]
self.assertTrue(long_trips.empty)

50. Find trips longer than the threshold
class TripDistanceTestCase(unittest.TestCase):
def setUp(self):
self.data = ...
def tearDown(self):
delattr(self, 'data')
def test_for_long_trips(self):
thresholds = [
('marathon', 42195),
('10km', 10000)
]
for severity, threshold in thresholds:
with self.subTest(severity=severity):
long_trips = self.data[
self.data['distance_meters'] > threshold]
self.assertTrue(long_trips.empty)

51. Assert none exist
class TripDistanceTestCase(unittest.TestCase):
def setUp(self):
self.data = ...
def tearDown(self):
delattr(self, 'data')
def test_for_long_trips(self):
thresholds = [
('marathon', 42195),
('10km', 10000)
]
for severity, threshold in thresholds:
with self.subTest(severity=severity):
long_trips = self.data[
self.data['distance_meters'] > threshold]
self.assertTrue(long_trips.empty)

52. $ python -m unittest test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips
self.assertTrue(long_trips.empty)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)

53. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US?

54. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US?
New data are automatically tested for long trips

55. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US?
New data are automatically tested for long trips
Don't have to remember howto test for long trips

56. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US?
New data are automatically tested for long trips
Don't have to remember howto test for long trips
Can easily run this test over historical data

57. TEST WRITING INTERLUDE...TEST WRITING INTERLUDE...

58. TEST WRITING INTERLUDE...TEST WRITING INTERLUDE...

59. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!

60. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!
1. We've thought through our assumptions about the data

61. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!
1. We've thought through our assumptions about the data
2. We've made them explicit by writing them down

62. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!
1. We've thought through our assumptions about the data
2. We've made them explicit by writing them down
3. We've made them executable

63. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!
1. We've thought through our assumptions about the data
2. We've made them explicit by writing them down
3. We've made them executable
4. We've automated them

64. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!
1. We've thought through our assumptions about the data
2. We've made them explicit by writing them down
3. We've made them executable
4. We've automated them
⭐⭐

65. MONTHS PASS...MONTHS PASS...

66. $ python -m unittest test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips
self.assertTrue(long_trips.empty)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)

67. Her: "Is there a way to see local variables in my unittest output?"
$ python -m unittest test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips
self.assertTrue(long_trips.empty)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)

68. Her: "Is there a way to see local variables in my unittest output?"
Him: "I think pytest does that..."
$ python -m unittest test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips
self.assertTrue(long_trips.empty)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)

69. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES

70. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES
What is this test doing? Why is it here?

71. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES
What is this test doing? Why is it here?
What am I supposed to do about this failure?

72. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES
What is this test doing? Why is it here?
What am I supposed to do about this failure?
How bad is it?

73. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES
What is this test doing? Why is it here?
What am I supposed to do about this failure?
How bad is it?
Have we seen this failure before? When?

74. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES
What is this test doing? Why is it here?
What am I supposed to do about this failure?
How bad is it?
Have we seen this failure before? When?
CONTEXT IS EXPENSIVE TO RECOVERCONTEXT IS EXPENSIVE TO RECOVER

75. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA,
BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA

76. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA,
BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA
1. Assumptions aren't black-and-white

77. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA,
BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA
1. Assumptions aren't black-and-white
2. Failures are usually introduced by someone else

78. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA,
BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA
1. Assumptions aren't black-and-white
2. Failures are usually introduced by someone else
3. Different tests require different follow-up

80. ANATOMY OF A MARBLES FAILURE MESSAGEANATOMY OF A MARBLES FAILURE MESSAGE

81. $ python -m marbles test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: False is not true
Source (/home/leif/test_bikeshare.py):
151 self.data['distance_meters'] > threshold]
> 152 self.assertTrue(long_trips.empty, note=note)
153
Locals:
severity = 'marathon'
threshold = 42195
long_trips =
start station latitude stop station latitude ...
27955 42.366277 0.0 ...
Note:
There appear to be some trips in the data that are longer than a
marathon! If these are legitimate trips, consider contacting the
local news station about a human-interest story. If these do not
appear to be legitimate trips, contact the bike share mechanics
to have affected bikes identified and repaired.

82. What is this test doing?
$ python -m marbles test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: False is not true
Source (/home/leif/test_bikeshare.py):
151 self.data['distance_meters'] > threshold]
> 152 self.assertTrue(long_trips.empty, note=note)
153
Locals:
severity = 'marathon'
threshold = 42195
long_trips =
start station latitude stop station latitude ...
27955 42.366277 0.0 ...
Note:
There appear to be some trips in the data that are longer than a
marathon! If these are legitimate trips, consider contacting the
local news station about a human-interest story. If these do not
appear to be legitimate trips, contact the bike share mechanics
to have affected bikes identified and repaired.

83. What is this test doing?
$ python -m marbles test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: False is not true
Source (/home/leif/test_bikeshare.py):
151 self.data['distance_meters'] > threshold]
> 152 self.assertTrue(long_trips.empty, note=note)
153
Locals:
severity = 'marathon'
threshold = 42195
long_trips =
start station latitude stop station latitude ...
27955 42.366277 0.0 ...
Note:
There appear to be some trips in the data that are longer than a
marathon! If these are legitimate trips, consider contacting the
local news station about a human-interest story. If these do not
appear to be legitimate trips, contact the bike share mechanics
to have affected bikes identified and repaired.

84. Why is it here?
$ python -m marbles test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: False is not true
Source (/home/leif/test_bikeshare.py):
151 self.data['distance_meters'] > threshold]
> 152 self.assertTrue(long_trips.empty, note=note)
153
Locals:
severity = 'marathon'
threshold = 42195
long_trips =
start station latitude stop station latitude ...
27955 42.366277 0.0 ...
Note:
There appear to be some trips in the data that are longer than a
marathon! If these are legitimate trips, consider contacting the
local news station about a human-interest story. If these do not
appear to be legitimate trips, contact the bike share mechanics
to have affected bikes identified and repaired.

85. What am I supposed to do about this failure?
$ python -m marbles test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: False is not true
Source (/home/leif/test_bikeshare.py):
151 self.data['distance_meters'] > threshold]
> 152 self.assertTrue(long_trips.empty, note=note)
153
Locals:
severity = 'marathon'
threshold = 42195
long_trips =
start station latitude stop station latitude ...
27955 42.366277 0.0 ...
Note:
There appear to be some trips in the data that are longer than a
marathon! If these are legitimate trips, consider contacting the
local news station about a human-interest story. If these do not
appear to be legitimate trips, contact the bike share mechanics
to have affected bikes identified and repaired.

86. How bad is it?
$ python -m marbles test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: False is not true
Source (/home/leif/test_bikeshare.py):
151 self.data['distance_meters'] > threshold]
> 152 self.assertTrue(long_trips.empty, note=note)
153
Locals:
severity = 'marathon'
threshold = 42195
long_trips =
start station latitude stop station latitude ...
27955 42.366277 0.0 ...
Note:
There appear to be some trips in the data that are longer than a
marathon! If these are legitimate trips, consider contacting the
local news station about a human-interest story. If these do not
appear to be legitimate trips, contact the bike share mechanics
to have affected bikes identified and repaired.

87. Can we add more context?
$ python -m marbles test_bikeshare.py
======================================================================
FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: False is not true
Source (/home/leif/test_bikeshare.py):
151 self.data['distance_meters'] > threshold]
> 152 self.assertTrue(long_trips.empty, note=note)
153
Locals:
severity = 'marathon'
threshold = 42195
long_trips =
start station latitude stop station latitude ...
27955 42.366277 0.0 ...
Note:
There appear to be some trips in the data that are longer than a
marathon! If these are legitimate trips, consider contacting the
local news station about a human-interest story. If these do not
appear to be legitimate trips, contact the bike share mechanics
to have affected bikes identified and repaired.

88. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS

89. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS
self.assertTrue((lower < x) and (x < upper))

90. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS
self.assertTrue((lower < x) and (x < upper))
self.assertGreater(x, lower)
self.assertGreater(upper, x)

91. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS
self.assertTrue((lower < x) and (x < upper))
self.assertGreater(x, lower)
self.assertGreater(upper, x)
self.assertTrue(all(a < b for a, b in zip([lower, x], [x, upper])))

92. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS
self.assertTrue((lower < x) and (x < upper))
self.assertGreater(x, lower)
self.assertGreater(upper, x)
self.assertTrue(all(a < b for a, b in zip([lower, x], [x, upper])))
self.assertBetween(x, lower, upper)

93. marbles.mixinsmarbles.mixins

94. marbles.mixinsmarbles.mixins
from marbles.mixins import mixins
class TripDistanceTestCase(BikeshareTestCase, mixins.BetweenMixins):
def setUp(self):
self.data = ...
def tearDown(self):
delattr(self, 'data')
def test_for_unreasonable_distances(self):
for distance in self.data['distance_meters']:
self.assertBetween(distance, 100, 42195)

96. CUSTOM ASSERTIONSCUSTOM ASSERTIONS

97. CUSTOM ASSERTIONSCUSTOM ASSERTIONS
self.assertTrue(long_trips.empty)

98. CUSTOM ASSERTIONSCUSTOM ASSERTIONS
self.assertTrue(long_trips.empty)
self.assertEqual(len(long_trips), 0)

99. CUSTOM ASSERTIONSCUSTOM ASSERTIONS
self.assertTrue(long_trips.empty)
self.assertEqual(len(long_trips), 0)
class DataFrameMixins(object):
def assertDataFrameEmpty(self, df, msg=None):
self.assertTrue(df.empty, msg=msg)

100. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED?

101. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED?
What is this test doing? Why is it here?

102. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED?
What am I supposed to do about this failure?
What is this test doing? Why is it here?

103. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED?
How bad is it?
What is this test doing? Why is it here?
What am I supposed to do about this failure?

104. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED?
Have we seen this failure before? When?
What is this test doing? Why is it here?
What am I supposed to do about this failure?
How bad is it?

105. ASSERTION LOGGINGASSERTION LOGGING

106. ASSERTION LOGGINGASSERTION LOGGING
import marbles.core
from marbles.core import log
class TripDistanceTestCase(BikeshareTestCase):
...
if __name__ == '__main__':
log.logger.configure(logfile='marbles.log')
marbles.core.main()

107. {
"case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)",
"test_case": "TripDistanceTestCase",
"test_method": "test_for_long_trips",
"assertion": "assertTrue",
...
"locals": [
{
"key": "severity",
"value": "marathon"
},
{
"key": "threshold",
"value": "42195"
},
{
"key": "long_trips",
"value": "
start station latitude stop station latitude ...
27955 42.366277 0.0 ... "
}
],
"month": "2016-07-01",
"severity": "marathon",
"anomalies": "1",
"result": "fail"
}

108. Which test was running?
{
"case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)",
"test_case": "TripDistanceTestCase",
"test_method": "test_for_long_trips",
"assertion": "assertTrue",
...
"locals": [
{
"key": "severity",
"value": "marathon"
},
{
"key": "threshold",
"value": "42195"
},
{
"key": "long_trips",
"value": "
start station latitude stop station latitude ...
27955 42.366277 0.0 ... "
}
],
"month": "2016-07-01",
"severity": "marathon",
"anomalies": "1",
"result": "fail"
}

109. What did we assert?
{
"case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)",
"test_case": "TripDistanceTestCase",
"test_method": "test_for_long_trips",
"assertion": "assertTrue",
...
"locals": [
{
"key": "severity",
"value": "marathon"
},
{
"key": "threshold",
"value": "42195"
},
{
"key": "long_trips",
"value": "
start station latitude stop station latitude ...
27955 42.366277 0.0 ... "
}
],
"month": "2016-07-01",
"severity": "marathon",
"anomalies": "1",
"result": "fail"
}

110. Local variables
{
"case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)",
"test_case": "TripDistanceTestCase",
"test_method": "test_for_long_trips",
"assertion": "assertTrue",
...
"locals": [
{
"key": "severity",
"value": "marathon"
},
{
"key": "threshold",
"value": "42195"
},
{
"key": "long_trips",
"value": "
start station latitude stop station latitude ...
27955 42.366277 0.0 ... "
}
],
"month": "2016-07-01",
"severity": "marathon",
"anomalies": "1",
"result": "fail"
}

111. Which data were we testing?
{
"case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)",
"test_case": "TripDistanceTestCase",
"test_method": "test_for_long_trips",
"assertion": "assertTrue",
...
"locals": [
{
"key": "severity",
"value": "marathon"
},
{
"key": "threshold",
"value": "42195"
},
{
"key": "long_trips",
"value": "
start station latitude stop station latitude ...
27955 42.366277 0.0 ... "
}
],
"month": "2016-07-01",
"severity": "marathon",
"anomalies": "1",
"result": "fail"
}

112. Other information about the assertion
{
"case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)",
"test_case": "TripDistanceTestCase",
"test_method": "test_for_long_trips",
"assertion": "assertTrue",
...
"locals": [
{
"key": "severity",
"value": "marathon"
},
{
"key": "threshold",
"value": "42195"
},
{
"key": "long_trips",
"value": "
start station latitude stop station latitude ...
27955 42.366277 0.0 ... "
}
],
"month": "2016-07-01",
"severity": "marathon",
"anomalies": "1",
"result": "fail"
}

113. More (not pictured)
{
"case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)",
"test_case": "TripDistanceTestCase",
"test_method": "test_for_long_trips",
"assertion": "assertTrue",
...
"locals": [
{
"key": "severity",
"value": "marathon"
},
{
"key": "threshold",
"value": "42195"
},
{
"key": "long_trips",
"value": "
start station latitude stop station latitude ...
27955 42.366277 0.0 ... "
}
],
"month": "2016-07-01",
"severity": "marathon",
"anomalies": "1",
"result": "fail"
}

114. HISTORICAL FAILURESHISTORICAL FAILURES

115. HISTORICAL FAILURESHISTORICAL FAILURES
"Have we seen this kind of problem before?"

116. HISTORICAL FAILURESHISTORICAL FAILURES
"Have we seen this kind of problem before?"

117. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS

118. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS
df = df.pivot_table(
index=['month'], columns=['severity'],
values='anomalies', aggfunc=sum)
df.describe()

119. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS
df = df.pivot_table(
index=['month'], columns=['severity'],
values='anomalies', aggfunc=sum)
df.describe()

120. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO

121. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO
$ python -m unittest
F
======================================================================
FAIL: test_return_code (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/leif/git/marbles/docs/examples/getting_started.py", line 43, in test_return_code
201
AssertionError: 409 != 201
----------------------------------------------------------------------
Ran 1 test in 0.000s

122. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO
$ python -m marbles
F
======================================================================
FAIL: test_return_code (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/home/leif/git/marbles/docs/examples/getting_started.py):
40 res = requests.put(endpoint, data=data)
> 41 self.assertEqual(
42 res.status_code,
43 201
44 )
Locals:
endpoint = 'http://example.com/api/v1/resource'
data = {'id': 1, 'name': 'Little Bobby Tables'}
res = <docs.examples.getting_started.Response object at 0x7fae97e78978>
----------------------------------------------------------------------
Ran 1 test in 0.001s

123. TWO STEPS TO MARBLESTWO STEPS TO MARBLES
$ pip install marbles
$ python -m marbles test_module.py

124. GITHUBGITHUB
github.com/twosigma/marbles

125. DOCUMENTATIONDOCUMENTATION
marbles.readthedocs.io

126. DOCUMENTATIONDOCUMENTATION
marbles.readthedocs.io

127. CONTRIBUTING AND GETTING HELPCONTRIBUTING AND GETTING HELP
github.com/twosigma/marbles/issues

128. ✨ READ BETTER TEST FAILURES ✨✨ READ BETTER TEST FAILURES ✨
&
github.com/twosigma/marbles
marbles.readthedocs.io
@thejunglejane @leifwalsh