Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Unit testing data with marbles - Jane Stewart Adams, Leif Walsh

1,153 views

Published on

In the same way that we need to make assertions about how code functions, we need to make assertions about data, and unit testing is a promising framework. In this talk, we'll explore what is unique about unit testing data, and see how Two Sigma's open source library Marbles addresses these unique challenges in several real-world scenarios.

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/ETv9m ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Unit testing data with marbles - Jane Stewart Adams, Leif Walsh

  1. 1. UNIT TESTING DATAUNIT TESTING DATA WITH MARBLESWITH MARBLES JANE ADAMS & LEIF WALSHJANE ADAMS & LEIF WALSH
  2. 2. ONCE UPON A TIME...ONCE UPON A TIME...
  3. 3. ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS.
  4. 4. ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS. THEY WERETHEY WERE ALOTALOTOLDER THAN THEIR PARENTS.OLDER THAN THEIR PARENTS.
  5. 5. ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS. THEY WERETHEY WERE ALOTALOTOLDER THAN THEIR PARENTS.OLDER THAN THEIR PARENTS. THEY WERE A LOT OLDER THANTHEY WERE A LOT OLDER THAN EVERYONEEVERYONE..
  6. 6. EVERYONE THAT WORKS WITH DATA HAS STORIES LIKEEVERYONE THAT WORKS WITH DATA HAS STORIES LIKE THISTHIS
  7. 7. HI, I'M JANEHI, I'M JANE
  8. 8. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS?
  9. 9. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS? 1. Children are born after their parents
  10. 10. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS? 1. Children are born after their parents 2. People can't live forever
  11. 11. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?
  12. 12. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct
  13. 13. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data
  14. 14. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique
  15. 15. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique Measurements are precise
  16. 16. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique Measurements are precise (this is a non-exhaustive list)
  17. 17. WHY DOES THIS MATTER?WHY DOES THIS MATTER?
  18. 18. WE DON'T JUST HAVE DATA TO HAVE IT.WE DON'T JUST HAVE DATA TO HAVE IT.
  19. 19. WE DON'T JUST HAVE DATA TO HAVE IT.WE DON'T JUST HAVE DATA TO HAVE IT. WE USE DATA TO MAKE DECISIONS.WE USE DATA TO MAKE DECISIONS.
  20. 20. WE SHOULD BE EXPLICIT ABOUT OUR ASSUMPTIONS.WE SHOULD BE EXPLICIT ABOUT OUR ASSUMPTIONS.
  21. 21. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?
  22. 22. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing
  23. 23. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent
  24. 24. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone
  25. 25. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone We're working with alotof data
  26. 26. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone We're working with alotof data We're working with a lot of differentkindsof data
  27. 27. I'M LEIFI'M LEIF
  28. 28. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO?
  29. 29. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form
  30. 30. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data
  31. 31. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold
  32. 32. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold Report allof the assumptions that don't hold
  33. 33. "Whatifwewroteunittestsfordata likewewriteunittestsforcode?"
  34. 34. unittestunittest
  35. 35. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM?
  36. 36. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Encode our assumptions in testable form
  37. 37. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Test those assumptions on incoming data Encode our assumptions in testable form
  38. 38. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Report when our assumptions don't hold Encode our assumptions in testable form Test those assumptions on incoming data
  39. 39. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Report allof the assumptions that don't hold Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold
  40. 40. tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  41. 41. How long was the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  42. 42. How far was the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  43. 43. Internal metadata tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  44. 44. Who took the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  45. 45. ??? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  46. 46. class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  47. 47. Load the data class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  48. 48. Pick some thresholds class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  49. 49. For each threshold class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  50. 50. Find trips longer than the threshold class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  51. 51. Assert none exist class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  52. 52. $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
  53. 53. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US?
  54. 54. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips
  55. 55. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips Don't have to remember howto test for long trips
  56. 56. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips Don't have to remember howto test for long trips Can easily run this test over historical data
  57. 57. TEST WRITING INTERLUDE...TEST WRITING INTERLUDE...
  58. 58. TEST WRITING INTERLUDE...TEST WRITING INTERLUDE...
  59. 59. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!
  60. 60. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data
  61. 61. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down
  62. 62. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable
  63. 63. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable 4. We've automated them
  64. 64. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable 4. We've automated them ⭐⭐
  65. 65. MONTHS PASS...MONTHS PASS...
  66. 66. $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
  67. 67. Her: "Is there a way to see local variables in my unittest output?" $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
  68. 68. Her: "Is there a way to see local variables in my unittest output?" Him: "I think pytest does that..." $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
  69. 69. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES
  70. 70. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here?
  71. 71. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure?
  72. 72. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it?
  73. 73. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it? Have we seen this failure before? When?
  74. 74. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it? Have we seen this failure before? When? CONTEXT IS EXPENSIVE TO RECOVERCONTEXT IS EXPENSIVE TO RECOVER
  75. 75. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA
  76. 76. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white
  77. 77. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white 2. Failures are usually introduced by someone else
  78. 78. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white 2. Failures are usually introduced by someone else 3. Different tests require different follow-up
  79. 79. ANATOMY OF A MARBLES FAILURE MESSAGEANATOMY OF A MARBLES FAILURE MESSAGE
  80. 80. $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  81. 81. What is this test doing? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  82. 82. What is this test doing? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  83. 83. Why is it here? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  84. 84. What am I supposed to do about this failure? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  85. 85. How bad is it? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  86. 86. Can we add more context? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  87. 87. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS
  88. 88. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper))
  89. 89. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x)
  90. 90. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x) self.assertTrue(all(a < b for a, b in zip([lower, x], [x, upper])))
  91. 91. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x) self.assertTrue(all(a < b for a, b in zip([lower, x], [x, upper]))) self.assertBetween(x, lower, upper)
  92. 92. marbles.mixinsmarbles.mixins
  93. 93. marbles.mixinsmarbles.mixins from marbles.mixins import mixins   class TripDistanceTestCase(BikeshareTestCase, mixins.BetweenMixins):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_unreasonable_distances(self): for distance in self.data['distance_meters']: self.assertBetween(distance, 100, 42195)
  94. 94. CUSTOM ASSERTIONSCUSTOM ASSERTIONS
  95. 95. CUSTOM ASSERTIONSCUSTOM ASSERTIONS self.assertTrue(long_trips.empty)
  96. 96. CUSTOM ASSERTIONSCUSTOM ASSERTIONS self.assertTrue(long_trips.empty) self.assertEqual(len(long_trips), 0)
  97. 97. CUSTOM ASSERTIONSCUSTOM ASSERTIONS self.assertTrue(long_trips.empty) self.assertEqual(len(long_trips), 0) class DataFrameMixins(object):   def assertDataFrameEmpty(self, df, msg=None): self.assertTrue(df.empty, msg=msg)
  98. 98. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED?
  99. 99. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? What is this test doing? Why is it here?
  100. 100. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? What am I supposed to do about this failure? What is this test doing? Why is it here?
  101. 101. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? How bad is it? What is this test doing? Why is it here? What am I supposed to do about this failure?
  102. 102. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? Have we seen this failure before? When? What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it?
  103. 103. ASSERTION LOGGINGASSERTION LOGGING
  104. 104. ASSERTION LOGGINGASSERTION LOGGING import marbles.core from marbles.core import log   class TripDistanceTestCase(BikeshareTestCase): ...   if __name__ == '__main__': log.logger.configure(logfile='marbles.log') marbles.core.main()
  105. 105. { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  106. 106. Which test was running? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  107. 107. What did we assert? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  108. 108. Local variables { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  109. 109. Which data were we testing? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  110. 110. Other information about the assertion { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  111. 111. More (not pictured) { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  112. 112. HISTORICAL FAILURESHISTORICAL FAILURES
  113. 113. HISTORICAL FAILURESHISTORICAL FAILURES "Have we seen this kind of problem before?"
  114. 114. HISTORICAL FAILURESHISTORICAL FAILURES "Have we seen this kind of problem before?"
  115. 115. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS
  116. 116. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS df = df.pivot_table( index=['month'], columns=['severity'], values='anomalies', aggfunc=sum) df.describe()
  117. 117. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS df = df.pivot_table( index=['month'], columns=['severity'], values='anomalies', aggfunc=sum) df.describe()
  118. 118. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO
  119. 119. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO $ python -m unittest F ====================================================================== FAIL: test_return_code (docs.examples.getting_started.ResponseTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/git/marbles/docs/examples/getting_started.py", line 43, in test_return_code 201 AssertionError: 409 != 201   ---------------------------------------------------------------------- Ran 1 test in 0.000s
  120. 120. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO $ python -m marbles F ====================================================================== FAIL: test_return_code (docs.examples.getting_started.ResponseTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: 409 != 201   Source (/home/leif/git/marbles/docs/examples/getting_started.py): 40 res = requests.put(endpoint, data=data) > 41 self.assertEqual( 42 res.status_code, 43 201 44 ) Locals: endpoint = 'http://example.com/api/v1/resource' data = {'id': 1, 'name': 'Little Bobby Tables'} res = <docs.examples.getting_started.Response object at 0x7fae97e78978>     ---------------------------------------------------------------------- Ran 1 test in 0.001s
  121. 121. TWO STEPS TO MARBLESTWO STEPS TO MARBLES $ pip install marbles $ python -m marbles test_module.py
  122. 122. GITHUBGITHUB github.com/twosigma/marbles
  123. 123. DOCUMENTATIONDOCUMENTATION marbles.readthedocs.io
  124. 124. DOCUMENTATIONDOCUMENTATION marbles.readthedocs.io
  125. 125. CONTRIBUTING AND GETTING HELPCONTRIBUTING AND GETTING HELP github.com/twosigma/marbles/issues
  126. 126. ✨ READ BETTER TEST FAILURES ✨✨ READ BETTER TEST FAILURES ✨ & github.com/twosigma/marbles marbles.readthedocs.io @thejunglejane @leifwalsh

×