This document provides best practices for fixing data issues that occur in production databases. It recommends treating data fixes as code by checking fixes into source control, testing them, and conducting code reviews. It also advises logging all data fix executions, changes, and exceptions. Developers should make fixes idempotent and reversible when possible, be fault-tolerant of exceptions, and optimize for bottlenecks like CPU, memory, and database usage. Database snapshots should be used for testing and reverting changes.

1. Fixing Web Data
in Production
Best practices for bad situations
Aaron Knight, Full Stack Engineer at Voxy

2. Aaron Knight (@iamaaronknight)
Full Stack Engineer
Voxy.com
I am

3. A Django web app
8 years old
12 engineers
10+ data stores
Voxy is

4. Oops.
There’s a problem with the data!
(It was probably my fault)

6. > SELECT * FROM feature_toggles LIMIT 2;
+-------+----------+--------------------+
| id | user_id | orientation_videos |
|-------+----------+--------------------+
| 1234 | 8923123 | f |
| 1235 | 9213483 | f |
| 1236 | 2136935 | f |

7. > UPDATE feature_toggles SET
orientation_videos = 't' WHERE...

8. Hold up!
What could co wrong?
● You bring down the site.
● You make the problem worse.
● You forget what you did.

9. Never change
data in prod
● Never introduce any bugs.
● Make all the right architecture decisions
the first time.

10. </sarcasm>

11. Data fixes are code.
● Check them in to source control.

12. Data fixes are code.
● Check them in to source control.
● Test them.

13. Data fixes are code.
● Check them in to source control.
● Test them.
● Code review them.

14. Track execution.
● Log when a script is executed.

15. Track execution.
● Log when a script is executed.
● Log everything that changed.

16. Track execution.
● Log when a script is executed.
● Log everything that changed.
● Log what did not change.

17. def fix_feature_toggles():
logger.info('Starting fix_feature_toggles script')
for toggle in FeatureToggle.objects.all():
if toggle.orientation_videos:
logger.info('FeatureToggle {} orientation_videos
already exists; skipping'.format(toggle.id))
else:
toggle.orientation_videos = get_correct_value(toggle)
toggle.save()
logger.info(
'FeatureToggle {} orientation_videos updated to
{}'.format(toggle.id, toggle.orientation_videos))
logger.info('Finished fix_feature_toggles script')

18. Track execution.
● Log when a script is executed.
● Log everything that changed.
● Log what did not change.
● Centralize your logging.

20. import boto3
firehose = boto3.client('firehose')
def log_to_kinesis(message):
data = OrderedDict([
('script_name', get_filename_of_caller()),
('environment', settings.ENVIRONMENT),
('ts', str(pytz.utc.localize(datetime.datetime.now()))),
('message', message),
])
firehose.put_record(
DeliveryStreamName='backfill-logs',
Record={'Data': (json.dumps(data, sort_keys=False) + 'n')}
)

21. Track execution.
● Log when a script is executed.
● Log everything that changed.
● Log what did not change.
● Centralize your logging.
● Track the script’s progress.

23. import tqdm
def backfill_toggles():
count = FeatureToggle.objects.count()
for org in tqdm(FeatureToggle.objects.all(), count=count):
...

24. Be fault-tolerant.
● Think of possible exceptions.

26. for user_id in list_of_user_ids:
try:
toggle = FeatureToggle.objects.get(user_id=user_id)
except FeatureToggle.DoesNotExist:
logger.info('FeatureToggle does not exist for User
{}'.format(User_id))
continue
toggle.orientation_videos = True
toggle.save()

27. Be fault-tolerant.
● Think of possible exceptions.
● Make your scripts idempotent, if possible.

28. for user_id in list_of_user_ids:
try:
toggle = FeatureToggle.objects.get(user_id=user_id,
backfilled=False)
except FeatureToggle.DoesNotExist:
continue
toggle.orientation_videos = True
toggle.backfilled = True
toggle.save()

29. Be fault-tolerant.
● Think of possible exceptions.
● Make your scripts idempotent, if possible.
● Make your changes reversible, if possible.

30. {"environment": "production", "ts": "2017-10-02 18:33:08.805645+00:00",
"message": "unit_id 117 resource_id: None > 597ca7531ce6856f34607de9"}
{"environment": "production", "ts": "2017-10-02 18:33:08.878832+00:00",
"message": "unit_id 28 resource_id: None > 54ca763ca8615a76184dd4a9"}

31. Know your bottlenecks.
● CPU?

32. Know your bottlenecks.
● CPU?
● Memory?

33. Know your bottlenecks.
● CPU?
● Memory?
● Database?

34. feature_toggles = []
for user_id in user_ids_to_backfill:
feature_toggles.append(FeatureToggle(
user_id=user_id,
orientation_videos=True
)
)
FeatureToggle.objects.bulk_create(feature_toggles)

35. def backfill_activity_progresses():
conn = psycopg2.connect("some_credentials")
cursor = conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
data_to_replicate = []
for index in tqdm(batch_range):
cursor.execute("SELECT user_id, type, correct_answers,
total_answers, is_complete FROM legacy_activity ORDER BY id;")
data_to_replicate.append(cursor.fetchall())
conn.close()
add_to_new_activity_table(data_to_replicate)

36. Know your bottlenecks.
● CPU?
● Memory?
● Database?
● Developer time?
● Cognitive overhead?

37. Execute at the right level
of abstraction.
● Use existing functions and the ORM when
you can afford to.
● Use SQL when execution time becomes
significant.

38. Use database
snapshots.
● Test your script on a backup from
production.

39. Use database
snapshots.
● Test your script on a backup from
production.
● Take snapshots before you make changes.

40. Use database
snapshots.
● Test your script on a backup from
production.
● Take snapshots before you make changes.
● Automate your backups.

41. Aaron Knight (@iamaaronknight)
Full Stack Engineer
Voxy.com
Thanks!