Data Con LA 2020 Description How do you version data? How do you keep track of data changes? How do you store the data changes? In this talk we explore some answers by learning about DeepDiff's newly introduced Delta feature. This is a talk from the creator of DeepDiff, Sep Dehpour. DeepDiff is a Python library to get the diff of nested data. And it has a new feature: Delta! In this talk we learn about: *Diffing the nested data *The ignore order problem *From Levenshtein distance to Deep Distance *Get and store the Delta *Apply the Delta, even to arbitrary objects Speaker Sep Dehpour, Fair, Staff Engineer

1. Delta for your data
How to efficiently track and store deeply nested data changes
By Sep Dehpour
Oct 2020

2. What is already out there?

3. Git

4. Git
[{
"key1": "value1",
"key2": "value2"
}]

5. Git
[{
"key2": "value2",
"key1": "value1"
}]

6. diff & patch
diff -u file1.html file2.html > patchfile.patch
patch file1.html patchfile.patch

7. Quilt
"Manage data like code"
https://github.com/quiltdata/quilt

8. QRI
https://qri.io
Similar to Quilt but built on top of ipfs.io

9. DVID
dvid.io
Distributed Versioned Image-oriented Dataservice

10. Dolt
https://www.dolthub.com/
Git for data experience in a SQL database,

11. Databricks Delta Lake
https://delta.io/
storage layer that brings ACID
transactions to Apache Spark™ and big data workloads.

12. DeepDiff
https://zepworks.com
pip install deepdiff

13. t1.csv
first_name,last_name,zip
Joe,Nobody,90011
Jimmy,Brown,90007
Sara,Smith,90007
t2.csv
first_name,last_name,zip
Joe,Nobody,90011
Jimmy,Brown,90404
Sara,Smith,90007

14. $ deepdiff t1.csv t2.csv
{ 'values_changed': { "root[1]['zip']": { 'new_value': '90404',
'old_value': '90007'}}}

15. t2.csv
first_name,last_name,zip
Joe,Nobody,90011
Jimmy,Brown,90404
Sara,Smith,90007

16. t3.csv
first_name,last_name,zip
Joe,Nobody,90011
Sara,Smith,90007
Jimmy,Brown,90404

17. $ deepdiff t1.csv t3.csv

18. $ deepdiff t1.csv t3.csv
{ 'values_changed': { "root[1]['first_name']": { 'new_value': 'Sara',
'old_value': 'Jimmy'},
"root[1]['last_name']": { 'new_value': 'Smith',
'old_value': 'Brown'},
"root[1]['zip']": { 'new_value': '90007',
'old_value': '90007'},
"root[2]['first_name']": { 'new_value': 'Jimmy',
'old_value': 'Sara'},
"root[2]['last_name']": { 'new_value': 'Brown',
'old_value': 'Smith'},
"root[2]['zip']": { 'new_value': '90404',
'old_value': '90007'}}}

19. $ deepdiff t1.csv t3.csv --ignore-order
{ 'values_changed': { "root[1]['zip']": { 'new_value': '90404',
'old_value': '90007'}}}

20. t4.csv
first_name,last_name,zip
Joe,Nobody,90011
Jimmy,Brown,90404
Sara,Smith,90007
Jimmy,Brown,90404

21. deepdiff t1.csv t4.csv --ignore-order
{ 'values_changed': { "root[1]['zip']": { 'new_value': '90404',
'old_value': '90007'}}}

22. Let's make a patch aka delta
deepdiff t1.csv t4.csv --ignore-order --report-repetition --create-patch > patch1

23. t1new.csv
first_name,last_name,zip
Joe,Nobody,90011
Jimmy,Brown,90007
Sara,Smith,90007
John,Doe,90001
Max,Foo,23232

24. DeepPatch
first_name,last_name,zip
Joe,Nobody,90011
Jimmy,Brown,90007
Sara,Smith,90007
John,Doe,90001
Max,Foo,23232

25. DeepPatch
$ deeppatch t1new.csv patch1 --backup
first_name,last_name,zip
Joe,Nobody,90011
Jimmy,Brown,90007
Sara,Smith,90007
John,Doe,90001
Max,Foo,23232

26. DeepPatch
$ deeppatch t1new.csv patch1 --backup
first_name,last_name,zip
Joe,Nobody,90011
Jimmy,Brown,90404
Sara,Smith,90007
John,Doe,90001
Max,Foo,23232

27. files
$ deepdiff t1.csv t1new.csv --ignore-order --report-repetition --create-patch > patch2
deepdiff t1.csv t4.csv --ignore-order --report-repetition --create-patch > patch1
t1.csv
patch1
patch2
deeppatch t1.csv patch1 --backup
deeppatch t1.csv patch2 --backup

28. some.yaml
---
-
first_name: Joe
last_name: Nobody
address: 3232 Main st.
phone: 323-123-2345
zip: 90011
-
first_name: Jimmy
last_name: Brown
address: 11th There sq.
phone: 111-123-9911
zip: 90002
-
first_name: Sara
last_name: Smith
address: Downtown LA
phone: 818-113-2005
zip: 90007

29. $ deeppatch some.yaml patch1 --backup
---
-
first_name: Joe
last_name: Nobody
address: 3232 Main st.
phone: 323-123-2345
zip: 90011
-
first_name: Jimmy
last_name: Brown
address: 11th There sq.
phone: 111-123-9911
zip: 90002
-
first_name: Sara
last_name: Smith
address: Downtown LA
phone: 818-113-2005
zip: 90007

30. $ deeppatch some.yaml patch1 --backup
- address: 3232 Main st.
first_name: Joe
last_name: Nobody
phone: 323-123-2345
zip: 90011
- address: 11th There sq.
first_name: Jimmy
last_name: Brown
phone: 111-123-9911
zip: '90404'
- address: Downtown LA
first_name: Sara
last_name: Smith
phone: 818-113-2005
zip: 90007

31. $ deeppatch some.yaml patch2 --backup
- address: 3232 Main st.
first_name: Joe
last_name: Nobody
phone: 323-123-2345
zip: 90011
- address: 11th There sq.
first_name: Jimmy
last_name: Brown
phone: 111-123-9911
zip: '90404'
- address: Downtown LA
first_name: Sara
last_name: Smith
phone: 818-113-2005
zip: 90007
- first_name: John
last_name: Doe
zip: '90001'
- first_name: Max
last_name: Foo
zip: '23232'

32. Looking for something in the file?

33. DeepGrep
$ deepgrep sara some.yaml -i

34. DeepGrep
$ deepgrep sara some.yaml -i
{'matched_values': OrderedSet(["root[2]['first_name']"])}

35. Extracting the value?
$ deepgrep sara some.yaml -i
{'matched_values': OrderedSet(["root[2]['first_name']"])}

36. DeepExtract
$ deepgrep sara some.yaml -i
{'matched_values': OrderedSet(["root[2]['first_name']"])}
$ deepextract "root[2]" some.yaml
{ 'address': 'Downtown LA',
'first_name': 'Sara',
'last_name': 'Smith',
'phone': '818-113-2005',
'zip': 90007}

37. Nested Data

38. [
[
[
[["a", "b", "c", "d"], ["a", "e", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[[10, "b", 2, "d"], ["a", "e", "c", "d"], ["f", 2, "d", 80]],
[["10", "d"], ["a", "d", 89], [10, 2, 2, 80]]
],
[
[["a", "b", "f", "d"], ["a", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[[10, "b", 2, "d"], ["a", "e", "c", "d", "d"], ["f", 22, "d", 80]],
[["10", "dd"], ["a", "d", 89], [10, 2]]
],
[
[["a", "b", "c", "d"], ["a", "e", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[["10", "d"], ["a", "d", 89], [10, 2, 2, 80]]
],
[
[["a", "b", "f", "d"], ["a", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[[10, "b", 22, "d"], ["a", "e", "c", "d", "d"], ["f", 22, "d", 80]],
[["10", "dd"], ["a", "d", 89], [10, 2]]
]
],
[
[
[["a", "b", "f", "d"], ["a", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[[10, "dada"], ["a", "e", "c", "d", "d"], ["f", 22, "d", 80]],
[["10", "dd"], ["a", "d", 389], [10, 2]]
],
[
[["a", "b", "c", "d"], ["a", "e", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[[10, "b", 2, "d"], ["a", "e", "c", "d"], ["f", 2, "d", 80]],
[["10", "d"], ["a", "d", 89], [10, 2, 2, 80]]
],
[
[["a", "b", "c", "d"], ["a", "e", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[["10", "d"], ["a", "d", 89], [10, 2, 2, 801]]
],
[
[["a", "b", "f", "d"], ["a", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[[10, "b", 22, "d"], ["a", "e", "d"], ["f", 22, "d", 80]],
[["10", "dd"], ["a", "d", 89], [10, 2, 11]]
]
]
]

39. [
[
[
[["a", "b", "c", "d"], ["a", "e", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[[1, "b", 2, "d"], ["a", "e", "c", "d"], ["f", 2, "d", 80]],
[["10", 0, "d"], ["a", "d", 89], [10, 2, 2, 80]]
],
[
[["a", "b", "f", "d"], ["a", "c", "d"], ["e", "c", "d", 9, 2, 4, 5]],
[[10, "dd3"], ["ab", "d", 89], [10, 2]],
[[10, "b", 2, "d"], ["a", "e", "c", "d", "d", "f"], ["f", 80]]
],
[
[["a", "b"], ["a", "e", "c1", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[["b", 2, "d"], ["a", "e", "c", "d"], ["f", 2, "d", 80]],
[["10", "d"], ["a", "d", 89], [10, 2, 2, 80]]
],
[
[["a", "b", "f", "d"], ["a", "c", "d"], ["e", "c", "d", 9, 2, 4, 5]],
[[10, "b", 2, "d"], ["f", 9], ["f", 80]],
[["10", "dd1"], ["ab", "d", 89], [10, 2]]
]
],
[
[
[["a", "b", "c", "d"], ["a", "e", "c", "d"], ["e", "c", "d", 1, 2, 4, 5]],
[[1, "b", 2, "d"], ["a", "e", "c", "d"], ["f", 2, "d", 80]],
[["10", 0, "d"], ["a", "d", 89], [10, 2, 2, 80]]
],
[
[["a12", "b", "f", "d"], ["a", "c", "d"], ["e", "c", "d", 9, 2, 4, 5]],
[[10, "dd3"], ["ab", "d", 89], [10, 2]],
[[10, "b", 2, "d"], ["a", "e", "c", "d", "d", "f"], ["f", 80]]
],
[
[["a", "b"], ["a", "e", "c1", "d"], ["e", "d", 1, 2, 4, 5]],
[["b", 22, "d"], ["a", "e", "c", "d"], ["f", 2, "d", 80]],
[["10", "d"], ["a", "d", 89], [10, 2, 2, 80]]
],
[
[["a", "b", "f", "d"], [], ["e", "c", "d", 9, 2, 4, 5]],
[[10, "b", 2, "d"], ["f", 9], ["f", 80]],
[["10", "dd1"], ["ab", "d", 89], [10, 2, "4"]]
]
]
]

40. nested_b_t1.json
[
{
"key3": [[[[[[[[[[1, 2, 4, 5]]], [[[8, 7, 3, 5]]]]]]]]]],
"key4": [7, 8]
},
{
"key5": "val5",
"key6": "val6"
}
]

41. nested_b_t2.json
[
{
"key5": "CHANGE",
"key6": "val6"
},
{
"key3": [[[[[[[[[[1, 3, 5, 4]]], [[[8, 8, 1, 5]]]]]]]]]],
"key4": [7, 8]
}
]

42. $ deepdiff nested_b_t1.json nested_b_t2.json
{ 'dictionary_item_added': [root[0]['key6'], root[0]['key5'],
root[1]['key4'], root[1]['key3']],
'dictionary_item_removed': [root[0]['key4'], root[0]['key3'],
root[1]['key6'], root[1]['key5']]}

43. $ deepdiff nested_b_t1.json nested_b_t2.json --ignore-order
{ 'values_changed': { 'root[0]': { 'new_value': { 'key5': 'CHANGE',
'key6': 'val6'},
'old_value': { 'key3': [ [ [ [ [ [ [ [ [ [ 1,
2,
4,
5]]],
[ [ [ 8,
7,
3,
5]]]]]]]]]],
'key4': [7, 8]}},
'root[1]': { 'new_value': { 'key3': [ [ [ [ [ [ [ [ [ [ 1,
3,
5,
4]]],
[ [ [ 8,
8,
1,
5]]]]]]]]]],
'key4': [7, 8]},
'old_value': { 'key5': 'val5',
'key6': 'val6'}}}}

44. $ deepdiff --help
--cutoff-distance-for-pairs FLOAT
[default: 0.3]
--cutoff-intersection-for-pairs FLOAT
[default: 0.7]
--cache-size INTEGER [default: 0]
--cache-tuning-sample-size INTEGER
[default: 0]
--cache-purge-level INTEGER RANGE
[default: 1]
--create-patch [default: False]
--exclude-paths TEXT
--exclude-regex-paths TEXT
--get-deep-distance [default: False]
--ignore-order [default: False]
--ignore-string-type-changes [default: False]
--ignore-numeric-type-changes [default: False]
--ignore-type-subclasses [default: False]
--ignore-string-case [default: False]
--ignore-nan-inequality [default: False]
--include-private-variables [default: False]
--log-frequency-in-sec INTEGER [default: 0]
--max-passes INTEGER [default: 10000000]
--max_diffs INTEGER
--number-format-notation [f|e] [default: f]
--progress-logger [info|error] [default: info]
--report-repetition [default: False]
--significant-digits INTEGER
--truncate-datetime [second|minute|hour|day]
--verbose-level INTEGER RANGE [default: 1]
--help Show this message and exit.

45. $ deepdiff nested_b_t1.json nested_b_t2.json --ignore-order
--cache-size 5000 --cutoff-intersection-for-pairs 1
{ 'iterable_item_removed': {"root[0]['key3'][0][0][0][0][0][0][1][0][0][1]": 7},
'values_changed': { "root[0]['key3'][0][0][0][0][0][0][0][0][0][1]": { 'new_value': 3,
'old_value': 2},
"root[0]['key3'][0][0][0][0][0][0][1][0][0][2]": { 'new_value': 1,
'old_value': 3},
"root[1]['key5']": { 'new_value': 'CHANGE',
'old_value': 'val5'}}}

46. File Types Supported
Json
Yaml
Toml
CSV
Pickle

47. Conclusion

48. Delta for your data
How to efficiently track and store deeply nested data changes
By Sep Dehpour
Oct 2020