Talk delivered at CodeStar Night, discussing details of data representation and serialisation.

1. Video

2. The nitty gritty of data serialisation
Maxim Zaks
@iceX33

3. _

4. One bit
2^1
true / false

5. _ _ _ _ _ _ _ _

7. One byte
2^8 = 256

8. {0..255}
{-128..0..127}
{ASCII / UTF-8}

9. _ _ _ _ _ _ _ _ | _ _ _ _ _ _ _ _

10. ___ | ___

11. Little endian
Big endian

12. 005 | 000
000 | 005

13. Two bytes
2^16 = 65,536

14. {0 .. 65,535}
{-32,768 .. 0 .. 32,767}
{UTF-16}

15. ___ | ___ | ___ | ___

16. 2^32 = 4,294,967,296

17. {0 .. 4,294,967,296}
{-2,147,483,648 .. 0 .. 2,147,483,647 }
{0, -0, NaN, Inf, -Inf, SinglePrecision}
{UTF-32}
{Pointer on arch-32}

18. ___ | ___ | ___ | ___ | ___ | ___ | ___ | ___

19. 2^64 = 1.844674407E19

20. {0 .. 1.844674407E19}
{-9.223372037E18 .. 0 .. 9.223372037E18}
{0, -0, NaN, Inf, -Inf, DoublePrecision}
{Pointer on arch-64}

21. Real numbers / IEEE 754

22. https://en.wikipedia.org/wiki/Single-precision_floating-
point_format

23. https://en.wikipedia.org/wiki/Double-precision_floating-
point_format

24. https://en.wikipedia.org/wiki/Half-precision_floating-
point_format

25. String / Text

26. https://en.wikipedia.org/wiki/ASCII

27. Unicode
• Code
• UTF-8
• UTF-16
• UTF-32
• etc...

28. https://en.wikipedia.org/wiki/Unicode

29. https://en.wikipedia.org/wiki/UTF-8

30. https://en.wikipedia.org/wiki/UTF-16

31. Human readable / text based serialisation
• CSV
• EDIFACT
• XML
• JSON
• YAML
• TOML
• Java property file

32. Second level encoding

33. Int as text
• Base 10
• Worst case 3x (* UTF-?) size
• Parsing overhead

34. Real number as text
• Min 3 bytes to ...
• What you see is not what you get *
• Parsing overhead

35. Bytes as Text
• Base 16 (2^4) - 2x
• Base 64 (2^6) - 1.3x

36. Form of structured data representation
• Table (row / column based)
• Tree / hierarchy
• Graph

37. Bit packing techniques

38. Variable length quantity
https://en.wikipedia.org/wiki/Variable-length_quantity

39. VLQ + ZigZag

40. Pack Multiple values in one byte
• Array of booleans
• Multiple enum values

41. Deduplication
• Lookup table as in GIF
• Most frequent values only

42. Storing Deltas

43. Memory alignment
• Fast/direct access
• Padding

45. Good Talk on Parque and Arrow
https://www.youtube.com/watch?v=dPb2ZXnt2_U

46. Binary serialisation
• Format specification
• Code
• Schema
• Tooling for human readable

47. Format specification
• Support for endianness and archs
• Bit-packed or memory aligned
• Need for temporary memory allocation
• Form of data representation (see schema)
• Evolution strategies

48. Code
• Code generation / library
• Supported languages
• Computational efficiency
• Memory vs. speed

49. Schema
• Schema based, or self described
• Language and tooling
• High level concepts in regards to data representation and
evolution
• Capabilities for documentation and meta data

50. Tooling for human readable
• Convert to and from a text based serialisation format
• GUI
• Data importer (e.g. DB integration)

51. Benefits of binary formats
• Compact size
• Efficient read and write
• Data representation
• Human readable

52. Environment
• Data centres consume 4% of world wide electricity

53. Thank you!
Maxim Zaks @iceX33
Questions?