Maxim Zaks: Deep dive into data serialisation
•
0 likes•140 views
This document compares different data serialization formats for persisting state and user generated content on mobile. JSON is human readable but large and inefficient for numbers. Protocol Buffers and FlatBuffers are binary formats that are more efficient in size and faster for reads/writes through the use of schemas and typed accessors. Both support evolution but Protocol Buffers has no partial reads while FlatBuffers enables this through references and random access.
More Related Content
What's hot
Similar to Maxim Zaks: Deep dive into data serialisation
Similar to Maxim Zaks: Deep dive into data serialisation (20)
More from mdevtalk
More from mdevtalk (20)
Recently uploaded
Recently uploaded (20)
Maxim Zaks: Deep dive into data serialisation
- 2. MAXIM ZAKS Freelance Software Developer
- 3. Deep dive into data serialisation
- 4. Persisting State / User generated Content Machine to Machine Communication Representation of Configuration / Read only data Why do we need data serialisation?
- 5. Custom binary representation Language provided binary serialisation Text based representation (CSV, XML, JSON, YAML) Embedded SQL or NoSQL DB Binary cross platform serialisation library (FlatBuffers, FlexBuffers Protocol Buffers, Cap’n Proto, SBE, Apache Thrift etc…) How can we persist data on mobile?
- 6. Size “on disk” Speed of read & write / partial read / memory consumption Human readable and writable Support of OO language type system Data versioning / evolution / migration Important criteria for persisting data
- 7. JSON vs. Protocol Buffers vs. FlatBuffers
- 8. Text based Self describing Weakly typed: Object, Array, String, Number, Bool, Null JSON
- 9. Binary IDL (interface definition language) based Schema Strongly typed: Message, fix length Numbers, variable length numbers, String, Bool, Enum, Bytes, Repeated values, Maps, Oneof, Any Evolution Strategy Protocol Buffers
- 10. Binary IDL (interface definition language) based Schema Strongly typed: Table, Struct, fix length Numbers, String, Bool, Enum, Bytes, typed Vector, Union Value / Reference type semantics Random Value access Evolution Strategy Flat Buffers
- 11. Size “on disk” Speed of read & write / partial read / memory consumption Human readable and writable Support of OO language type system Data versioning / evolution / migration Important criteria for persisting data
- 12. 👎 JSON, bad for representing numbers, lots of repetition, needs to be at least minified 👍 Protocol Buffers, stores only values and a bit of meta data, thanks to VLQ very efficient for storing numbers 🤔 Flat Buffers, has some overhead compared to Protocol Buffers because of Ref semantics and random value access Important criteria for persisting data: Size
- 13. 👎 JSON, text need to be parsed and translated to intermediate data representation 🤔 Protocol Buffers, no partial read, VLQ means you need to do some operations before value is available 👍 Flat Buffers, support partial reading thanks to Ref semantics and random value access mechanism Important criteria for persisting data: Speed of read & write / partial read / memory consumption
- 14. 👍 JSON, text based and can be nicely formatted 🤔 Protocol Buffers, provides tools for binary to JSON conversion and vice versa 🤔 Flat Buffers, provides tools for binary to JSON conversion and vice versa Important criteria for persisting data: Human readable and writable
- 15. 🤔 JSON, is generally weakly typed, there are ways to transform/validate against OO types 👍 Protocol Buffers, code generator creates Accessor classes which can be comfortably used for encoding and decoding 👍 Flat Buffers, code generator creates Accessor classes which can be comfortably used for decoding, performant encoding can be a bit painful Important criteria for persisting data: Support of OO language type system
- 16. 🤔 JSON, is implicit because of self describing nature 👍 Protocol Buffers, provides a set of rule how a schema can be evolved 👍 Flat Buffers, provides a set of rule how a schema can be evolved Important criteria for persisting data: Data versioning / evolution / migration
- 17. Size Efficiency Human Readable Types Evolution JSON 👎 👎 👍 🤔 🤔 Proto Buffers 👍 🤔 🤔 👍 👍 Flat Buffers 🤔 👍 🤔 👍 👍
- 18. Mobile Warsaw 2016: https://www.youtube.com/watch? v=OUNpNo-oyQY MobileTechCon 2016: https://medium.com/@icex33/beyond- json-introduction-to-flatbuffers-fba1dfd0dcfe AppBuilders 2017: https://www.youtube.com/watch? v=Ve9POqZJymw BerlinBuzzWords 2017: https://www.youtube.com/watch? v=qF44UetsLsQ Previous Talks
- 19. Protocol Buffers Encoding: https://developers.google.com/ protocol-buffers/docs/encoding FlatBuffers: https://google.github.io/flatbuffers/ FlatBuffersSwift: https://github.com/mzaks/FlatBuffersSwift FlexBuffersSwift: https://github.com/mzaks/FlexBuffersSwift Data Serialisation formats: https://en.wikipedia.org/wiki/ Comparison_of_data_serialization_formats JVM Serialisers: https://github.com/eishay/jvm-serializers More Links