Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Talk Binary to Me

402 views

Published on

Presentation held at AppBuiders conference 2017

Published in: Education
  • Be the first to comment

  • Be the first to like this

Talk Binary to Me

  1. 1. Talk Binary to Me Maxim Zaks @iceX33
  2. 2. What’s going to happen here??? • Rant about JSON • Explain why binary formats are better • Save the world
  3. 3. I love JSON
  4. 4. Is this JSON Valid?
  5. 5. Is this JSON Valid?
  6. 6. Is this JSON Valid?
  7. 7. 0 22,5 45 67,5 90 JSON JSON minified FlexBuffers FlatBuffers colfer 22 40 70 66 81 Size in bytes
  8. 8. WTF is colfer?
  9. 9. Colfer is binary serialization format optimized for speed and size https://github.com/pascaldekloe/colfer
  10. 10. https://github.com/eishay/jvm-serializers/wiki Ser Time+Deser Time (ns)
  11. 11. { "name":"Maxim Zaks", "birthday":{ "year":1981, "month":6, "day":12 } }
  12. 12. { "name":"Maxim Zaks", "birthday":{ "year":1981, "month":6, "day":12 } } package demo type Person struct { name text birthday Date } type Date struct { year uint16 month uint8 day uint8 }
  13. 13. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  14. 14. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  15. 15. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  16. 16. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  17. 17. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  18. 18. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  19. 19. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  20. 20. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  21. 21. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  22. 22. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  23. 23. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  24. 24. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  25. 25. 0: [10] Maxim Zaks 1: 0: 1981 1: 6 2: 12 <END> <END>
  26. 26. GENERATED CODE
  27. 27. colfer all the things?
  28. 28. Data serialisation use cases • Machine to Machine communication • State persistence • Representation of configurations or data dumps
  29. 29. Machine to Machine Requirements • Small payload • Efficient encoding & decoding • Formalised / Typed API • Versioning & compatibility
  30. 30. colfer • Small payload 👍 • Efficient encoding & decoding 👍 • Formalised / Typed API 👍 • Versioning & compatibility 👎
  31. 31. can we do better?
  32. 32. FlatBuffers is an efficient cross platform serialization library for games and other memory constrained apps. https://github.com/google/flatbuffers
  33. 33. Swift port from yours truly https://github.com/mzaks/ FlatBuffersSwift
  34. 34. 0 1500 3000 4500 6000 JSONSerialization FlexBuffersSwift FlatBuffersSwift 17335 4.771 559 848 5.794 encode in ms decode in ms https://github.com/mzaks/FlexBuffersSwift
  35. 35. 0 125 250 375 500 JSONSerialization FlexBuffersSwift FlatBuffersSwift 0,0111,563 235,762 77 102 436 encode in MB decode in MB https://github.com/mzaks/FlexBuffersSwift
  36. 36. package demo type Person struct { name text birthday Date } type Date struct { year uint16 month uint8 day uint8 } table Person { name : string; birthday : Date; } struct Date { year : short; month : byte; day : byte; } root_type Person;
  37. 37. root table: 12 vTable size: 8 table size: 12 property offsets: 1: 4 2: 8 vTable: 8 name offset: 8 date inline: 1981 6 12 name:[10]Maxim Zaks[0]
  38. 38. root table: 12 vTable size: 8 table size: 12 property offsets: 1: 4 2: 8 vTable: 8 name offset: 8 date inline: 1981 6 12 name:[10]Maxim Zaks[0]
  39. 39. root table: 12 vTable size: 8 table size: 12 property offsets: 1: 4 2: 8 vTable: 8 name offset: 8 date inline: 1981 6 12 name:[10]Maxim Zaks[0]
  40. 40. root table: 12 vTable size: 8 table size: 12 property offsets: 1: 4 2: 8 vTable: 8 name offset: 8 date inline: 1981 6 12 name:[10]Maxim Zaks[0]
  41. 41. root table: 12 vTable size: 8 table size: 12 property offsets: 1: 4 2: 8 vTable: 8 name offset: 8 date inline: 1981 6 12 name:[10]Maxim Zaks[0]
  42. 42. root table: 12 vTable size: 8 table size: 12 property offsets: 1: 4 2: 8 vTable: 8 name offset: 8 date inline: 1981 6 12 name:[10]Maxim Zaks[0]
  43. 43. FlatBuffers • Small payload 👍* • Efficient encoding & decoding 👍* • Formalised / Typed API 👍 • Versioning & compatibility 👍 * not as small/fast as colfer
  44. 44. Let’s top that!
  45. 45. Data serialisation use cases • Machine to Machine communication • State persistence • Representation of configurations or data dumps
  46. 46. Application state is interconnected
  47. 47. table Message { ... } table MessageLink { message : Message; nextMessages : [MessageLink]; } table History { seenMessages : [Message]; nextMessageLink : MessageLink; timeStamp : double; } root_type History; http://resiapp.io
  48. 48. Data serialisation use cases • Machine to Machine communication • State persistence • Representation of configurations or data dumps
  49. 49. Random value access is KING!
  50. 50. Show case: find city by name or country code https://github.com/mzaks/FindCityFB
  51. 51. Data dump of all cities in the world with country code, name, normalised name and map coordinates 150 MB as CSV 177 MB as FlatBuffers binary
  52. 52. table CityList { cityByCountryCode : [City]; cityByName : [City]; } table City { countryCode : string; searchName : string; name : string; latitude : float; longitude : float; } root_type CityList;
  53. 53. https://youtu.be/P5tRwY39_JU
  54. 54. 45MB214MB
  55. 55. table CityList { cityByCountryCode : [City]; cityByName : [City]; } table City { countryCode : string; searchName : string; name : string; latitude : float; longitude : float; } root_type CityList; by Name by Country Code
  56. 56. Are we done?
  57. 57. There was one more format
  58. 58. 0 1500 3000 4500 6000 JSONSerialization FlexBuffersSwift FlatBuffersSwift 17335 4.771 559 848 5.794 encode in ms decode in ms
  59. 59. Schema-less binary encoding http://google.github.io/flatbuffers/ flexbuffers.html https://github.com/mzaks/FlexBuffersSwift
  60. 60. Benefits of FlexBuffers • No schema, no code generation needed • Random value access • Can easily transform JSON to FlexBuffers • Supports every type JSON does and a bit more • Keeps type information in the binary
  61. 61. let data = FlexBuffer.dataFrom(jsonData: "{name:"Maxim", birthday:{"year": 1981, month: 6, day: 12}}" .data(using: .utf8)! ) let accessor = FlexBuffer.decode(data:data) let name = accessor?["name"]?.asString let day = accessor?["birthday"]?["day"]?.asInt
  62. 62. Outlook • Don’t use JSON just because other people do, think about your use case • Imagine how much more responses your server could deliver if it took 200x less time to process a request • Start thinking in Graphs not Trees • please ask questions after the session • … and btw. I like to collaborate 😉 @iceX33
  63. 63. References • All Gifs are powered by giphy.com • JSON number grammar diagrams taken from json.org • Binary Visualiser „HEX Fiend“ by ridiculous_fish http://ridiculousfish.com/hexfiend/ • https://github.com/eishay/jvm-serializers/wiki • https://github.com/pascaldekloe/colfer • https://github.com/google/flatbuffers • https://github.com/mzaks/FlatBuffersSwift • https://github.com/mzaks/FlexBuffersSwift • https://github.com/mzaks/FindCityFB • http://resiapp.io

×