Improving sanity with structured binary formats and protocols

7,870 views
7,634 views

Published on

Did you know that you can turn a JSON API into a binary one using MessagePack with one line of code?

There are a lot of myths surrounding working with binary formats, and a lot of things even the "experts" get wrong. Just as important as making a binary format that is high performance, is making one that is easy for developers to work with. This talk describes how not to implement a binary format, and then introduces structured ways to implement them that will maximize language compatibility and make everybody's lives easier.

An early version of this talk was introduced at Realtime Conf Europe, and was recorded here: http://www.youtube.com/watch?v=ZlKrnOD-4TQ

Published in: Technology
0 Comments
13 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,870
On SlideShare
0
From Embeds
0
Number of Embeds
575
Actions
Shares
0
Downloads
34
Comments
0
Likes
13
Embeds 0
No embeds

No notes for slide

Improving sanity with structured binary formats and protocols

  1. Learning how to let go:Improving sanity with structured binary data-interchange formats and protocolsKyle DrakeNet Brew Ventures
  2. If you’re doing a lot oftext-based serializedobject passing, andwant a way to improveserialization time andpayload size, try abinary format.
  3. Why• Smaller size, representing the sameinformation (esp. vs XML)• Usually faster for CPU as aconsequence• Surprisingly easy to implement (withhelp)• Lots of well-tested, streamlinedsolutions
  4. Why Not• No quick language support (if custom)• Hard to understand without good docs• Potentially harder to debug low-level ifsomething bad happens (dataentanglement/corruption?)• Binary != Performance (Measure!)
  5. Two Approaches• Custom, hand-rolled format you write• Frameworks (Protocol Buffers,MessagePack, BERT, others)
  6. Custom Binary Format• This is often a bad idea.• No, seriously.• Requires a very compelling reasonIMHO.
  7. Let’s look at a poorlydesigned custom binaryformat, via: The ApplePush NotificationService.
  8. Wait, why notjust use JSONfor the wholething then?
  9. Success: no responseError: socket disconnectionInterim: unreported data lossW.T.F.
  10. ROFLSCALE TIPS FTW
  11. My solution:
  12. Meanwhile, at Google:
  13. I think this crap is whypeople hate working withbinary so much.But there is a better way.
  14. Let’s look at somestructured ways to workwith binary!
  15. Protocol Buffers• Developed by Google• “3 to 10 times smaller, 20 to 100times faster than XML”• Requires a pre-defined .proto file• In effect, it has a language agnostic“schema”.
  16. Protocol Buffers
  17. Protocol Buffers
  18. Protocol BuffersI’m doing work with the Bitcoin protocolright now. I really wish it used protocolbuffers. Not just for size, but for safetyand ease of use.Bitcoin lead core developer is warm toidea:https://bitcointalk.org/index.php?topic=632.msg6656#msg6656
  19. Let’s look at some“schemaless” structuredbinary formats.
  20. JSON• Number (50, 2.33)• String “howdy”• Boolean (true or false)• Array [1,2,3]• Object {“key”: “value”}• null (empty)
  21. MessagePackBasically the same thing.Except faster and smaller!Super simple:
  22. SOURCE: INDIEGAMR.COMHTTP://INDIEGAMR.COM/CUT-YOUR-DATA-EXCHANGE-TRAFFIC-BY-UP-TO-50-WITH-ONE-LINE-OF-CODE-MSGPACK-VS-JSON/
  23. BINARYJS.COM
  24. Painless API Integration• /route?format=msgpack• Content-Type: application/x-msgpack• Accept: application/x-msgpack• Plugs right in, if you’re using JSON.
  25. HAVE A JSON API? TRY ASCHEMALESS STRUCTUREDBINARY FORMAT! IT’S EASY.
  26. THANKS!

×