Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Improving sanity with structured binary formats and protocols


Published on

Did you know that you can turn a JSON API into a binary one using MessagePack with one line of code?

There are a lot of myths surrounding working with binary formats, and a lot of things even the "experts" get wrong. Just as important as making a binary format that is high performance, is making one that is easy for developers to work with. This talk describes how not to implement a binary format, and then introduces structured ways to implement them that will maximize language compatibility and make everybody's lives easier.

An early version of this talk was introduced at Realtime Conf Europe, and was recorded here:

Published in: Technology
  • Be the first to comment

Improving sanity with structured binary formats and protocols

  1. Learning how to let go:Improving sanity with structured binary data-interchange formats and protocolsKyle DrakeNet Brew Ventures
  2. If you’re doing a lot oftext-based serializedobject passing, andwant a way to improveserialization time andpayload size, try abinary format.
  3. Why• Smaller size, representing the sameinformation (esp. vs XML)• Usually faster for CPU as aconsequence• Surprisingly easy to implement (withhelp)• Lots of well-tested, streamlinedsolutions
  4. Why Not• No quick language support (if custom)• Hard to understand without good docs• Potentially harder to debug low-level ifsomething bad happens (dataentanglement/corruption?)• Binary != Performance (Measure!)
  5. Two Approaches• Custom, hand-rolled format you write• Frameworks (Protocol Buffers,MessagePack, BERT, others)
  6. Custom Binary Format• This is often a bad idea.• No, seriously.• Requires a very compelling reasonIMHO.
  7. Let’s look at a poorlydesigned custom binaryformat, via: The ApplePush NotificationService.
  8. Wait, why notjust use JSONfor the wholething then?
  9. Success: no responseError: socket disconnectionInterim: unreported data lossW.T.F.
  11. My solution:
  12. Meanwhile, at Google:
  13. I think this crap is whypeople hate working withbinary so much.But there is a better way.
  14. Let’s look at somestructured ways to workwith binary!
  15. Protocol Buffers• Developed by Google• “3 to 10 times smaller, 20 to 100times faster than XML”• Requires a pre-defined .proto file• In effect, it has a language agnostic“schema”.
  16. Protocol Buffers
  17. Protocol Buffers
  18. Protocol BuffersI’m doing work with the Bitcoin protocolright now. I really wish it used protocolbuffers. Not just for size, but for safetyand ease of use.Bitcoin lead core developer is warm toidea:
  19. Let’s look at some“schemaless” structuredbinary formats.
  20. JSON• Number (50, 2.33)• String “howdy”• Boolean (true or false)• Array [1,2,3]• Object {“key”: “value”}• null (empty)
  21. MessagePackBasically the same thing.Except faster and smaller!Super simple:
  24. Painless API Integration• /route?format=msgpack• Content-Type: application/x-msgpack• Accept: application/x-msgpack• Plugs right in, if you’re using JSON.
  26. THANKS!