Improving sanity with structured binary formats and protocols

  • 6,439 views
Uploaded on

Did you know that you can turn a JSON API into a binary one using MessagePack with one line of code? …

Did you know that you can turn a JSON API into a binary one using MessagePack with one line of code?

There are a lot of myths surrounding working with binary formats, and a lot of things even the "experts" get wrong. Just as important as making a binary format that is high performance, is making one that is easy for developers to work with. This talk describes how not to implement a binary format, and then introduces structured ways to implement them that will maximize language compatibility and make everybody's lives easier.

An early version of this talk was introduced at Realtime Conf Europe, and was recorded here: http://www.youtube.com/watch?v=ZlKrnOD-4TQ

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
6,439
On Slideshare
0
From Embeds
0
Number of Embeds
10

Actions

Shares
Downloads
31
Comments
0
Likes
12

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Learning how to let go:Improving sanity with structured binary data-interchange formats and protocolsKyle DrakeNet Brew Ventures
  • 2. If you’re doing a lot oftext-based serializedobject passing, andwant a way to improveserialization time andpayload size, try abinary format.
  • 3. Why• Smaller size, representing the sameinformation (esp. vs XML)• Usually faster for CPU as aconsequence• Surprisingly easy to implement (withhelp)• Lots of well-tested, streamlinedsolutions
  • 4. Why Not• No quick language support (if custom)• Hard to understand without good docs• Potentially harder to debug low-level ifsomething bad happens (dataentanglement/corruption?)• Binary != Performance (Measure!)
  • 5. Two Approaches• Custom, hand-rolled format you write• Frameworks (Protocol Buffers,MessagePack, BERT, others)
  • 6. Custom Binary Format• This is often a bad idea.• No, seriously.• Requires a very compelling reasonIMHO.
  • 7. Let’s look at a poorlydesigned custom binaryformat, via: The ApplePush NotificationService.
  • 8. Wait, why notjust use JSONfor the wholething then?
  • 9. Success: no responseError: socket disconnectionInterim: unreported data lossW.T.F.
  • 10. ROFLSCALE TIPS FTW
  • 11. My solution:
  • 12. Meanwhile, at Google:
  • 13. I think this crap is whypeople hate working withbinary so much.But there is a better way.
  • 14. Let’s look at somestructured ways to workwith binary!
  • 15. Protocol Buffers• Developed by Google• “3 to 10 times smaller, 20 to 100times faster than XML”• Requires a pre-defined .proto file• In effect, it has a language agnostic“schema”.
  • 16. Protocol Buffers
  • 17. Protocol Buffers
  • 18. Protocol BuffersI’m doing work with the Bitcoin protocolright now. I really wish it used protocolbuffers. Not just for size, but for safetyand ease of use.Bitcoin lead core developer is warm toidea:https://bitcointalk.org/index.php?topic=632.msg6656#msg6656
  • 19. Let’s look at some“schemaless” structuredbinary formats.
  • 20. JSON• Number (50, 2.33)• String “howdy”• Boolean (true or false)• Array [1,2,3]• Object {“key”: “value”}• null (empty)
  • 21. MessagePackBasically the same thing.Except faster and smaller!Super simple:
  • 22. SOURCE: INDIEGAMR.COMHTTP://INDIEGAMR.COM/CUT-YOUR-DATA-EXCHANGE-TRAFFIC-BY-UP-TO-50-WITH-ONE-LINE-OF-CODE-MSGPACK-VS-JSON/
  • 23. BINARYJS.COM
  • 24. Painless API Integration• /route?format=msgpack• Content-Type: application/x-msgpack• Accept: application/x-msgpack• Plugs right in, if you’re using JSON.
  • 25. HAVE A JSON API? TRY ASCHEMALESS STRUCTUREDBINARY FORMAT! IT’S EASY.
  • 26. THANKS!