Sadayuki Furuhashi, profile picture
Uploaded bySadayuki Furuhashi
PDF, PPTX4,855 views

Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019

This document summarizes Sadayuki Furuhashi's presentation on performance optimization techniques for MessagePack-Ruby. It introduces MessagePack as a data format like JSON but faster and more compact. It discusses how MessagePack has a language agnostic type system and is supported by developers worldwide. Examples are given of how MessagePack is used in large-scale systems and by major projects like Fluentd for high performance log collection and storage.

Embed presentation

Download as PDF, PPTX
Performance Optimization Techniques of MessagePack-Ruby Sadayuki Furuhashi RubyKaigi 2019 #MyMessagePack Tweet your msgpack usage with
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 About me A founder of Treasure Data, Inc. Located in Silicon Valley, USA. OSS Hacker. Github: @frsyuki
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 Basics of MessagePack
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 What’s MessagePack? { “compact”: true, “schema”: 0 } 82 A7 compact C3 A6 schema 00 JSON MessagePack It’s like JSON, but fast and small. 7-byte string2-element map true 6-byte string 0 27 bytes 18 bytes (34% smaller)
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 What’s MessagePack? It’s like JSON, but fast and small. > Self-descriptive, Schema-on-Read semantics > Everyone knows > De facto standard data format > Human-readable > Self-descriptive, Schema-on-Read semantics > Everyone who uses JSON can use > Drop-in improvement of JSON > Machine-readable JSON MessagePack
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 timestamp 32 timestamp 64 timestamp 96 Language Agnostic Type System MessagePack
 Type System MessagePack
 Format String Timestamp Language Types fixstr str 8 str 16 str 32 String Timestamp (JSON compatible)(Ruby, Swift, Java, Go, …) Convert
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Supported by Super Skilled Engineers All Over The World Ruby /msgpack
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack-Ruby Major Committers
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Real World MessagePack We import over 2,000,000 records/sec and
 store 30PB of data in MessagePack format. 15 trillion rows processed every day. Sada Furuhashi, Arm Treasure Data Sada Furuhashi, Initial Creator of Fluentd MessagePack is an essential component of Fluentd to achieve high performance and flexibility at the same time.
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Adoption of MessagePack Today Mobile Apps Microprocessors Automotive Telematics Sensors Cloud Infrastructure Middleware Machine Learning Games Analytical Databases MessgePack Zero-overhead, heterogeneous data exchange
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 MessagePack-Ruby implementation
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 4-element array
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 4-element array 01 Integer 1
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 4-element array 01 Integer 1 02 Integer 2
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 4-element array 01 Integer 1 02 Integer 2 81 1-element map
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 01 02 81 A1 1-byte string 4-element array Integer 1 Integer 2 1-element map
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 01 02 81 ‘k’ ‘k’ A1 1-byte string 4-element array Integer 1 Integer 2 1-element map
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 01 02 81 ‘k’ ‘k’ A1 1-byte string ‘v’ ‘v’ A1 1-byte string 4-element array Integer 1 Integer 2 1-element map
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 nil 4-element array Integer 1 Integer 2 1-element map ‘k’ 1-byte string ‘v’ 1-byte string
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ □, □, □, □ ] 4-element array
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ □, □, □, □ ] 1 Integer 1
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, □, □, □ ] 1 complete object
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, □, □, □ ] 2 Integer 2
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] 2 complete object
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { □ => □ }1-element map
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { □ => □ } “k”1-byte string
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { “k” => □ } “k” complete object
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { “k” => □ } “v” 1-byte string
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { “k” => “v” } “v” complete object
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [1, 2,{“k”=>“v”}, □ ] { “k” => “v” } complete object
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [1, 2,{“k”=>“v”}, □ ] nil nil
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [1, 2,{“k”=>“v”},nil] nil complete object
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [1, 2,{“k”=>“v”},nil] Complete object
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 Optimization
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Buffer MessagePack::Buffer next mapped_string mem { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] Object 83 A5 ‘c’ ‘o’ ’u’ ’n’ ’t' 01 A4 ‘p’ ’a’ ’g’ ’e’ A4 ‘b’ ‘o’ ’d’ ’y’
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Buffer MessagePack::Buffer next mapped_string mem next mapped_string mem { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] Object 83 A5 ‘c’ ‘o’ ’u’ ’n’ ’t' 01 A4 ‘p’ ’a’ ’g’ ’e’ A4 ‘b’ ‘o’ ’d’ ’y’ 91 A9 ‘L’ ‘O’ ’N’ ’G’ ’ ’ ’T‘ ’E’ ‘X’ ’T’ Add more buffer chunks instead of realloc()
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Zero-copy write optimization MessagePack::Buffer next mapped_string mem next mapped_string mem next mapped_string mem { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] Object 83 A5 ‘c’ ‘o’ ’u’ ’n’ ’t' 01 A4 ‘p’ ’a’ ’g’ ’e’ A4 ‘b’ ‘o’ ’d’ ’y’ 91 A9 “LONG TEXT” rb_str_dup() Fast copy-on-write
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Zero-copy read optimization MessagePack::Buffer next mapped_string mem Object { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] rb_str_substr() Fast copy-on-write x83 A5 c o u n t t x01 xA4 p a g e xA4 b o d y x91 A9 L O N G
 T E X T ” “ ※ if SHARABLE_SUBSTRING_P() returns true x83 A5 c o u n t t x01 xA4 p a g e xA4 b o d y x91 A9 L O N G
 T E X T ” Source String “ rb_str_dup() Fast copy-on-write
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Reserved memory pool MessagePack::Buffer next mapped_string mem Global memory pool next mapped_string mem 4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB 83 A5 ‘c’ ‘o’ ’u’ ’n’ ’t' 01 A4 ‘p’ ’a’ ’g’ ’e’ A4 ‘b’ ‘o’ ’d’ ’y’ 91 A9
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Reserved memory pool MessagePack::Buffer next mapped_string mem Global memory pool next mapped_string mem 4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 Benchmark & further optimization
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Benchmark Data Sets DB 100_000.times.map {|i| r.rand(i+1) } [{ "id" : "gfmg-6ppu", "name" : "Mortgage Complaints", "averageRating" : 0, "createdAt" : 1433953219, "moderationStatus" : true, “numberOfComments" : 0, "description" : "Each week we …”, … Blogs Integers [ 0,0,0,1,3,0,0, …,
 42991,26906,18655,7015
 ] { “results”: [ { “attachments”:[], “body”:
 ”Dear Friends and Colleagues,nn
 I always look forward to the… https://www.justice.gov/api/v1/blog_entries.json http://data.consumerfinance.gov/api/views.json Benchmark code available at https://gist.github.com/frsyuki/9777c4adba2b5c957695b64f17b64ba1
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack vs JSON DB Integers Blogs 0 600 1200 1800 2400 3000 Serialization time MessagePack JSON (Oj) JSON (JSON) 100% DB Integers Blogs 0 600 1200 1800 2400 3000 Deserialization time 100%
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack without optimization DB Integers Blogs 0 30 60 90 120 150 Serialization time 100% DB Integers Blogs 0 30 60 90 120 150 Deserialization time 100% Default No memory pool No zero-copy read
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Zero-copy read optimization MessagePack::Buffer next mapped_string mem Object { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] x83 A5 c o u n t t x01 xA4 p a g e xA4 b o d y x91 L O N G
 T E X T ” “x83 A5 c o u n t t x01 xA4 p a g e xA4 b o d y x91 L O N G
 T E X T ” Source String “ rb_str_dup() Fast copy-on-write rb_str_substr() Fast copy-on-write ※ if SHARABLE_SUBSTRING_P() returns true
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Copy-on-write substring SHARABLE_SUBSTRING_P() returns true only when substring shares the last 0 termination with the original string s = "a" * 1_000_100 10_000.times do s.slice(100, 1_000_000) end #=> 0.002 sec s = "a" * 1_000_100 10_000.times do s.slice(100, 1_000_000 - 1) end #=> 2.7 sec Not including the last character disables Copy-on-Write
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Copy-on-write substring SHARABLE_SUBSTRING_P() returns true only when substring shares the last 0 termination with the original string OR Ruby is compiled with SHARABLE_MIDDLE_SUBSTRING flag (not enabled by default) s = "a" * 1_000_100 10_000.times do s.slice(100, 1_000_000) end #=> 0.002 sec s = "a" * 1_000_100 10_000.times do s.slice(100, 1_000_000 - 1) end #=> 2.7 sec 0.002 sec Using ruby binary compiled with SHARABLE_MIDDLE_SUBSTRING
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Deserialization with copy-on-write substring DB Integers Blogs 0 30 60 90 120 150 Serialization time 100% DB Integers Blogs 0 30 60 90 120 150 Deserialization time Default With SHARABLE_MIDDLE_SUBSTRING
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 What is deserialization bottleneck? 0 14,000,000 28,000,000 42,000,000 56,000,000 70,000,000 Deserialization objects/sec DB Integers Blogs Boolean (2^62)-1 (2^62) Immediate On-Heap On-Heap
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 What is deserialization bottleneck? 0 14,000,000 28,000,000 42,000,000 56,000,000 70,000,000 Deserialization objects/sec DB Integers Blogs Boolean (2^62)-1 (2^62) Immediate Immediate Immediate On-Heap => Object allocation is slow On-Heap On-Heap
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Reusing Hash key objects data: [ { “id”: “s6ew-h6mp”, “name”: “Consumer Complaints”, … }, { “id”: “nsyy-je5y”, “name”: “Beta Consumers”, … } { “id”: “wkue-ycpk”, “name”: “Survey”, … } ] Same key repeats. Keys of Hash are always frozen. => We can reuse objects!
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 How to reuse Hash key objects? Use fstring. Ruby uses it to reuse same objects for immutable strings (but C API is not available…yet): p “a”.object_id == “a”.object_id #=> false # frozen_string_literal: true p “a”.object_id == “a”.object_id #=> true p (-“a”).object_id == (-“a”).object_id #=> true
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Reusing Hash key objects using fstring DB Integers Blogs 0 30 60 90 120 150 Serialization time 100% DB Integers Blogs 0 25 50 75 100 125 150 Deserialization time Default Hash key fstring using
 hacked Ruby binary
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Link Time Optimization DB Integers Blogs 0 30 60 90 120 150 Serialization time 100% DB Integers Blogs 0 25 50 75 100 125 150 Deserialization time All optimization And -flto=thin
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 Reading other MessagePack implementations
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 msgpack-java: TypeProfile bypassing interface MessageBuffer { public void putInt(int index, int value); } class MessageBufferBE implements MessageBuffer { public void putInt(int index, int value) { byteBuffer.putInt(index, value); } } This code has significant overhead:
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 msgpack-java: TypeProfile bypassing interface MessageBuffer { public void putInt(int index, int value); } class MessageBufferBE implements MessageBuffer { public void putInt(int index, int value) { byteBuffer.putInt(index, value); } } This code has significant overhead: Dynamic method lookup
 (slow even after JIT) Dynamic method lookup
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 msgpack-java: TypeProfile bypassing interface MessageBuffer { public void putInt(int index, int value) { v = Integer.reverseBytes(v); unsafe.putInt(base, address + index, v); } public static MessageBuffer newInstance() { // … } Much faster code: No override. JVM JIT inlines them.
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 msgpack-java: TypeProfile bypassing interface MessageBuffer { public void putInt(int index, int value) { v = Integer.reverseBytes(v); unsafe.putInt(base, address + index, v); } public static MessageBuffer newInstance() { // … } Much faster code: JVM intrinsics
 (1-to-1 mapping to CPU instruction,
 no function call) JVM intrinsics Load inherited class lazily.
 No override on little-endian machine. No override. JVM JIT inlines them.
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack-CSharp: Lookup cache of optimized Ser/De class User { public int Age { get; set; } public string FirstName { get; set; } public string LastName { get; set; } } [ 31, “Sadayuki”, “Furuhashi” ] Mapping between class and semi-structured data Java: Jackson databind, JAXB, … C#: MessagePack-CSharp, System.Runtime.Serialization, … Swift: Codable, SwiftMsgPack, … …
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack-CSharp: Native LZ4 integration
1 1 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack-CSharp: Native LZ4 integration
It's like JSON.
 but fast and small. Sadayuki Furuhashi #MyMessagePack Tweet your msgpack usage with

More Related Content

PDF
Kafka Streams: Perfectly Balanced as all things should be
byconfluent
 
PDF
Cassandra 2.1
byjbellis
 
PDF
Php user groupmemcached
byJason Anderson
 
PDF
Cassandra Summit 2015
byjbellis
 
PDF
Cassandra summit keynote 2014
byjbellis
 
PDF
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
byInfluxData
 
PDF
Cassandra EU - Data model on fire
byPatrick McFadin
 
PDF
Vielseitiges In-Memory Computing mit Apache Ignite und Kubernetes
byQAware GmbH
 
Kafka Streams: Perfectly Balanced as all things should be
byconfluent
 
Cassandra 2.1
byjbellis
 
Php user groupmemcached
byJason Anderson
 
Cassandra Summit 2015
byjbellis
 
Cassandra summit keynote 2014
byjbellis
 
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...
byInfluxData
 
Cassandra EU - Data model on fire
byPatrick McFadin
 
Vielseitiges In-Memory Computing mit Apache Ignite und Kubernetes
byQAware GmbH
 

What's hot

PDF
Probabilistic Logic Programming with Beta-Distributed Random Variables
byFederico Cerutti
 
PPTX
Jsr107 come, code, cache, compute!
byC2B2 Consulting
 
PDF
Image classification with Deeplearning4j
byEric Jain
 
PDF
Spark Streaming and Suicidal Tendencies
byDataWorks Summit/Hadoop Summit
 
PDF
The Ring programming language version 1.6 book - Part 80 of 189
byMahmoud Samir Fayed
 
ODP
Caching and tuning fun for high scalability
byWim Godden
 
PDF
CCM Escape Case Study - SkySQL Paris Meetup 17.12.2013
byMariaDB Corporation
 
PDF
Common scenarios in vcl
byVarnish Software
 
PDF
Parallel Computing With Dask - PyDays 2017
byChristian Aichinger
 
PDF
Elasticsearch sur Azure : Make sense of your (BIG) data !
byMicrosoft
 
TXT
Logs
byaloha ragudo
 
Probabilistic Logic Programming with Beta-Distributed Random Variables
byFederico Cerutti
 
Jsr107 come, code, cache, compute!
byC2B2 Consulting
 
Image classification with Deeplearning4j
byEric Jain
 
Spark Streaming and Suicidal Tendencies
byDataWorks Summit/Hadoop Summit
 
The Ring programming language version 1.6 book - Part 80 of 189
byMahmoud Samir Fayed
 
Caching and tuning fun for high scalability
byWim Godden
 
CCM Escape Case Study - SkySQL Paris Meetup 17.12.2013
byMariaDB Corporation
 
Common scenarios in vcl
byVarnish Software
 
Parallel Computing With Dask - PyDays 2017
byChristian Aichinger
 
Elasticsearch sur Azure : Make sense of your (BIG) data !
byMicrosoft
 
Logs
byaloha ragudo
 

Similar to Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019

PPTX
Unpack mechanism of the msgpack-c
byTakatoshi Kondo
 
PDF
MessagePack(msgpack): Compact and Fast Serialization Library
byTakatoshi Kondo
 
PDF
Architecture of MessagePack
bySadayuki Furuhashi
 
KEY
Erlang/OTP for Rubyists
bySean Cribbs
 
PDF
The Kumofs Project and MessagePack-RPC
bySadayuki Furuhashi
 
PDF
Build Your Own Tools
byShugo Maeda
 
PPTX
MessagePack - An efficient binary serialization format
byLarry Nung
 
KEY
RFC4627 Reading
byYuuki Tan-nai
 
PDF
Pharo: Syntax in a Nutshell
byMarcus Denker
 
PDF
1 hour dive into Erlang/OTP
byJordi Llonch
 
PDF
1 hour dive into erlang
byJoan Valduvieco
 
KEY
Erjang - A JVM-based Erlang VM
byKresten Krab Thorup
 
PDF
Crystal presentation in NY
byCrystal Language
 
PDF
Erlang
byESUG
 
PDF
Railswaycon Inside Matz Ruby
byLourens Naudé
 
PDF
Oop basic concepts
bySwarup Kumar Boro
 
PDF
Introduction to Ruby Programming Language
byNicolò Calcavecchia
 
PPT
What lies beneath the beautiful code?
byNiranjan Sarade
 
PDF
Fighting the Branch Predictor (ESUG 2025)
byESUG
 
ODP
RailswayCon 2010 - Dynamic Language VMs
byLourens Naudé
 
Unpack mechanism of the msgpack-c
byTakatoshi Kondo
 
MessagePack(msgpack): Compact and Fast Serialization Library
byTakatoshi Kondo
 
Architecture of MessagePack
bySadayuki Furuhashi
 
Erlang/OTP for Rubyists
bySean Cribbs
 
The Kumofs Project and MessagePack-RPC
bySadayuki Furuhashi
 
Build Your Own Tools
byShugo Maeda
 
MessagePack - An efficient binary serialization format
byLarry Nung
 
RFC4627 Reading
byYuuki Tan-nai
 
Pharo: Syntax in a Nutshell
byMarcus Denker
 
1 hour dive into Erlang/OTP
byJordi Llonch
 
1 hour dive into erlang
byJoan Valduvieco
 
Erjang - A JVM-based Erlang VM
byKresten Krab Thorup
 
Crystal presentation in NY
byCrystal Language
 
Erlang
byESUG
 
Railswaycon Inside Matz Ruby
byLourens Naudé
 
Oop basic concepts
bySwarup Kumar Boro
 
Introduction to Ruby Programming Language
byNicolò Calcavecchia
 
What lies beneath the beautiful code?
byNiranjan Sarade
 
Fighting the Branch Predictor (ESUG 2025)
byESUG
 
RailswayCon 2010 - Dynamic Language VMs
byLourens Naudé
 

More from Sadayuki Furuhashi

PDF
Scripting Embulk Plugins
bySadayuki Furuhashi
 
PDF
Making KVS 10x Scalable
bySadayuki Furuhashi
 
PDF
Automating Workflows for Analytics Pipelines
bySadayuki Furuhashi
 
PDF
Digdagによる大規模データ処理の自動化とエラー処理
bySadayuki Furuhashi
 
PDF
Fluentd at Bay Area Kubernetes Meetup
bySadayuki Furuhashi
 
PDF
DigdagはなぜYAMLなのか？
bySadayuki Furuhashi
 
PDF
Logging for Production Systems in The Container Era
bySadayuki Furuhashi
 
PDF
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
bySadayuki Furuhashi
 
PDF
Fighting Against Chaotically Separated Values with Embulk
bySadayuki Furuhashi
 
PDF
Embulk - 進化するバルクデータローダ
bySadayuki Furuhashi
 
PDF
Plugin-based software design with Ruby and RubyGems
bySadayuki Furuhashi
 
PDF
Embuk internals
bySadayuki Furuhashi
 
PDF
Embulk, an open-source plugin-based parallel bulk data loader
bySadayuki Furuhashi
 
PDF
Understanding Presto - Presto meetup @ Tokyo #1
bySadayuki Furuhashi
 
PDF
Prestogres internals
bySadayuki Furuhashi
 
PDF
Presto+MySQLで分散SQL
bySadayuki Furuhashi
 
PDF
Presto - Hadoop Conference Japan 2014
bySadayuki Furuhashi
 
PDF
Fluentd - Set Up Once, Collect More
bySadayuki Furuhashi
 
PDF
Prestogres, ODBC & JDBC connectivity for Presto
bySadayuki Furuhashi
 
PDF
What's new in v11 - Fluentd Casual Talks #3 #fluentdcasual
bySadayuki Furuhashi
 
Scripting Embulk Plugins
bySadayuki Furuhashi
 
Making KVS 10x Scalable
bySadayuki Furuhashi
 
Automating Workflows for Analytics Pipelines
bySadayuki Furuhashi
 
Digdagによる大規模データ処理の自動化とエラー処理
bySadayuki Furuhashi
 
Fluentd at Bay Area Kubernetes Meetup
bySadayuki Furuhashi
 
DigdagはなぜYAMLなのか？
bySadayuki Furuhashi
 
Logging for Production Systems in The Container Era
bySadayuki Furuhashi
 
分散ワークフローエンジン『Digdag』の実装 at Tokyo RubyKaigi #11
bySadayuki Furuhashi
 
Fighting Against Chaotically Separated Values with Embulk
bySadayuki Furuhashi
 
Embulk - 進化するバルクデータローダ
bySadayuki Furuhashi
 
Plugin-based software design with Ruby and RubyGems
bySadayuki Furuhashi
 
Embuk internals
bySadayuki Furuhashi
 
Embulk, an open-source plugin-based parallel bulk data loader
bySadayuki Furuhashi
 
Understanding Presto - Presto meetup @ Tokyo #1
bySadayuki Furuhashi
 
Prestogres internals
bySadayuki Furuhashi
 
Presto+MySQLで分散SQL
bySadayuki Furuhashi
 
Presto - Hadoop Conference Japan 2014
bySadayuki Furuhashi
 
Fluentd - Set Up Once, Collect More
bySadayuki Furuhashi
 
Prestogres, ODBC & JDBC connectivity for Presto
bySadayuki Furuhashi
 
What's new in v11 - Fluentd Casual Talks #3 #fluentdcasual
bySadayuki Furuhashi
 

Recently uploaded

PDF
2026 Safety Roadmap: Systems, Skills, and Scale for EHS Leaders
byTECH EHS Solution
 
PDF
Project Tracking in Software Project Management
bySahanaBarveen1
 
PDF
Accelerating Data Validation with QuerySurge AI
byRTTS
 
PDF
SLIDE PROPARTNER SUMMIT 2025 - ITALY.pdf
bygiuo
 
PPTX
HackYourBrain_JFokusConference_03022026.pptx
bySimonedeGijt
 
PDF
Why I Volunteer at FOSDEM and You Should Too
byImma Valls Bernaus
 
PPTX
Applicius Technologies: Full Service Overview for Branding, SEO, Web & App De...
byappliciustech
 
PDF
Physiotherapist App Development for Modern Rehabilitation
byV3CUBE TECHNOLABS
 
PDF
Windows Server 2025 Administration Fundamentals 4ED
byraynasolomon136
 
PDF
Figma systems development services
bydavidjohansen1597
 
PDF
2026-01-26 - AWS - Architecting Multi-Agent Developer Workflows.pdf
byhlibco
 
PDF
LogiNext Media Kit & Company Overview 2025
bypriyajoshi2929
 
PDF
Pet Care Service App Development Company
byV3CUBE TECHNOLABS
 
PDF
LogiNext Media Kit & Company Overview 2025
bysanikaroy72
 
PPTX
Best eCommerce Development Agency in USA | The Dataflux
byKey Medsolutions Inc
 
PDF
Introducing MySQL HeatWave Migration Assistant
byAlexander Hitrov
 
PDF
AI-native development: from Prompt to Platform
byMaxim Salnikov
 
PDF
LogiNext Media Kit & Company Overview 2025
bynidhisharmaq7184
 
PDF
LogiNext Media Kit & Company Overview 2025
bysmritipatil055
 
PDF
Monitoring your MySQL Server's Health Trends
byIgor Donchovski
 
2026 Safety Roadmap: Systems, Skills, and Scale for EHS Leaders
byTECH EHS Solution
 
Project Tracking in Software Project Management
bySahanaBarveen1
 
Accelerating Data Validation with QuerySurge AI
byRTTS
 
SLIDE PROPARTNER SUMMIT 2025 - ITALY.pdf
bygiuo
 
HackYourBrain_JFokusConference_03022026.pptx
bySimonedeGijt
 
Why I Volunteer at FOSDEM and You Should Too
byImma Valls Bernaus
 
Applicius Technologies: Full Service Overview for Branding, SEO, Web & App De...
byappliciustech
 
Physiotherapist App Development for Modern Rehabilitation
byV3CUBE TECHNOLABS
 
Windows Server 2025 Administration Fundamentals 4ED
byraynasolomon136
 
Figma systems development services
bydavidjohansen1597
 
2026-01-26 - AWS - Architecting Multi-Agent Developer Workflows.pdf
byhlibco
 
LogiNext Media Kit & Company Overview 2025
bypriyajoshi2929
 
Pet Care Service App Development Company
byV3CUBE TECHNOLABS
 
LogiNext Media Kit & Company Overview 2025
bysanikaroy72
 
Best eCommerce Development Agency in USA | The Dataflux
byKey Medsolutions Inc
 
Introducing MySQL HeatWave Migration Assistant
byAlexander Hitrov
 
AI-native development: from Prompt to Platform
byMaxim Salnikov
 
LogiNext Media Kit & Company Overview 2025
bynidhisharmaq7184
 
LogiNext Media Kit & Company Overview 2025
bysmritipatil055
 
Monitoring your MySQL Server's Health Trends
byIgor Donchovski
 

Performance Optimization Techniques of MessagePack-Ruby - RubyKaigi 2019

  • 1.
    Performance Optimization Techniquesof MessagePack-Ruby Sadayuki Furuhashi RubyKaigi 2019 #MyMessagePack Tweet your msgpack usage with
  • 2.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 About me A founder of Treasure Data, Inc. Located in Silicon Valley, USA. OSS Hacker. Github: @frsyuki
  • 3.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 Basics of MessagePack
  • 4.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 What’s MessagePack? { “compact”: true, “schema”: 0 } 82 A7 compact C3 A6 schema 00 JSON MessagePack It’s like JSON, but fast and small. 7-byte string2-element map true 6-byte string 0 27 bytes 18 bytes (34% smaller)
  • 5.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 What’s MessagePack? It’s like JSON, but fast and small. > Self-descriptive, Schema-on-Read semantics > Everyone knows > De facto standard data format > Human-readable > Self-descriptive, Schema-on-Read semantics > Everyone who uses JSON can use > Drop-in improvement of JSON > Machine-readable JSON MessagePack
  • 6.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 timestamp 32 timestamp 64 timestamp 96 Language Agnostic Type System MessagePack
 Type System MessagePack
 Format String Timestamp Language Types fixstr str 8 str 16 str 32 String Timestamp (JSON compatible)(Ruby, Swift, Java, Go, …) Convert
  • 7.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Supported by Super Skilled Engineers All Over The World Ruby /msgpack
  • 8.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack-Ruby Major Committers
  • 9.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Real World MessagePack We import over 2,000,000 records/sec and
 store 30PB of data in MessagePack format. 15 trillion rows processed every day. Sada Furuhashi, Arm Treasure Data Sada Furuhashi, Initial Creator of Fluentd MessagePack is an essential component of Fluentd to achieve high performance and flexibility at the same time.
  • 10.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Adoption of MessagePack Today Mobile Apps Microprocessors Automotive Telematics Sensors Cloud Infrastructure Middleware Machine Learning Games Analytical Databases MessgePack Zero-overhead, heterogeneous data exchange
  • 11.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 MessagePack-Ruby implementation
  • 12.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 4-element array
  • 13.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 4-element array 01 Integer 1
  • 14.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 4-element array 01 Integer 1 02 Integer 2
  • 15.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 4-element array 01 Integer 1 02 Integer 2 81 1-element map
  • 16.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 01 02 81 A1 1-byte string 4-element array Integer 1 Integer 2 1-element map
  • 17.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 01 02 81 ‘k’ ‘k’ A1 1-byte string 4-element array Integer 1 Integer 2 1-element map
  • 18.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 01 02 81 ‘k’ ‘k’ A1 1-byte string ‘v’ ‘v’ A1 1-byte string 4-element array Integer 1 Integer 2 1-element map
  • 19.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Packer Object Buffer [ 1, 2, {“k” => “v”}, nil ] 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 nil 4-element array Integer 1 Integer 2 1-element map ‘k’ 1-byte string ‘v’ 1-byte string
  • 20.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ □, □, □, □ ] 4-element array
  • 21.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ □, □, □, □ ] 1 Integer 1
  • 22.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, □, □, □ ] 1 complete object
  • 23.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, □, □, □ ] 2 Integer 2
  • 24.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] 2 complete object
  • 25.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { □ => □ }1-element map
  • 26.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { □ => □ } “k”1-byte string
  • 27.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { “k” => □ } “k” complete object
  • 28.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { “k” => □ } “v” 1-byte string
  • 29.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [ 1, 2, □, □ ] { “k” => “v” } “v” complete object
  • 30.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [1, 2,{“k”=>“v”}, □ ] { “k” => “v” } complete object
  • 31.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [1, 2,{“k”=>“v”}, □ ] nil nil
  • 32.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [1, 2,{“k”=>“v”},nil] nil complete object
  • 33.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Unpacker Buffer Stack 94 01 02 81 ‘k’ A1 ‘v’ A1 C0 [1, 2,{“k”=>“v”},nil] Complete object
  • 34.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 Optimization
  • 35.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Buffer MessagePack::Buffer next mapped_string mem { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] Object 83 A5 ‘c’ ‘o’ ’u’ ’n’ ’t' 01 A4 ‘p’ ’a’ ’g’ ’e’ A4 ‘b’ ‘o’ ’d’ ’y’
  • 36.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack::Buffer MessagePack::Buffer next mapped_string mem next mapped_string mem { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] Object 83 A5 ‘c’ ‘o’ ’u’ ’n’ ’t' 01 A4 ‘p’ ’a’ ’g’ ’e’ A4 ‘b’ ‘o’ ’d’ ’y’ 91 A9 ‘L’ ‘O’ ’N’ ’G’ ’ ’ ’T‘ ’E’ ‘X’ ’T’ Add more buffer chunks instead of realloc()
  • 37.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Zero-copy write optimization MessagePack::Buffer next mapped_string mem next mapped_string mem next mapped_string mem { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] Object 83 A5 ‘c’ ‘o’ ’u’ ’n’ ’t' 01 A4 ‘p’ ’a’ ’g’ ’e’ A4 ‘b’ ‘o’ ’d’ ’y’ 91 A9 “LONG TEXT” rb_str_dup() Fast copy-on-write
  • 38.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Zero-copy read optimization MessagePack::Buffer next mapped_string mem Object { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] rb_str_substr() Fast copy-on-write x83 A5 c o u n t t x01 xA4 p a g e xA4 b o d y x91 A9 L O N G
 T E X T ” “ ※ if SHARABLE_SUBSTRING_P() returns true x83 A5 c o u n t t x01 xA4 p a g e xA4 b o d y x91 A9 L O N G
 T E X T ” Source String “ rb_str_dup() Fast copy-on-write
  • 39.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Reserved memory pool MessagePack::Buffer next mapped_string mem Global memory pool next mapped_string mem 4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB 83 A5 ‘c’ ‘o’ ’u’ ’n’ ’t' 01 A4 ‘p’ ’a’ ’g’ ’e’ A4 ‘b’ ‘o’ ’d’ ’y’ 91 A9
  • 40.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Reserved memory pool MessagePack::Buffer next mapped_string mem Global memory pool next mapped_string mem 4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB
  • 41.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 Benchmark & further optimization
  • 42.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Benchmark Data Sets DB 100_000.times.map {|i| r.rand(i+1) } [{ "id" : "gfmg-6ppu", "name" : "Mortgage Complaints", "averageRating" : 0, "createdAt" : 1433953219, "moderationStatus" : true, “numberOfComments" : 0, "description" : "Each week we …”, … Blogs Integers [ 0,0,0,1,3,0,0, …,
 42991,26906,18655,7015
 ] { “results”: [ { “attachments”:[], “body”:
 ”Dear Friends and Colleagues,nn
 I always look forward to the… https://www.justice.gov/api/v1/blog_entries.json http://data.consumerfinance.gov/api/views.json Benchmark code available at https://gist.github.com/frsyuki/9777c4adba2b5c957695b64f17b64ba1
  • 43.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack vs JSON DB Integers Blogs 0 600 1200 1800 2400 3000 Serialization time MessagePack JSON (Oj) JSON (JSON) 100% DB Integers Blogs 0 600 1200 1800 2400 3000 Deserialization time 100%
  • 44.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack without optimization DB Integers Blogs 0 30 60 90 120 150 Serialization time 100% DB Integers Blogs 0 30 60 90 120 150 Deserialization time 100% Default No memory pool No zero-copy read
  • 45.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Zero-copy read optimization MessagePack::Buffer next mapped_string mem Object { “count” => 1, “page” => 0, “body” => [ “LONG TEXT” ] ] x83 A5 c o u n t t x01 xA4 p a g e xA4 b o d y x91 L O N G
 T E X T ” “x83 A5 c o u n t t x01 xA4 p a g e xA4 b o d y x91 L O N G
 T E X T ” Source String “ rb_str_dup() Fast copy-on-write rb_str_substr() Fast copy-on-write ※ if SHARABLE_SUBSTRING_P() returns true
  • 46.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Copy-on-write substring SHARABLE_SUBSTRING_P() returns true only when substring shares the last 0 termination with the original string s = "a" * 1_000_100 10_000.times do s.slice(100, 1_000_000) end #=> 0.002 sec s = "a" * 1_000_100 10_000.times do s.slice(100, 1_000_000 - 1) end #=> 2.7 sec Not including the last character disables Copy-on-Write
  • 47.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Copy-on-write substring SHARABLE_SUBSTRING_P() returns true only when substring shares the last 0 termination with the original string OR Ruby is compiled with SHARABLE_MIDDLE_SUBSTRING flag (not enabled by default) s = "a" * 1_000_100 10_000.times do s.slice(100, 1_000_000) end #=> 0.002 sec s = "a" * 1_000_100 10_000.times do s.slice(100, 1_000_000 - 1) end #=> 2.7 sec 0.002 sec Using ruby binary compiled with SHARABLE_MIDDLE_SUBSTRING
  • 48.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Deserialization with copy-on-write substring DB Integers Blogs 0 30 60 90 120 150 Serialization time 100% DB Integers Blogs 0 30 60 90 120 150 Deserialization time Default With SHARABLE_MIDDLE_SUBSTRING
  • 49.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 What is deserialization bottleneck? 0 14,000,000 28,000,000 42,000,000 56,000,000 70,000,000 Deserialization objects/sec DB Integers Blogs Boolean (2^62)-1 (2^62) Immediate On-Heap On-Heap
  • 50.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 What is deserialization bottleneck? 0 14,000,000 28,000,000 42,000,000 56,000,000 70,000,000 Deserialization objects/sec DB Integers Blogs Boolean (2^62)-1 (2^62) Immediate Immediate Immediate On-Heap => Object allocation is slow On-Heap On-Heap
  • 51.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Reusing Hash key objects data: [ { “id”: “s6ew-h6mp”, “name”: “Consumer Complaints”, … }, { “id”: “nsyy-je5y”, “name”: “Beta Consumers”, … } { “id”: “wkue-ycpk”, “name”: “Survey”, … } ] Same key repeats. Keys of Hash are always frozen. => We can reuse objects!
  • 52.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 How to reuse Hash key objects? Use fstring. Ruby uses it to reuse same objects for immutable strings (but C API is not available…yet): p “a”.object_id == “a”.object_id #=> false # frozen_string_literal: true p “a”.object_id == “a”.object_id #=> true p (-“a”).object_id == (-“a”).object_id #=> true
  • 53.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Reusing Hash key objects using fstring DB Integers Blogs 0 30 60 90 120 150 Serialization time 100% DB Integers Blogs 0 25 50 75 100 125 150 Deserialization time Default Hash key fstring using
 hacked Ruby binary
  • 54.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 Link Time Optimization DB Integers Blogs 0 30 60 90 120 150 Serialization time 100% DB Integers Blogs 0 25 50 75 100 125 150 Deserialization time All optimization And -flto=thin
  • 55.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 Reading other MessagePack implementations
  • 56.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 msgpack-java: TypeProfile bypassing interface MessageBuffer { public void putInt(int index, int value); } class MessageBufferBE implements MessageBuffer { public void putInt(int index, int value) { byteBuffer.putInt(index, value); } } This code has significant overhead:
  • 57.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 msgpack-java: TypeProfile bypassing interface MessageBuffer { public void putInt(int index, int value); } class MessageBufferBE implements MessageBuffer { public void putInt(int index, int value) { byteBuffer.putInt(index, value); } } This code has significant overhead: Dynamic method lookup
 (slow even after JIT) Dynamic method lookup
  • 58.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 msgpack-java: TypeProfile bypassing interface MessageBuffer { public void putInt(int index, int value) { v = Integer.reverseBytes(v); unsafe.putInt(base, address + index, v); } public static MessageBuffer newInstance() { // … } Much faster code: No override. JVM JIT inlines them.
  • 59.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 msgpack-java: TypeProfile bypassing interface MessageBuffer { public void putInt(int index, int value) { v = Integer.reverseBytes(v); unsafe.putInt(base, address + index, v); } public static MessageBuffer newInstance() { // … } Much faster code: JVM intrinsics
 (1-to-1 mapping to CPU instruction,
 no function call) JVM intrinsics Load inherited class lazily.
 No override on little-endian machine. No override. JVM JIT inlines them.
  • 60.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack-CSharp: Lookup cache of optimized Ser/De class User { public int Age { get; set; } public string FirstName { get; set; } public string LastName { get; set; } } [ 31, “Sadayuki”, “Furuhashi” ] Mapping between class and semi-structured data Java: Jackson databind, JAXB, … C#: MessagePack-CSharp, System.Runtime.Serialization, … Swift: Codable, SwiftMsgPack, … …
  • 61.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack-CSharp: Native LZ4 integration
  • 62.
    1 1 01 0 1 0 1 1 0 1 1 0 0 1 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 0 0 1 1 1 0 1 0 0 0 0 1 1 0 1 1 1 0 0 1 1 0 1 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 MessagePack-CSharp: Native LZ4 integration
  • 63.
    It's like JSON.
 butfast and small. Sadayuki Furuhashi #MyMessagePack Tweet your msgpack usage with