Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Sadayuki Furuhashi
Kumofs
MessagePack
Project
#msgpack @frsyuki
Distributed key-value store
and e
About me:
Sadayuki Furuhashi (@frsyki)
Diskless network system
VIVER
kumofs
Distributed key-value store
MessagePack
Cross-...
About me:
Sadayuki Furuhashi
Diskless network system
VIVER
kumofs
Distributed key-value store
MessagePack
Cross-language c...
kumofs
kumo = 雲 = cloud
What s kumofs?
• Distributed key-value store
• Get, Set, Delete and CAS (Compare-And-Swap)
• Linear scalability as servers...
http://itpro.nikkeibp.co.jp/article/COLUMN/20090617/331953/
memcachedを超える成果も、Interopで若手技術者がクラウドを支える技術を競う
http://gihyo.jp/dev/serial/01/alpha-geek/0041
“小飼弾のアルファギークに いたい♥”
Shows and manages hundreds of photos smoothly
> gives lots of accesses on metadata servers
http://yapcasia2009.ficia.com/pl...
Why kumofs?
• Need to manage heavy random reads and writes
• Need low latency for user-experience
• Need elastic scalabili...
Archtecture
of
kumofs
Manager
Gateway
kumo-manager:
Watches kumo-servers to
detect/detach crashed severs
Archtecture of kumofs
kumo-server:
Stor...
Application
Server
Manager
Gateway
Manager
Duplicated
(HA)
Server
Server
Server
Server
Server
Application
Gateway
Applicat...
hash(key1)
Server A
Server B
Server C
Server D
Consistent Hashing
Server A
Server B
Server C
Server D
coordinated by
Server B
Server A
Server B
Server C
Server D
coordinated by
Server B
Server C
Server D
Server A
Server A
Server B
Server C
Server D
set
Replicate
Replication mechanism
Server A
Server B
Server C
Server D
get
High-availability
Server A
Server B
Server C
Server D
get
High-availability
Server A
Server B
Server C
Server D
get
High-availability
Server A
Server B
Server C
Server D
coordinated by
Server B
Server C
Server D
Server A
Adding nodes dynamically
Server A
Server B
Server C
Server D
coordinated by
Server B
Server C
Server A
Server E
Copies data
Adding nodes dynamically
Server A
Server B
Server C
Server D
Server E
Data are moving...
Adding nodes dynamically
Server A
Server B
Server C
Server D
Server E
set
get
Adding nodes dynamically
Server A
Server B
Server C
Server D
Server E
set
get
Adding nodes dynamically
get
set
Consistent Hashing for
Reading
Consistent Hashing for
Writing
Dynamic rebalacing algorithm of kumofs
(double-hash-...
Application
Gateway
Gateway
Application
Server Server Server
memcached protocol
MessagePack-RPC
・Hides cluster configuratio...
MessagePack-RPC
Server Server Server
Tools
Administrator
Manager
Gets cluster configuration
Asynchronous RPC library
Tools
Written in Ruby
> automates operation tasks
> easy to implement
> easy to customize
kumoctl
kumostat
kumotop
Message...
Tools
Written in Ruby
> automates operation tasks
> easy to implement
> easy to customize
kumoctl
kumostat
kumotop
Message...
What s MessagePack?
• Binary-based object serialization format
• fast and small
• Cross-language
• Java, C++, Ruby, Python...
MessagePack?
{“msgpack”:“json”, “hello”:“world”}
24% compact
MessagePack
26 bytes
JSON
34 bytes
It's like JSON, but very f...
It's like JSON, but very fast and small.
MessagePack?
Faster!
MessagePack for Java
require 'msgpack' # gem install “msgpack"
msg = [1,2,3].to_msgpack #=> "x93x01x02x03"
MessagePack.unpack(msg) #=> [1,2,3]
...
require 'msgpack' # gem install “msgpack"
u = MessagePack::Unpacker.new($stdin)
u.each do |obj|
puts obj.to_json
end
Messa...
MessagePack?
Annotation + dynamic code generation in Java
@MessagePackMessage
public static class MyClass {
public String ...
MessagePack
• C++
> Sadayuki Furuhashi
• Ruby
> Sadayuki Furuhashi
• Java
> Muga Nishizawa
• Python
> INADA Naoki
• Haskel...
MessagePack-RPC?
Rails app
C++ serverJava app
MessagePack-RPC
Cross-language messaging library
What s MessagePack-RPC?
• Cross-language messaging library
• Java, C++, Ruby, Python, Erlang, Haskell, ...
• Simple protoc...
What s MessagePack-RPC?
• Asynchronous RPC with parallel pipelining
• Concurrent communication over multiple servers
• Con...
Protocol
MessagePack-RPC protocol
[0, msgid, method, params]
[1, msgid, error, result]
[2, method, params]
Request
Respons...
Pipelining
Client Server Pipelining
Sends one request after
another before receiving
responses to speed-up.
Examples:
・HTT...
Client Server
Pipelining
heavy task
light task
Examples:
・HTTP/1.1 pipelining
・Thrift
・SOAP
Heavy task may keep
light task...
Client Server
Parallel pipelining
Example:
・MessagePack-RPC
MessagePack-RPC doesn t
require responses to be sorted.
> Retu...
Future
Client Server1 Server2
call_async(method1)
call_async(method2)
future1
future2
Event-driven I/O
Server-side event-driven I/O
Dispatcher
Server
Client
Client
Loop
multi-threaded event-driven I/O
(C++, J...
Event-driven I/O
Client
Session Loop Server
Client
ServerSession Loop
Client-side event-driven I/O
Event-driven I/O
shared
event loop
Client
Client
Server
Server
Loop
Session
Session
Client-side event-driven I/O
communica...
Event-driven I/O
Session Pool
Server
Server
pools these
connectionsLoop
Session
Session
connection
Client-side event-drive...
Dispatcher
Event-driven I/O
shared
event loop
Client
Server
Server
Loop
Session
Session
Client-side event-driven I/O
Client
shared
event looptimer, signal handler,
other protocol, etc...
Dispatcher
Event-driven I/O
Client
Server
Loop
Session
Clie...
MessagePack-RPC
• C++ - Sadayuki Furuhashi
• Erlang - UENISHI Kota
• Haskell - Hideyuki Tanaka
• Java - Muga Nishizawa
• P...
Users
• Ameba Now
• Microblog service; 18 million posts per month
• Apache Solr + MessagePack for realtime search
• Sedue
...
http://sedue.sc/
Sedue Cloud Search: 簡単に使える高性能検索エンジン・レコメンドエンジンクラウドサービス
Conclusion
• Kumofs - Distributed key-value store
> Optimized for low latency
always zero-hop
> No SPOF
> Scales-out witho...
Conclusion
• MessagePack
> Cross-language object serialization library
> Supports optional fields
• MessagePack-RPC
> Cross...
http://msgpack.org/
http://kumofs.sourceforge.net/
NoSQL afternoon in Japan Kumofs & MessagePack
NoSQL afternoon in Japan Kumofs & MessagePack
Upcoming SlideShare
Loading in …5
×

NoSQL afternoon in Japan Kumofs & MessagePack

5,215 views

Published on

  • Be the first to comment

NoSQL afternoon in Japan Kumofs & MessagePack

  1. 1. Sadayuki Furuhashi Kumofs MessagePack Project #msgpack @frsyuki Distributed key-value store and e
  2. 2. About me: Sadayuki Furuhashi (@frsyki) Diskless network system VIVER kumofs Distributed key-value store MessagePack Cross-language communication library University of Tsukuba Master’s cource in Computer Science supported by “Mitoh” project run by IPA, Information-technology Promotion Agancy, Japan Projects:
  3. 3. About me: Sadayuki Furuhashi Diskless network system VIVER kumofs Distributed key-value store MessagePack Cross-language communication library University of Tsukuba Master’s cource in Computer Science supported by “Mitoh” project run by IPA, Information-technology Promotion Agancy, Japan Projects:
  4. 4. kumofs kumo = 雲 = cloud
  5. 5. What s kumofs? • Distributed key-value store • Get, Set, Delete and CAS (Compare-And-Swap) • Linear scalability as servers added • both read and write throughput scales-out • dynamic rebalancing without any impacts on applications • Optimized for read-latency • always zero-hop • No SPOF
  6. 6. http://itpro.nikkeibp.co.jp/article/COLUMN/20090617/331953/ memcachedを超える成果も、Interopで若手技術者がクラウドを支える技術を競う
  7. 7. http://gihyo.jp/dev/serial/01/alpha-geek/0041 “小飼弾のアルファギークに いたい♥”
  8. 8. Shows and manages hundreds of photos smoothly > gives lots of accesses on metadata servers http://yapcasia2009.ficia.com/pl/album/7BF0E66A-A35F-11DE-966B-860A873069EA
  9. 9. Why kumofs? • Need to manage heavy random reads and writes • Need low latency for user-experience • Need elastic scalability • builds small cluster at the first • enlarges it when number of users increases • “I won’t wakeup at midnight!” • Systems that have SPOF rushes administrators to recover crashed servers even if it’s at midnight.
  10. 10. Archtecture of kumofs
  11. 11. Manager Gateway kumo-manager: Watches kumo-servers to detect/detach crashed severs Archtecture of kumofs kumo-server: Stores and replicates data kumo-gateway: Relays requests from applications to kumo-servers Server
  12. 12. Application Server Manager Gateway Manager Duplicated (HA) Server Server Server Server Server Application Gateway Application Gateway Replication Archtecture of kumofs Tokyo CabinetTokyo Cabinet
  13. 13. hash(key1) Server A Server B Server C Server D Consistent Hashing
  14. 14. Server A Server B Server C Server D coordinated by Server B
  15. 15. Server A Server B Server C Server D coordinated by Server B Server C Server D Server A
  16. 16. Server A Server B Server C Server D set Replicate Replication mechanism
  17. 17. Server A Server B Server C Server D get High-availability
  18. 18. Server A Server B Server C Server D get High-availability
  19. 19. Server A Server B Server C Server D get High-availability
  20. 20. Server A Server B Server C Server D coordinated by Server B Server C Server D Server A Adding nodes dynamically
  21. 21. Server A Server B Server C Server D coordinated by Server B Server C Server A Server E Copies data Adding nodes dynamically
  22. 22. Server A Server B Server C Server D Server E Data are moving... Adding nodes dynamically
  23. 23. Server A Server B Server C Server D Server E set get Adding nodes dynamically
  24. 24. Server A Server B Server C Server D Server E set get Adding nodes dynamically
  25. 25. get set Consistent Hashing for Reading Consistent Hashing for Writing Dynamic rebalacing algorithm of kumofs (double-hash-space)
  26. 26. Application Gateway Gateway Application Server Server Server memcached protocol MessagePack-RPC ・Hides cluster configuration from applications. memcached server on localhost localhost:11211 Asynchronous RPC library
  27. 27. MessagePack-RPC Server Server Server Tools Administrator Manager Gets cluster configuration Asynchronous RPC library
  28. 28. Tools Written in Ruby > automates operation tasks > easy to implement > easy to customize kumoctl kumostat kumotop MessagePack-RPC Asynchronous RPC library Cross-language RPC
  29. 29. Tools Written in Ruby > automates operation tasks > easy to implement > easy to customize kumoctl kumostat kumotop MessagePack-RPC Asynchronous RPC library Cross-language RPC MessagePack
  30. 30. What s MessagePack? • Binary-based object serialization format • fast and small • Cross-language • Java, C++, Ruby, Python, Erlang, Haskell, ... • Dynamic typing like JSON,Avro, etc. (Unlike Thrift) • type-system is compatible with JSON • Streaming deserialization
  31. 31. MessagePack? {“msgpack”:“json”, “hello”:“world”} 24% compact MessagePack 26 bytes JSON 34 bytes It's like JSON, but very fast and small.
  32. 32. It's like JSON, but very fast and small. MessagePack? Faster! MessagePack for Java
  33. 33. require 'msgpack' # gem install “msgpack" msg = [1,2,3].to_msgpack #=> "x93x01x02x03" MessagePack.unpack(msg) #=> [1,2,3] MessagePack? It's like JSON, but very fast and small.
  34. 34. require 'msgpack' # gem install “msgpack" u = MessagePack::Unpacker.new($stdin) u.each do |obj| puts obj.to_json end MessagePack? It's like JSON, but very fast and small.
  35. 35. MessagePack? Annotation + dynamic code generation in Java @MessagePackMessage public static class MyClass { public String str; public double num; @MessagePackOptional public int flag = 0; }
  36. 36. MessagePack • C++ > Sadayuki Furuhashi • Ruby > Sadayuki Furuhashi • Java > Muga Nishizawa • Python > INADA Naoki • Haskell > Hideyuki Tanaka • Lua > Nobuyuki Kubota • Perl > tokuhirom, gfx, ... • Erlang > UENISHI Kota • Node.JS > Peter Griess • JavaScript > uupaa • PHP > advect • D > repeatedly Authors
  37. 37. MessagePack-RPC? Rails app C++ serverJava app MessagePack-RPC Cross-language messaging library
  38. 38. What s MessagePack-RPC? • Cross-language messaging library • Java, C++, Ruby, Python, Erlang, Haskell, ... • Simple protocol • 3 messages: Request, Response and Notify
  39. 39. What s MessagePack-RPC? • Asynchronous RPC with parallel pipelining • Concurrent communication over multiple servers • Connection pooling • omits overheads of reconnecting • Concept of Future • hides complicated event-driven I/O • Concept of Session • hides transport connections • Dynamic typing
  40. 40. Protocol MessagePack-RPC protocol [0, msgid, method, params] [1, msgid, error, result] [2, method, params] Request Response Notify Message ID →Parallel Pipelining
  41. 41. Pipelining Client Server Pipelining Sends one request after another before receiving responses to speed-up. Examples: ・HTTP/1.1 pipelining ・Thrift ・SOAP
  42. 42. Client Server Pipelining heavy task light task Examples: ・HTTP/1.1 pipelining ・Thrift ・SOAP Heavy task may keep light tasks waiting > makes parallelism worse must be sorted
  43. 43. Client Server Parallel pipelining Example: ・MessagePack-RPC MessagePack-RPC doesn t require responses to be sorted. > Returns light tasks faster Faster heavy task light task
  44. 44. Future Client Server1 Server2 call_async(method1) call_async(method2) future1 future2
  45. 45. Event-driven I/O Server-side event-driven I/O Dispatcher Server Client Client Loop multi-threaded event-driven I/O (C++, Java) Servers deals with many clients efficiently.
  46. 46. Event-driven I/O Client Session Loop Server Client ServerSession Loop Client-side event-driven I/O
  47. 47. Event-driven I/O shared event loop Client Client Server Server Loop Session Session Client-side event-driven I/O communications concurrently
  48. 48. Event-driven I/O Session Pool Server Server pools these connectionsLoop Session Session connection Client-side event-driven I/O
  49. 49. Dispatcher Event-driven I/O shared event loop Client Server Server Loop Session Session Client-side event-driven I/O Client
  50. 50. shared event looptimer, signal handler, other protocol, etc... Dispatcher Event-driven I/O Client Server Loop Session Client-side event-driven I/O Client
  51. 51. MessagePack-RPC • C++ - Sadayuki Furuhashi • Erlang - UENISHI Kota • Haskell - Hideyuki Tanaka • Java - Muga Nishizawa • PHP - h0x10 • Python - INADA Naoki • Ruby - Sadayuki Furuhashi Authors
  52. 52. Users • Ameba Now • Microblog service; 18 million posts per month • Apache Solr + MessagePack for realtime search • Sedue • Commercial distributed search engine
  53. 53. http://sedue.sc/ Sedue Cloud Search: 簡単に使える高性能検索エンジン・レコメンドエンジンクラウドサービス
  54. 54. Conclusion • Kumofs - Distributed key-value store > Optimized for low latency always zero-hop > No SPOF > Scales-out without any impacts on applications Gateway hides cluster configuration Consistency control algorithm
  55. 55. Conclusion • MessagePack > Cross-language object serialization library > Supports optional fields • MessagePack-RPC > Cross-language messaging library > Server/client-side event-driven I/O > Asynchronous RPC with parallel pipelining > Concept of Future and Session > Connection pooling
  56. 56. http://msgpack.org/ http://kumofs.sourceforge.net/

×