Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ruby and Distributed Storage Systems

9,113 views

Published on

RubyKaigi 2017

Published in: Software
  • Be the first to comment

Ruby and Distributed Storage Systems

  1. 1. Ruby for Distributed Storage Systems RubyKaigi 2017: Sep 20, 2017 Satoshi Tagomori (@tagomoris) Treasure Data, Inc.
  2. 2. Satoshi Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, Woothee, ... Treasure Data, Inc.
  3. 3. -45°
  4. 4. Ruby for Distributed Storage Systems
  5. 5. Ruby for and Distributed Storage Systems
  6. 6. Ruby and Performance • Web? or Not? • Disk & Network I/O • "I/O spends most of time on servers"... is it real? • Storages are getting faster and faster
 (SSD, NVMe, ...) • Networks too (10GbE, fast network in Cloud, ...)
  7. 7. Storage Systems • Disk I/O • Network I/O • Serialization / Deserialization (json, msgpack, ...) • read/write data from/to disk • parse/generate HTTP request/response • Indexing (update, search) • Timer • Threads + Locks
  8. 8. Distributed Storage Systems • Data replication • Checksum • Asynchronous network I/O • Quorum • More Threads + Locks
  9. 9. Replication w/ 3 replicas • Create 3 replica of data, including local storage accept request to write data write the data into local storage (1) receive responses to replicate data (3) send response to write data input input input input input input input input input input input input input input input send requests to replicate data
  10. 10. Replication in Quoram Systems: In Action • Create 2 replica of data at least (max 3), including local storage accept request to write data, and write it locally (1) send response to write data input input input input input ? ? input input input input input input create 2 threads to send requests to replicate data input input receive a successful
 response to
 replicate data (2) ? ? ? Discard a thread for another node
  11. 11. Bigdam
  12. 12. Bigdam • Brand new data ingestion pipeline
 in Treasure Data • Huge data • Extraordinary large number of connections / requests • Many edge endpoints on the planet
  13. 13. Bigdam:
 Edge locations on the earth + the Central location
  14. 14. Bigdam components @narittan @tagomoris @nalsh @k0kubun @komamitsu_tw
  15. 15. Bigdam-pool • OSS (in future... not yet) • Distributed key-value storage • for buffer pool in Bigdam • to build S3 free data ingestion pipeline
  16. 16. Bigdam-pool: Small Buffers • Small buffers (MBs) • Write: append support for many small chunks (KBs) • Read: secondary index to query/read many buffers at once • Short buffer lifetime: minutes (create - append - read - delete) • Buffers store ids of chunks (for deduplication) buffer buffer buffer buffer chunk chunk chunk chunk chunk chunk chunk chunk chunk chunk account_id, database, table
  17. 17. Bigdam-pool: Replication • Replication in a cluster • without maintaining replica factor • Clients send requests to all living nodes
  18. 18. Bigdam-pool: Buffer Transferring over Clusters Edge location Central location Over Internet Using HTTPS or HTTP/2 Buffer committed (size or timeout)
  19. 19. written in Java
  20. 20. Designing Bigdam • Architecture Design - split a system to 5 microservices • consistency, availability • performance (how to scale it out?) • deployment, cost • API Design • Mocking • Interface Test • Integration Test
  21. 21. Mocking Bigdam using Ruby • Mocking • build mock servers of all components • implement all public APIs between components • Find/add missing parameters required • Prepare to develop components in parallel • Mocked using Ruby, Sinatra • public APIs - it's just a Webapp • fast and easy to do :D
  22. 22. Interface/Integration Tests of Bigdam using Ruby • Interface tests: • verify all public APIs are implemented correctly • Integration tests • verify the whole pipeline can import data correctly • Written in Ruby, test-unit • less code to serialize/deserialize various req/res • readable test cases • fast and easy to do :D
  23. 23. And,
  24. 24. Bigdam-pool-ruby • Port bigdam-pool from Java to Ruby • Experiment to know Ruby is good enough or not
  25. 25. Bigdam-pool-ruby • Perfectly compatible with Java implementation • Public API, Private API • Data formats on local storage, of secondary index • Under development • only supports stand alone mode, for now
  26. 26. Studies: Serialization / Deserialization • All network API call requires it • parsing HTTP request • parsing request content body (json/msgpack) • building response content body (json/msgpack) • building HTTP response • Should be parallelized on CPU cores
  27. 27. Studies: Asynchronous Network I/O • EventMachine? Cool.io? Celluloid::IO? • 🤔 • I want to use only async network I/O at a time!
 (not disk, not timer) • Event driven I/O library? • Thread pools + callback? • or any idea?
  28. 28. Threading / Timers • ExecutorService in Java is very useful... • Fixed / non-fixed thread pools with Queue • (and some other executor models) • Runner of Runnable tasks • "Runnable task" is just like a lambda w/o args • To be implemented as Gem? • Queue and SizedQueue look useful for it
  29. 29. Queue#peek
 Get the head object w/o removing it from queue https://github.com/ruby/ruby/pull/1698
  30. 30. Queue#peek
 Get the head object w/o removing it from queue https://github.com/ruby/ruby/pull/1698
  31. 31. MonitorMixin#mon_locked? and #mon_owned? • Mutex#owned? exists https://github.com/ruby/ruby/pull/1699
  32. 32. Resource Control Make sure to release resources: try-with-resources in Java
  33. 33. Resource Control Make sure to release resources: try-with-resources in Java
  34. 34. Typing? • Defining APIs • Rubyists (including me) MAY be using:
 [string, integer, boolean, string, ...] • Rubyists (including me) MAY be using:
 {"time": unix_time (but sometimes float)}
 • Explicit definition makes nothing bad in designing APIs • Json schema or something others may help us...
  35. 35. Typing: in logging and others https://bugs.ruby-lang.org/issues/13913
  36. 36. Process Built-in Application Servers • Distributed Storage Systems: • Background worker threads • Timers • Communication workers to other nodes • Various async operation workers • Public API request handlers • Private API request handlers (inter-nodes) • Startup/Shutdown hooks • It's NOT just web application, but handles HTTP requests
  37. 37. https://github.com/tagomoris/bigdam-pool-ruby
  38. 38. https://github.com/tagomoris/bigdam-pool-ruby NOT YET
  39. 39. "Why Do You Want to Write Such Code in Ruby?"
  40. 40. "Why Do You Want to Write Such Code in Ruby?" "Because I WANT TO DO IT!"
  41. 41. "Why Do You Want to Write Such Code in Ruby?" "Because I WANT TO DO IT!" "... And we already have Fluentd :P" Thank you. @tagomoris

×