Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"fireap" - fast task runner on consul

1,922 views

Published on

"Fireap" is a consul-based rapid propagative task runner for large systems.
https://github.com/key-amb/fireap

Published in: Technology
  • Be the first to comment

  • Be the first to like this

"fireap" - fast task runner on consul

  1. 1. “fireap” - fast task runner on Consul @key_amb In 渋谷.rb[:20160420] #shibuyarb
  2. 2. Who am I ? ❏ @key_amb ❏ Software Engineer ❏ Serverside, Web ❏ https://github.com/key-amb ❏ https://rubygems.org/profiles/key-amb
  3. 3. fireap
  4. 4. Overview of “fireap” ❏ https://github.com/key-amb/fireap ❏ Propagative task runner on Consul Cluster ❏ O(log N) … N = node number ❏ Consul key features used in “fireap”: ❏ Kv, Event, Watch ❏ And powered by diplomat, gem for Consul API ❏ Blog: ❏ Consul クラスタ上で動作する S3 非依存の pull 型デプロイツール "fireap" を作った
  5. 5. Related Products ❏ https://github.com/fujiwara/stretcher ❏ https://github.com/sorah/mamiya These above enable O(1) deployment. But both need storage with big I/O capacity (like S3) => difficult to use in private DC. Then, I came up with “fireap”.
  6. 6. How fast is “fireap”? ❏ Benchmark: ❏ rsync for 100 of t2.micro nodes on AWS EC2. GNU Parallel fireap real 0m46.906s 0m18.992s user 0m40.407s 0m00.527s sys 0m04.241s 0m00.046s
  7. 7. How “fireap” works?
  8. 8. Overview
  9. 9. config/fireap.toml [task.apps.bench] exec_commands = [ "rsync -az --delete <%= @remote.address %>:<%= ENV ['HOME'] %>/<%= @app %> <%= ENV['HOME'] %>/", ]
  10. 10. Run “fireap fire” on Publisher node // on publisher node % fireap fire -a bench -v 0.1.0-demo
  11. 11. Subscriber node exec commands // on subscriber node % tailf fireap.log : 2016-04-20 17:08:08 +0900 [INFO] EXEC rsync -az --delete 172.31.0.1:/home/bob/bench /home/bob/ :
  12. 12. Propagates through the whole cluster
  13. 13. Changes in v0.1 ~ v0.4 ❏ Enable to filter target nodes by Consul “service”, “tag” and “node name”. ❏ Add config validation before every command execution. ❏ Add `task` subcommand to show tasks in config file.
  14. 14. Things to be done ❏ Write more tests and do more benchmarks ❏ Plan to monitor Consul ❏ seems preferable especially in bigger systems ❏ Datastore alternatives (maybe) ❏ may be needed if Consul Kv goes slow in bigger systems ❏ Clean up codes
  15. 15. How to get started ❏ Documentation is Available ❏ https://github.com/key-amb/fireap/wiki ❏ If it doesn’t work, let me know as GitHub issue. ❏ I’m happy if you try “fireap”.
  16. 16. Thank you!
  17. 17. ❏ Serf/Consulで管理を自動化! ~実践的な 手法を紹介~:特集|gihyo.jp ❏ Consulと自作OSSを活用した100台規模の Webサービス運用 // Speaker Deck Appendix

×