Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
ServerEngine 
Server programming framework for Ruby 
Masahiro Nakagawa 
Sep 19, 2014 
RubyKaigi 2014
Who are you? 
> Masahiro Nakagawa 
> github/twitter: @repeatedly 
> Treasure Data, Inc. 
> Senior Software Engineer 
> Flu...
0. Background + Intro
Ruby is not only for web apps! 
> System programs 
• Chef - server configuration management tool 
• Serverspec - spec fram...
Problem: server programming is hard 
Server programs should support: 
> multi-process or multi-thread 
> robust error hand...
Solution: Use a framework 
ServerEngine A framework for server programming in Ruby 
github.com/fluent/serverengine
What’s ServerEngine? 
With ServerEngine, we can write multi-process 
server programs, like Unicorn, easily. 
What we need ...
Hello world in ServerEngine 
require 
'serverengine' 
! 
module 
MyWorker 
def 
run 
until 
@stop 
logger.info 
"Hello 
wo...
How ServerEngine works? 
1. Robust process management (supervisor) 
2. Multi-process and multi-threading 
3. Dynamic confi...
1. Robust process management 
Heartbeat via pipe 
& auto-restart 
Supervisor Server 
Dynamic reconfiguration 
& live resta...
Each role overview 
Supervisor Server Worker 
• Manage Server 
• heartbeat 
• attach / detach 
• restart 
! 
• Disable by ...
2. Multi-process & multi-threading 
require 
'serverengine' 
! 
module 
MyWorker 
def 
run 
until 
@stop 
logger.info 
"Aw...
2. Multi-process & multi-threading 
embedded process thread 
fork 
Worker 
• default mode 
• use main thread 
Server 
Work...
3. Dynamic reconfiguration 
module 
MyWorker 
def 
initialize 
reload 
end 
! 
def 
run 
# 
… 
end 
! 
def 
reload 
@messa...
4. Log rotation 
> Support useful features 
> multi-process aware log rotation 
> support “trace” level 
> Port to Ruby co...
5. Signal handling 
> Queue based signal handling 
> serialize signal processing 
> signal handling is separated from 
sig...
5. Signal handling - register 1 
SignalThread 
INT { process_int } 
Register Signal 
INT
5. Signal handling - register 2 
SignalThread 
USR1 { process_usr1 } 
INT { process_int } 
Register Signal 
USR1
5. Signal handling - register 3 
SignalThread 
QUIT { process_quit } 
TERM { process_term } 
USR1 { process_usr1 } 
INT { ...
5. Signal handling - process 1 
SignalThread 
QUIT { process_quit } 
TERM { process_term } 
USR1 { process_usr1 } 
{ proce...
5. Signal handling - process 2 
SignalThread 
QUIT { process_quit } 
TERM { process_term } 
USR1 { process_usr1 } 
INT { p...
5. Signal handling - process 3 
SignalThread 
QUIT { process_quit } 
TERM { process_term } 
USR1 { process_usr1 } 
INT { p...
6. Live Restart 
> Minimize server restart downtime 
> via INT signal 
> enable_detach and 
supervisor parameters must be ...
6. Live Restart - flow 1 
1. start a server 
Supervisor Server WWWooorrkrkkeeerrr
6. Live Restart - flow 2 
2. receive SIGINT and 
wait for shutdown of the server 
Supervisor Server WWWooorrkrkkeeerrr
6. Live Restart - flow 3 
3. start new server if 
the server doesn’t exit in server_detach_wait 
Supervisor Server 
Worker...
7. “sigdump” 
> SIGQUIT of JavaVM for Ruby 
> https://github.com/frsyuki/sigdump 
> dump backtrace of running threads 
and...
7. sigdump example 
% kill -CONT pid 
% cat /tmp/sigdump-66276.log 
Sigdump at 2014-09-18 18:44:43 +0900 process 66276 (se...
Use-case1: Sneakers 
> A fast background processing 
framework for Ruby 
> use ServerEngine and RabbitMQ 
> jondot.github....
Use-case2: Fluentd v1 
> Data collector for unified logging layer 
> http://www.fluentd.org/ 
> Improve core features 
> L...
Fluentd v1 - Multi-process 
Worker 
Supervisor 
Worker Worker 
<Worker> 
input tail 
output forward 
</worker> 
<Worker> 
...
Fluentd v1 - Zero downtime restart 
> SocketManager shares the resource 
32 
Supervisor 
TCP 
1. Listen TCP socket
Fluentd v1 - Zero downtime restart 
> SocketManager shares the resource 
33 
Worker 
Supervisor 
heartbeat 
TCP 
TCP 
1. L...
Fluentd v1 - Zero downtime restart 
> SocketManager shares the resource 
34 
Worker 
Supervisor 
1. Listen TCP socket 
2. ...
Demo (if I have a time…)
Cloud service for the entire data pipeline 
Check: www.treasuredata.com
Upcoming SlideShare
Loading in …5
×

RubyKaigi 2014: ServerEngine

8,331 views

Published on

http://rubykaigi.org/2014/presentation/S-MasahiroNakagawa
Movie in Japanese: http://www.ustream.tv/recorded/52839701

Published in: Technology
  • Be the first to comment

RubyKaigi 2014: ServerEngine

  1. 1. ServerEngine Server programming framework for Ruby Masahiro Nakagawa Sep 19, 2014 RubyKaigi 2014
  2. 2. Who are you? > Masahiro Nakagawa > github/twitter: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > I love OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC- D and Python (only RPC) > The organizer of Presto Source Code Reading > etc…
  3. 3. 0. Background + Intro
  4. 4. Ruby is not only for web apps! > System programs • Chef - server configuration management tool • Serverspec - spec framework for servers •Apache Deltacloud - IaaS API abstraction library > Network servers • Starling - distributed message queue server •Unicorn - multiprocess HTTP server > Log servers • Fluentd - extensible data collection tool
  5. 5. Problem: server programming is hard Server programs should support: > multi-process or multi-thread > robust error handling > log rotation > signal handling > dynamic reconfiguration > metrics collection > etc...
  6. 6. Solution: Use a framework ServerEngine A framework for server programming in Ruby github.com/fluent/serverengine
  7. 7. What’s ServerEngine? With ServerEngine, we can write multi-process server programs, like Unicorn, easily. What we need to write is a 2 modules: Worker module and Server module. Everything else, including daemonize, logging, dynamic reconfiguration, multi-processing is done by ServerEngine.
  8. 8. Hello world in ServerEngine require 'serverengine' ! module MyWorker def run until @stop logger.info "Hello world!" sleep 1 end end ! def stop @stop = true end end ! se Worker Server = ServerEngine.create(nil, MyWorker, { log: 'myserver.log', pid_path: 'myserver.pid', }) se.run Config
  9. 9. How ServerEngine works? 1. Robust process management (supervisor) 2. Multi-process and multi-threading 3. Dynamic configuration reloading 4. Log rotation 5. Signal handling 6. Live restart 7. “sigdump”
  10. 10. 1. Robust process management Heartbeat via pipe & auto-restart Supervisor Server Dynamic reconfiguration & live restart support Multi-process Worker Worker or Multi-thread Worker
  11. 11. Each role overview Supervisor Server Worker • Manage Server • heartbeat • attach / detach • restart ! • Disable by default ! • No extension point • Manage Worker • monitor • restart ! • Some execution types • Embedded • Thread • Process ! • Extension point • before_run • after_run • after_restart • Execution unit • implement run method ! • Extension point • stop • reload • before_fork • after_start
  12. 12. 2. Multi-process & multi-threading require 'serverengine' ! module MyWorker def run until @stop logger.info "Awesome work!" sleep 1 end end ! def stop @stop = true end end ! se = ServerEngine.create(nil, MyWorker, { daemonize: true, log: 'myserver.log', pid_path: 'myserver.pid', worker_type: 'process', workers: 4, }) se.run > 3 server types > embedded > process > thread - thread example se = ServerEngine.create(nil, MyWorker,{ daemonize: true, log: 'myserver.log', pid_path: 'myserver.pid', worker_type: 'thread', workers: 8, }) se.run
  13. 13. 2. Multi-process & multi-threading embedded process thread fork Worker • default mode • use main thread Server Worker • use fork for parallel execution • not work on Windows • use thread for parallel execution • for JRuby and Rubinius Server WWorokrekrer Server Thread.new WWorokrekrer Worker
  14. 14. 3. Dynamic reconfiguration module MyWorker def initialize reload end ! def run # … end ! def reload @message = config["message"] || "default" @sleep = config["sleep"] || 1 end end ! se = ServerEngine.create(nil, MyWorker) do YAML.load_file("config.yml").merge({ :daemonize => true, :worker_type => 'process', }) end se.run > Overwrite method > reload in worker > reload_config in server > Send USR2 signal
  15. 15. 4. Log rotation > Support useful features > multi-process aware log rotation > support “trace” level > Port to Ruby core > https://github.com/ruby/ruby/pull/428 se = ServerEngine.create(MyServer, MyWorker, { log: 'myserver.log', log_level: 'debug', log_rotate_age: 5, log_rotate_size: 1 * 1024 * 1024, }) se.run
  16. 16. 5. Signal handling > Queue based signal handling > serialize signal processing > signal handling is separated from signal handler to avoid lock issues SignalThread.new do |st| st.trap(:TERM) { server.stop(true) } st.trap(:QUIT) { server.stop(false) } st.trap(:USR1) { server.restart(true) } st.trap(:HUP) { server.restart(false) } st.trap(:USR2) { server.reload } # ... end
  17. 17. 5. Signal handling - register 1 SignalThread INT { process_int } Register Signal INT
  18. 18. 5. Signal handling - register 2 SignalThread USR1 { process_usr1 } INT { process_int } Register Signal USR1
  19. 19. 5. Signal handling - register 3 SignalThread QUIT { process_quit } TERM { process_term } USR1 { process_usr1 } INT { process_int } Register Signal XXX
  20. 20. 5. Signal handling - process 1 SignalThread QUIT { process_quit } TERM { process_term } USR1 { process_usr1 } { process_int } USR1 Send Signal USR1 INT Monitor queue
  21. 21. 5. Signal handling - process 2 SignalThread QUIT { process_quit } TERM { process_term } USR1 { process_usr1 } INT { process_int } QUIT USR1 Send Signal QUIT
  22. 22. 5. Signal handling - process 3 SignalThread QUIT { process_quit } TERM { process_term } USR1 { process_usr1 } INT { process_int } QUIT Send Signal XXX
  23. 23. 6. Live Restart > Minimize server restart downtime > via INT signal > enable_detach and supervisor parameters must be true > Network server can’t use live restart > “Address already in use” occurred > use “process” worker and USR1 instead • restart workers, not server
  24. 24. 6. Live Restart - flow 1 1. start a server Supervisor Server WWWooorrkrkkeeerrr
  25. 25. 6. Live Restart - flow 2 2. receive SIGINT and wait for shutdown of the server Supervisor Server WWWooorrkrkkeeerrr
  26. 26. 6. Live Restart - flow 3 3. start new server if the server doesn’t exit in server_detach_wait Supervisor Server Worker Server WWWooorrkrkkeeerrr
  27. 27. 7. “sigdump” > SIGQUIT of JavaVM for Ruby > https://github.com/frsyuki/sigdump > dump backtrace of running threads and allocated object list > for debugging, slow code, dead-lock, … > ServerEngine traps SIGCONT for sigdump > Trapping signal is configurable using “SIGDUMP_SIGNAL” environment variable
  28. 28. 7. sigdump example % kill -CONT pid % cat /tmp/sigdump-66276.log Sigdump at 2014-09-18 18:44:43 +0900 process 66276 (se.rb) Thread #<Thread:0x007fdc130cb7e0> status=sleep priority=0 se.rb:7:in `sleep' se.rb:7:in `run' /Users/repeatedly/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/serverengine-1.5.9/ lib/serverengine/worker.rb:67:in `main' /Users/repeatedly/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/serverengine-1.5.9/ lib/serverengine/embedded_server.rb:24:in `run' /Users/repeatedly/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/serverengine-1.5.9/ lib/serverengine/server.rb:85:in `main' /Users/repeatedly/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/serverengine-1.5.9/ lib/serverengine/daemon.rb:101:in `main' … Thread #<ServerEngine::SignalThread:0x007fdc13314038> status=run priority=0 … Built-in objects: 33,017: TOTAL 16,845: T_STRING … All objects: 8,939: String 1,459: Array … String 210,569 bytes Array 4 elements Hash 14 pairs
  29. 29. Use-case1: Sneakers > A fast background processing framework for Ruby > use ServerEngine and RabbitMQ > jondot.github.io/sneakers/ Server WWWooorrkrkkeeerrr Task Sneakers RabbitMQ
  30. 30. Use-case2: Fluentd v1 > Data collector for unified logging layer > http://www.fluentd.org/ > Improve core features > Logging > Signal handling > New features based on ServerEngine > Multi-process support > Zero downtime restart > etc…
  31. 31. Fluentd v1 - Multi-process Worker Supervisor Worker Worker <Worker> input tail output forward </worker> <Worker> input forward output webhdfs </worker> <Worker> input foo output bar </worker> Separate stream pipelines in one instance!
  32. 32. Fluentd v1 - Zero downtime restart > SocketManager shares the resource 32 Supervisor TCP 1. Listen TCP socket
  33. 33. Fluentd v1 - Zero downtime restart > SocketManager shares the resource 33 Worker Supervisor heartbeat TCP TCP 1. Listen TCP socket 2. Pass its socket to worker
  34. 34. Fluentd v1 - Zero downtime restart > SocketManager shares the resource 34 Worker Supervisor 1. Listen TCP socket 2. Pass its socket to worker 3. Do same action at worker restarting with keeping TCP socket Worker TCP TCP heartbeat
  35. 35. Demo (if I have a time…)
  36. 36. Cloud service for the entire data pipeline Check: www.treasuredata.com

×