In Fluentd new version v0.14, ServerEngine is integrated. I'll explain about the advantages of ServerEngine.

1. ServerEngine Integration & Windows support
Ritta Narita
Fluentd Meetup 2016

2. About me.
Ritta Narita
joined Treasure Data after internship in 2015
2
@naritta
@narittan

3. Fluentd v0.14
ServerEngine is integrated

4. What’s ServerEngine?
A framework to implement robust multiprocess servers like Unicorn
worker
worker
worker
super
visor
Server
Dynamic reconfig
live restart
Heartbeat via pipe
auto restart
multi
process/thread

5. how to use ServerEngine
worker
super
visor
Server
module MyServer
def before_run
@sock = TCPServer.new(
config[:bind], config[:port])
end
attr_reader :sock
end
module MyWorker
def run
c = server.sock.accept
c.write "Awesome work!"
end
def stop
c.close
end
end
1. write server module and worker module

6. how to use ServerEngine
worker
super
visor
Server
se = ServerEngine.create(MyServer, MyWorker, {
daemonize: true,
log: 'myserver.log',
pid_path: 'myserver.pid',
worker_type: 'process',
workers: 4,
bind: '0.0.0.0',
port: 9071,
})
se.run
2. write configuration and run

7. module MyWorker
def spawn(process_manager)
process_manager.spawn(env, config[:spawn_cmd])
end
end
ServerEngine.create(MyServer, MyWorker, {
worker_type:‘spawn’,
spawn_cmd: cmd
})
ServerEngine.create(MyServer, MyWorker, {
worker_type:‘thread'
})
workerServer
workerServer
workerServer
spawn
thread
processServerEngine.create(MyServer, MyWorker, {
worker_type:‘process'
})
worker type

8. What’s the advantages for integrating serverengine into fluentd?

9. serverengine advantages
Auto restart
live restart
socket manager
signal handler
log rotation

10. Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages

11. if worker died unexpectedly…
worker
worker
super
visor
Server
worker

12. if worker died unexpectedly…
worker
worker
worker
super
visor
Server

13. if worker died unexpectedly…
worker
worker
super
visor
Server
It’s detected by heartbeat and server launch worker again.
worker

16. Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages

17. worker
super
visor
Server
tcp tcp
send SIGHUP
live restart

18. super
visor
Server worker
tcp
kill the worker, leaving server’s condition
live restart

19. worker
super
visor
Server worker
tcp
start new worker and read config again
live restart

20. worker
super
visor
Server worker
tcp
tcp
share the TCP with living zero downtime
live restart

23. Signal and RPC server
api/config.reloadlive restart/config reload HUP
/api/processes
.interruptWorkers
INT
/api/processes
.killWorkers
/api/plugins.
flushBuffers
system config reload USR2
TERM
USR1flush buffers
shutdown

24. worker
super
visor
Server
you can reload system config without restart
system config reload
send USR2
check mtime and previous load time,
if there is new one, reload!
serverengine
config cash
new config

25. Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages

26. Socket Manager
worker
Socket
Manager
client
Socket
Manager
server
Server
26
①socket listen request②find or create listening socket
③share the socket FD(unix)/
duplicated socket info(windows)
UDS(unix)/
TCP(windows)
④get socket and accept in worker

27. how to use Socket Manager
worker
Socket
Manager
client
Socket
Manager
server
Server
27
#generate path for UDS/TCP for server connection
socket_manager_path =
ServerEngine::SocketManager::Server.generate_path
#open socket manager server
socket_manager_server =
ServerEngine::SocketManager::Server.open(socket_manager_path)
In server

28. worker
Socket
Manager
client
Socket
Manager
server
Server
28
#get socket manager client from socket manager path
socket_manager =
ServerEngine::SocketManager::Client.new(socket_manager_path)
#get listening socket from socket manager
lsock = socket_manager.listen_tcp(bind, port)
In worker
how to use Socket Manager

29. What’s the merit of Socket Manager?
worker
worker
worker
Socket
Manager
client
Socket
Manager
client
Socket
Manager
client
Socket
Manager
server
super
visor
Server
29
we can use multicore easily in future

30. easy to use multicore power fully
30
with SocketManagerwith in_multiprocess plugin
<source>
type multiprocess
<process>
cmdline -c /etc/td-agent/td-agent-child1.conf
</process>
<process>
cmdline -c /etc/td-agent/td-agent-child2.conf
</process>
</source>
#/etc/td-agent/td-agent-child1.conf
<source>
type forward
port 24224
</source>
#/etc/td-agent/td-agent-child2.conf
<source>
type forward
port 24225
</source>
<source>
type forward
port 24224
</source>
setting when using 2 core

33. Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages

34. Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
send USR1

35. Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
USR1

36. Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
QUIT
USR1
send QUIT

37. Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
QUIT

38. Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
USR1
QUIT
run!

39. Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
QUIT

40. Queue based signal handling
When user sent USR1 and QUIT…
signal handler thread
user
QUIT run!

41. Auto restart
live restart
socket manager
signal handler
log rotation
serverengine advantages

42. support new features
(These are already ported to ruby core.)
・multiprocess aware log rotation
・reopening of log file
・'trace' level, which is lower level than ‘debug'
ServerEngine.create(MyServer, MyWorker, {
log: 'myserver.log',
log_level: 'debug',
log_rotate_age: 5,
log_rotate_size: 1*1024*1024,
})

43. summary
windows
・signal → rpc handler
・no socket FD and UDS
・fork based → spawn based
・live & auto restart
・multicore with socket manager
・sophisticated signal handler
・easier log rotation

44. Thank you!

45. appendix

46. unix→fork based damon
windows→spawn based damon
damonize
fluentd
server
engine
①set windows_daemon_cmdline
in configuration
②run windows_daemon_cmdline
when damonize
③windows_daemon_cmdline
runs daemon.rb
④run_server with
servermodule and workermodule