Downloaded 23 times
![Configuration by JSON
{
"ctl" : {
"sockpath" : "/var/run/neoagent_ctl.sock",
"logpath" : "/var/run/neoagent_ctl.log",
},
"environments" :
[
{
"name" : "env1",
"sockpath" : "/var/run/neoagent.sock",
"target_server" : "127.0.0.1:11212",
"backup_server" : "127.0.0.1:11213",
(abbreviated)
},
{
"name" : "env2",
"sockpath" : "/var/run/neoagent2.sock",
"target_server" : "127.0.0.1:11214",
"backup_server" : "127.0.0.1:11215",
(abbreviated)
},
]
}
13年9月20日金曜日](https://image.slidesharecdn.com/yapc2013-130919223718-phpapp01/85/memcached-proxy-server-development-and-operation-47-320.jpg)
![Configuration by JSON
{
"ctl" : {
"sockpath" : "/var/run/neoagent_ctl.sock",
"logpath" : "/var/run/neoagent_ctl.log",
},
"environments" :
[
{
"name" : "env1",
"sockpath" : "/var/run/neoagent.sock",
"target_server" : "127.0.0.1:11212",
"backup_server" : "127.0.0.1:11213",
(abbreviated)
},
{
"name" : "env2",
"sockpath" : "/var/run/neoagent2.sock",
"target_server" : "127.0.0.1:11214",
"backup_server" : "127.0.0.1:11215",
(abbreviated)
},
]
}
13年9月20日金曜日](https://image.slidesharecdn.com/yapc2013-130919223718-phpapp01/85/memcached-proxy-server-development-and-operation-48-320.jpg)
![Configuration by JSON
{
"ctl" : {
"sockpath" : "/var/run/neoagent_ctl.sock",
"logpath" : "/var/run/neoagent_ctl.log",
},
"environments" :
[
{
"name" : "env1",
"sockpath" : "/var/run/neoagent.sock",
"target_server" : "127.0.0.1:11212",
"backup_server" : "127.0.0.1:11213",
(abbreviated)
},
{
"name" : "env2",
"sockpath" : "/var/run/neoagent2.sock",
"target_server" : "127.0.0.1:11214",
"backup_server" : "127.0.0.1:11215",
(abbreviated)
},
]
}
13年9月20日金曜日](https://image.slidesharecdn.com/yapc2013-130919223718-phpapp01/85/memcached-proxy-server-development-and-operation-50-320.jpg)
![Configuration by JSON
{
"ctl" : {
"sockpath" : "/var/run/neoagent_ctl.sock",
"logpath" : "/var/run/neoagent_ctl.log",
},
"environments" :
[
{
"name" : "env1",
"sockpath" : "/var/run/neoagent.sock",
"target_server" : "127.0.0.1:11212",
"backup_server" : "127.0.0.1:11213",
(abbreviated)
},
{
"name" : "env2",
"sockpath" : "/var/run/neoagent2.sock",
"target_server" : "127.0.0.1:11214",
"backup_server" : "127.0.0.1:11215",
(abbreviated)
},
]
}
13年9月20日金曜日](https://image.slidesharecdn.com/yapc2013-130919223718-phpapp01/85/memcached-proxy-server-development-and-operation-51-320.jpg)
Uploaded byTatsuhiko Kubo
memcached proxy server development and operation
memcached proxy server development and operaiton
More Related Content
Similar to memcached proxy server development and operation
memcached proxy server development and operation
- 1. memcached proxy server developmentand operation Tatsuhiko Kubo bokko@pixiv.com 13年9月20日金曜日
- 2. About me ✦ TatsuhikoKubo(bokko@pixiv.com) ✦ @cubicdaiya(twitter, github) ✦ Senior Software Engineer@pixiv Inc. 13年9月20日金曜日
- 3. Recent Work ✦ mruby_nginx_module ✦Embedded mruby into Nginx ✦ ngx_small_light ✦ Dynamic Image Transformation for Nginx ✦ Contributions/pull requests to many other projects on github 13年9月20日金曜日
- 4. Recent Work WEB+DB PRESSVol.72 □detailed nginx flexible configuration and brilliant scalability □collaborators @harukasan @semind 13年9月20日金曜日
- 5. memcached proxy server developmentand operation 13年9月20日金曜日
- 6. memcached proxy server developmentand operation ∼at pixiv∼ 13年9月20日金曜日
- 7.
- 8.
- 9.
- 10. About datastore & cachingstrategy at pixiv 13年9月20日金曜日
- 11. Previously(2 years ago...) ・・・・・・ 13年9月20日金曜日
- 12. Previously(2 years ago...) Therewas a memcached on each application server. ・・・ ・・・ 13年9月20日金曜日
- 13. Previously(2 years ago...) ・・・・・・ 13年9月20日金曜日
- 14. Previously(2 years ago...) APconnected through UNIX domain socket. ・・・ ・・・ 13年9月20日金曜日
- 15. Previously(2 years ago...) Connectsthrough TCP socket, too. (For data synchronization) 13年9月20日金曜日
- 16. rough-and-ready patch forthis 13年9月20日金曜日
- 17. Problems ✦ data synchronizationis straining ✦ TCP connection overhead ✦ processing time is directly proprtional the number of APs ✦ data restoration is painful ✦ memcached is not persistent ✦ e.g, when AP server goes down. 13年9月20日金曜日
- 18. Migration to KyotoTycoon ✦KyotoTycoon supports ✦ data persistency. ✦ data expiration ✦ memcached ASCII protocol. ✦ We could migrate without modifying application! 13年9月20日金曜日
- 19.
- 20.
- 21. KyotoTycoon overwhelmed ✦ KyotoTycoon(usingmemcached protocol plugin) compared with memcached, ✦ Lower performance 13年9月20日金曜日
- 22. KyotoTycoon overwhelmed ✦ KyotoTycoon(usingmemcached protocol plugin) compared with memcached, ✦ Lower performance Even so, persistency is some good! 13年9月20日金曜日
- 23. KyotoTycoon overwhelmed ✦ KyotoTycoon(usingmemcached protocol plugin) compared with memcached, ✦ Lower performance Even so, persistency is some good! We wanted it at that time! 13年9月20日金曜日
- 24. Other Problems ✦ Weneeded ✦ seamless fail-over mechanism ✦ convenient monitoring mechanism ✦ slow query log(like MySQL) ✦ statistics for whole queries 13年9月20日金曜日
- 25. dealing with theseproblems ✦ Scale out(For example, add servers) ✦ This was difficult for us at least at that time... ✦ Scale up(For example, buy expensive servers) ✦ This was more difficult for us. ✦ Use proxy server! ✦ Actually, I developed one in C. ✦ Twemproxy was not released yet. 13年9月20日金曜日
- 26. ✦ Scale out(Forexample, add servers) ✦ This was difficult for us at least at that time... ✦ Scale up(For example, buy expensive servers) ✦ This was more difficult for us. ✦ Use proxy server! ✦ Actually, I developed one in C. ✦ Twemproxy was not released yet. dealing with these problems 13年9月20日金曜日
- 27.
- 28. Currently ✦ Applications connectto neoagent by UNIX domain socket ✦ reduced consumption of local-ports ✦ neoagent keeps persistent TCP connections to KyotoTycoon ✦ bypass overhead of TCP connection 13年9月20日金曜日
- 29. neoagent ∼A Yet AnotherMemcached Protocol Proxy Server∼ 13年9月20日金曜日
- 30. neoagent ✦ event-driven &multi-threaded server ✦ proxies a subset of memcached protocol ✦ get, set, add, delete, incr, decr, quit ✦ supports following features ✦ connection-pooling ✦ graceful-restart & online-update ✦ convenient monitoring ✦ etc... 13年9月20日金曜日
- 31. neoagent architecture ✦ core ✦written in C (C99 style) ✦ tools(CLI) ✦ written in Python ✦ dependencies ✦ libev, json-c, pthread(core) ✦ SCons, Sphinx(tools) 13年9月20日金曜日
- 32. neoagent architecture ✦ Single-masterand multiple workers ✦ Configuration by JSON ✦ Multi-threaded ✦ Event-driven 13年9月20日金曜日
- 33. Single Master andmultiple workers 13年9月20日金曜日
- 34. neoagent process 1 neogent process 2 neoagent process3 KyotoTycoon KyotoTycoon KyotoTycoon proxies proxies proxies Apache (mod_php) send request Previously 13年9月20日金曜日
- 35. Previously ✦ each neoagent-processwas independent ✦ neoagent-process was daemonized by daemontools 13年9月20日金曜日
- 36. neoagent process 1 KyotoTycoon proxies neoagentprocesses spawn gradually 13年9月20日金曜日
- 37. neoagent process 1 KyotoTycoon proxies neoagent process2 KyotoTycoon proxies neoagent processes spawn gradually 13年9月20日金曜日
- 38. neoagent process 1 KyotoTycoon proxies neoagent process2 KyotoTycoon proxies neoagent process 3 KyotoTycoon proxies neoagent processes spawn gradually 13年9月20日金曜日
- 39. neoagent process 1 KyotoTycoon proxies neoagent process2 KyotoTycoon proxies neoagent process 3 KyotoTycoon proxies ・ ・ ・ neoagent processes spawn gradually 13年9月20日金曜日
- 40. neoagent process 1 KyotoTycoon proxies neoagent process2 KyotoTycoon proxies neoagent process 3 KyotoTycoon proxies neoagent process n KyotoTycoon proxies ・ ・ ・ neoagent processes spawn gradually 13年9月20日金曜日
- 41.
- 42.
- 43. neoagent worker process 1 neogent worker process 2 neoagent worker process3 KyotoTycoon KyotoTycoon KyotoTycoon proxies proxies proxies Currently neoagent master process fork 13年9月20日金曜日
- 44. neoagent worker process 1 neogent worker process 2 neoagent worker process3 KyotoTycoon KyotoTycoon KyotoTycoon proxies proxies proxies Apache (mod_php) send request Currently neoagent master process 13年9月20日金曜日
- 45. single-master and multipleworkers ✦ Master is responsible for ✦ controlling worker processes ✦ Workers are responsible for ✦ processing user’s requests ✦ Supervisor may control only Master ✦ Now we use Monit instead of daemontools 13年9月20日金曜日
- 46.
- 47. Configuration by JSON { "ctl": { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log", }, "environments" : [ { "name" : "env1", "sockpath" : "/var/run/neoagent.sock", "target_server" : "127.0.0.1:11212", "backup_server" : "127.0.0.1:11213", (abbreviated) }, { "name" : "env2", "sockpath" : "/var/run/neoagent2.sock", "target_server" : "127.0.0.1:11214", "backup_server" : "127.0.0.1:11215", (abbreviated) }, ] } 13年9月20日金曜日
- 48. Configuration by JSON { "ctl": { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log", }, "environments" : [ { "name" : "env1", "sockpath" : "/var/run/neoagent.sock", "target_server" : "127.0.0.1:11212", "backup_server" : "127.0.0.1:11213", (abbreviated) }, { "name" : "env2", "sockpath" : "/var/run/neoagent2.sock", "target_server" : "127.0.0.1:11214", "backup_server" : "127.0.0.1:11215", (abbreviated) }, ] } 13年9月20日金曜日
- 49. “ctl” block sockpath socketpath for controlling neoagent logpath log path for master process "ctl" : { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log", }, This block is for master 13年9月20日金曜日
- 50. Configuration by JSON { "ctl": { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log", }, "environments" : [ { "name" : "env1", "sockpath" : "/var/run/neoagent.sock", "target_server" : "127.0.0.1:11212", "backup_server" : "127.0.0.1:11213", (abbreviated) }, { "name" : "env2", "sockpath" : "/var/run/neoagent2.sock", "target_server" : "127.0.0.1:11214", "backup_server" : "127.0.0.1:11215", (abbreviated) }, ] } 13年9月20日金曜日
- 51. Configuration by JSON { "ctl": { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log", }, "environments" : [ { "name" : "env1", "sockpath" : "/var/run/neoagent.sock", "target_server" : "127.0.0.1:11212", "backup_server" : "127.0.0.1:11213", (abbreviated) }, { "name" : "env2", "sockpath" : "/var/run/neoagent2.sock", "target_server" : "127.0.0.1:11214", "backup_server" : "127.0.0.1:11215", (abbreviated) }, ] } 13年9月20日金曜日
- 52. “environments” block ✦ Inneoagent, ✦ environment is the configuration for workers. ✦ Each entry in “environments” is the configuration for each worker. ✦ Master manages workers for each of their of environments. 13年9月20日金曜日
- 53. “environments” block name environmentname sockpath socket path for communicating neoagent’s worker target_server primary memcached server backup_server secondary memcached server conn_max max connections(backlog size) connpool_max connection pool size etc... document is here http://cubicdaiya.github.io/neoagent/ 13年9月20日金曜日
- 54.
- 55. neoagent’s threads ✦ Master-process ✦sigwait-thread ✦ ctl-thread ✦ Worker-process ✦ sigwait-thread ✦ event-threads ✦ support-thread 13年9月20日金曜日
- 56. Master’s threads ✦ sigwait-thread ✦waiting for a signal from ctl- thread ✦ ctl-thread ✦ receiving a instruction from neoctl through ctl-socket 13年9月20日金曜日
- 57. Worker’s threads ✦ sigwait-thread ✦waiting for a signal from ctl- thread ✦ event-threads ✦ processing client requests ✦ support-thread ✦ health-checking & statictics 13年9月20日金曜日
- 58.
- 59. Event-threads’ architecture ✦ event-driven ✦non-blocking I/O ✦ multi-threaded ✦ request-queuing & thread-pooling 13年9月20日金曜日
- 60.
- 61. multithreaded with request-queuing &thread-pooling event-thread event-thread event-thread request1 request2 request3 empty ・ ・ ・ empty dequeue dequeue dequeue event-thread(accept) request-queue enqueue processing requests 13年9月20日金曜日
- 62.
- 63. Usage ✦ Building neoagent ✦Monitoring neoagent ✦ Controlling neoagent 13年9月20日金曜日
- 64.
- 65.
- 66.
- 67. SCons ✦ Simple &flexible & programable build tool ✦ Auto anaylysis of dependencies ✦ Build-configuration is written with Python ✦ Suitable for small or medium scale projects 13年9月20日金曜日
- 68.
- 69. Monitoring ✦ health-checking workerwith Nagios & NRPE ✦ monitoring neostat & Munin 13年9月20日金曜日
- 70. neostat $ neostat -s/var/run/neoagent_st.sock -o { "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated) } 13年9月20日金曜日
- 71. neostat -> MuninGraph -> current connections $ neostat -s /var/run/neoagent_st.sock -o { "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated) } 13年9月20日金曜日
- 72. neostat -> MuninGraph -> neostat outputs current status of neoagent. (For example, current connection numbers) current connections $ neostat -s /var/run/neoagent_st.sock -o { "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated) } 13年9月20日金曜日
- 73. neostat internal ✦ neoagentworkers have a suppport-thread. ✦ support-thread returns JSON as against request of neostat. 13年9月20日金曜日
- 74. slowlog for neoagent ■configuration ■ output "slow_query_sec" : 0.01, # milli-seconds "slow_query_log_path" : "/var/log/neoagent_slowlog.log", "slow_query_log_format" : "json", "slow_query_log_access_mask" : "0666", { ..., "na_to_ts": 0.000006, "na_from_ts": 0.000003, "na_to_client": 0.014320, "querytxt": "get key1" } { ..., "na_to_ts": 0.000012, "na_from_ts": 0.000004, "na_to_client": 0.011587, "querytxt": "get key2" } { ..., "na_to_ts": 0.000009, "na_from_ts": 0.000003, "na_to_client": 0.017788, "querytxt": "get key3" } { ..., "na_to_ts": 0.000009, "na_from_ts": 0.000009, "na_to_client": 0.036082, "querytxt": "get key4" } { ..., "na_to_ts": 0.000008, "na_from_ts": 0.000008, "na_to_client": 0.010506, "querytxt": "get key5" } { ..., "na_to_ts": 0.000014, "na_from_ts": 0.000003, "na_to_client": 0.029464, "querytxt": "get key6" } { ..., "na_to_ts": 0.000007, "na_from_ts": 0.000004, "na_to_client": 0.012440, "querytxt": "get key7" } { ..., "na_to_ts": 0.000007, "na_from_ts": 0.000003, "na_to_client": 0.043834, "querytxt": "get key8" } { ..., "na_to_ts": 0.000007, "na_from_ts": 0.050442, "na_to_client": 0.000015, "querytxt": "get key9" } 13年9月20日金曜日
- 75. slowlog for neoagent na_to_tstaken time for sending request to target server na_from_ts taken time for receiving response from target server na_to_client taken time for sending response to client querytxt request body { ..., "na_to_ts": 0.000006, "na_from_ts": 0.000003, "na_to_client": 0.014320, "querytxt": "get key1" } { ..., "na_to_ts": 0.000012, "na_from_ts": 0.000004, "na_to_client": 0.011587, "querytxt": "get key2" } { ..., "na_to_ts": 0.000009, "na_from_ts": 0.000003, "na_to_client": 0.017788, "querytxt": "get key3" } { ..., "na_to_ts": 0.000009, "na_from_ts": 0.000009, "na_to_client": 0.036082, "querytxt": "get key4" } { ..., "na_to_ts": 0.000008, "na_from_ts": 0.000008, "na_to_client": 0.010506, "querytxt": "get key5" } { ..., "na_to_ts": 0.000014, "na_from_ts": 0.000003, "na_to_client": 0.029464, "querytxt": "get key6" } { ..., "na_to_ts": 0.000007, "na_from_ts": 0.000004, "na_to_client": 0.012440, "querytxt": "get key7" } { ..., "na_to_ts": 0.000007, "na_from_ts": 0.000003, "na_to_client": 0.043834, "querytxt": "get key8" } { ..., "na_to_ts": 0.000007, "na_from_ts": 0.050442, "na_to_client": 0.000015, "querytxt": "get key9" } 13年9月20日金曜日
- 76.
- 77. slowlog for neoagent slowlogis sent to MongoDB by Fluentd！ 13年9月20日金曜日
- 78.
- 79. slowlog for neoagent Monitoringslow queries with Monitoring Web Application! 13年9月20日金曜日
- 80.
- 81.
- 82. Problem with signals ✦complicated & easy to mistake ✦ must send multiple signals to master and workers at a time ✦ must link worker’s PID to neoagent’s environment 13年9月20日金曜日
- 83.
- 84. neoctl internal ✦ neoctlsends instructions to master ✦ by UNIX domain socket ✦ master manages worker’ PIDs ✦ and the link between each worker’s PID and environment name, too. ✦ master sends signals master-self and workers. 13年9月20日金曜日
- 85.
- 86. Graceful restart neoctl neoagent master process graceful env1neoagent worker env1 neoagent worker env2 neoagent worker env3 through neoagent_ctl.sock 13年9月20日金曜日
- 87. Graceful restart neoctl neoagent master process graceful env1neoagent worker env1 neoagent worker env2 neoagent worker env3 through neoagent_ctl.sock SIGUSR2 13年9月20日金曜日
- 88. Graceful restart neoctl neoagent master process graceful env1neoagent worker env1 neoagent worker env2 neoagent worker env3 through neoagent_ctl.sock SIGUSR2 worker(env1) exits after processing remaining requests 13年9月20日金曜日
- 89. Graceful restart neoctl neoagent master process graceful env1neoagent worker env1 neoagent worker env2 neoagent worker env3 through neoagent_ctl.sock SIGUSR2 SIGCONT 13年9月20日金曜日
- 90. Graceful restart neoctl neoagent master process graceful env1neoagent worker env1 neoagent worker env2 neoagent worker env3 through neoagent_ctl.sock SIGUSR2 SIGCONT neoagent worker env1(new) fork 13年9月20日金曜日
- 91.
- 92.
- 93.
- 94.
- 95.
- 96. Online version update neoctl neoagent master process neoagent worker env1 neoagent worker env2 neoagent worker env3 update through neoagent_ctl.sock SIGUSR2 workersexits after processing remaining requests 13年9月20日金曜日
- 97.
- 98.
- 99.
- 100.
- 101.
- 102.
- 103.
- 104. Summary ✦ Why memcachedprotocol proxy server is required? ✦ reduced TCP connections ✦ enhance scalability of system ✦ Monitoring is important！ ✦ Don’t guess！ Measure！ ✦ Create tools for simplifying operation！ 13年9月20日金曜日
- 105.