Download to read offline
![conntrack
5 ïðîáëåì
ïðè
Àëåêñàíäð
Êîëåñåíü
wargaming.net
net.ipv4.netfilter.ip_conntrack_max = 294912
net.ipv4.netfilter.ip_conntrack_count = 294118
Ïåðåïîëíåíèå:
[8727241.821130] ip_conntrack: table full, dropping packet.
[8727243.790372] ip_conntrack: table full, dropping packet.
[8727248.908501] ip_conntrack: table full, dropping packet.
[8727253.859720] ip_conntrack: table full, dropping packet.
[8727258.846404] ip_conntrack: table full, dropping packet.
[8727264.276727] ip_conntrack: table full, dropping packet.
Áîëüøîé conntrack (over 300K): 97.5%si
Ðåøåíèå: dport 80 -> NOTRACK](https://image.slidesharecdn.com/5-issues-120212095057-phpapp01/85/5-issues-2-320.jpg)
![Out Of TCP Mem
5 ïðîáëåì
ïðè
Àëåêñàíäð
Êîëåñåíü
3494-[9404385.020387] TCP: too many of orphaned sockets
wargaming.net
3496:[9404390.463850] Out of socket memory
Äèàãíîñòèêà:
$ cat /proc/net/sockstat
sockets: used 7385
TCP: inuse 5236 orphan 98 tw 2696 alloc 7126 mem 25421
UDP: inuse 10 mem 0
RAW: inuse 0
FRAG: inuse 0 memory 0
$ /sbin/sysctl -a | grep tcp_mem
net.ipv4.tcp_mem = 196608 262144 393216
$ /sbin/sysctl -a | grep orphans
net.ipv4.tcp_max_orphans = 131072](https://image.slidesharecdn.com/5-issues-120212095057-phpapp01/85/5-issues-4-320.jpg)
5 issues
- 1. TCP socket backlog 5 ïðîáëåì ïðè Àëåêñàíäð Êîëåñåíü Î÷åðåäü íà ñîêåòå 1K -> 20K rpm wargaming.net Ïàäàåò êëàñòåð - âñå èäóò íà ñàéò: Èíâàëèäàöèÿ êýøà - âñå æäóò: 20K -> 1K rpm Ðåøåíèå: net.core.somaxconn = 32768 net.core.netdev_max_backlog = 32768 net.ipv4.tcp_max_syn_backlog = 32768 ÍÅ UNIX ñîêåò (ECONNREFUSED) 502 Bad Gateway
- 2. conntrack 5 ïðîáëåì ïðè Àëåêñàíäð Êîëåñåíü wargaming.net net.ipv4.netfilter.ip_conntrack_max = 294912 net.ipv4.netfilter.ip_conntrack_count = 294118 Ïåðåïîëíåíèå: [8727241.821130] ip_conntrack: table full, dropping packet. [8727243.790372] ip_conntrack: table full, dropping packet. [8727248.908501] ip_conntrack: table full, dropping packet. [8727253.859720] ip_conntrack: table full, dropping packet. [8727258.846404] ip_conntrack: table full, dropping packet. [8727264.276727] ip_conntrack: table full, dropping packet. Áîëüøîé conntrack (over 300K): 97.5%si Ðåøåíèå: dport 80 -> NOTRACK
- 3. TIME_WAIT 5 ïðîáëåì ïðè Àëåêñàíäð Êîëåñåíü frontend äëÿ îáùåíèÿ ñ âíåøíèì ìèðîì wargaming.net áûñòðûé backend (1..10ms) frontend <-> backend HTTP/1.0 (conn per req) $ netstat -napt | awk '{print $6}' | sort | uniq -c ... 63564 TIME_WAIT Íåõâàòêà èñõîäÿùèõ ïîðòîâ! Ðåøåíèå: net.ipv4.ip_local_port_range = 2000 61000 net.ipv4.tcp_max_tw_buckets = 131072 net.ipv4.tcp_tw_recycle = 0 net.ipv4.tcp_tw_reuse = 1
- 4. Out Of TCPMem 5 ïðîáëåì ïðè Àëåêñàíäð Êîëåñåíü 3494-[9404385.020387] TCP: too many of orphaned sockets wargaming.net 3496:[9404390.463850] Out of socket memory Äèàãíîñòèêà: $ cat /proc/net/sockstat sockets: used 7385 TCP: inuse 5236 orphan 98 tw 2696 alloc 7126 mem 25421 UDP: inuse 10 mem 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ /sbin/sysctl -a | grep tcp_mem net.ipv4.tcp_mem = 196608 262144 393216 $ /sbin/sysctl -a | grep orphans net.ipv4.tcp_max_orphans = 131072
- 5. memcached 5 ïðîáëåì ïðè Àëåêñàíäð Êîëåñåíü wargaming.net ïëîõî ìàñøòàáèðóåòñÿ thread-àìè (-t 64 çëî) 8192 pthread mutex íà âñå êëþ÷è, -t 6 - OK! (íå áîëüøå) Nthreads == Ncpu Ìàñøòàáèðîâàòü ïðîöåññàìè! Êîíñèñòåíòíîå êîëüöî êåøèðîâàíèÿ - íåñêîëüêî èíñòàíñîâ
- 6. nscd (bonus) 5ïðîáëåì ïðè Àëåêñàíäð Êîëåñåíü Êýøèðîâàíèå îòâåòîâ DNS wargaming.net nscd must have!