Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

解决Lvs上行丢包的过程和收获

479 views

Published on

解决Lvs上行丢包的过程和收获

Published in: Technology
  • Be the first to comment

解决Lvs上行丢包的过程和收获

  1. 1. LVS -
  2. 2. 100ms
  3. 3. 200ms
  4. 4. 10.11.128.29 qf.56.com - [25/Jul/2017:18:46:33 +0800] GET /chatpay/v1/ sendGiftHistory.do?roomId=7991332&callback=jsonp_callback_qf56com sendGiftHistory HTTP/1.1 "200" 711 "https://qf.56.com/7991332" "Mozilla/ 5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36" "60.6.202.75" "0.212" "MISS" "0.212" 10.10.83.166 qf.56.com - [25/Jul/2017:18:46:33 +0800] GET /chatpay/v1/ sendGiftHistory.do?roomId=7991332&callback=jsonp_callback_qf56com sendGiftHistory HTTP/1.1 "200" 711 "https://qf.56.com/7991332" "Mozilla/ 5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36" "60.6.202.75, 10.11.128.29" "0.010" "10.10.193.228:8080" "0.009"
  5. 5. 200ms
  6. 6. 3 5
  7. 7. proxy -> vip: location ^~ /chatpay/v1/xxx { proxy_pass http://10.10.91.173; }
  8. 8. netstat -s
  9. 9. 1w reset
  10. 10. ipvsadm -lnc TCP 00:21 FIN_WAIT 10.10.91.126:57236 10.10.91.173:80 10.10.93.152:80 TCP 01:56 FIN_WAIT 10.10.83.166:37574 10.10.91.173:80 10.10.93.158:80
  11. 11. FIN_WAIT
  12. 12. LVS tcpdump -i eth0 “ip[9]==4"
  13. 13. TCP Retransmission
  14. 14. TCP segment of a reassembled PDU
  15. 15. [@zw_91_126 /opt/nginx_logs]# cat access.log|grep "/chatpay/v1/ getMsg.do?ts=1500994855&withTop=1&callback=head_line_cb" 122.228.21.182 qf.56.com - [25/Jul/2017:23:01:59 +0800] GET /chatpay/ v1/getMsg.do?ts=1500994855&withTop=1&callback=head_line_cb HTTP/1.1 "200" 442 "https://qf.56.com/1012148" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" "115.194.33.144" "0.209" "BYPASS" "0.209"
  16. 16. /etc/sysctl.conf net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_fin_timeout = 3 net.ipv4.tcp_keepalive_time = 30 net.ipv4.tcp_synack_retries = 1 net.ipv4.tcp_syn_retries = 1 net.ipv4.tcp_timestamps = 1
  17. 17. /usr/sbin/ntpdate ntp.sohu.com > /dev/null 2>&1
  18. 18. lb_algo
  19. 19. RS frame 200ms
  20. 20. RS mtu IPIP /sbin/ifconfig tunl0 mtu 1440
  21. 21. Retransmission mtu
  22. 22. LVS LVS RS “ ”
  23. 23. ethtool -k eth0 Features for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: on
  24. 24. LVS LRO/GRO LVS LRO/GRO sequence LVS LVS MTU IMCP
  25. 25. LVS LRO/GRO ethtool -K eth0 lro off ethtool -K eth0 gro off ethtool -k eth0 …… generic-receive-offload: off large-receive-offload: off ……
  26. 26. http://blog.csdn.net/hzsunshine/article/details/58638310 http://blog.csdn.net/wzyzzu/article/details/51381578 http://www.bo56.com/ %E4%BD%BF%E7%94%A8lvs%EF%BC%8C%E5%85%B3%E9%97%AD %E7%BD%91%E5%8D%A1lrogro%E5%8A%9F%E8%83%BD/ http://blog.chinaunix.net/uid-20639775-id-3529535.html http://man.linuxde.net/ethtool

×