Container Networking Deep Dive
•Download as PPTX, PDF•
17 likes•11,247 views
Japan Container Days v18.04 で発表したスライドです。
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Container Networking Deep Dive
Similar to Container Networking Deep Dive (20)
More from Hirofumi Ichihara
More from Hirofumi Ichihara (10)
Container Networking Deep Dive
- 1. Copyright©2018 NTT corp. All Rights Reserved. 1 Container Networking Deep Dive 日本電信電話株式会社 ソフトウェアイノベーションセンタ 市原 裕史
- 2. 2Copyright©2018 NTT corp. All Rights Reserved. 自己紹介 市原 裕史 • 所属 • NTT SIC 分散処理基盤技術プロジェクト • OpenStack Developer • Neutron Core Reviewer、その他プロジェクトへの貢献 • 技術: SDN/NFV中心 • パブリッククラウドの開発 • LinuxconでDPDKの性能の発表 • Neutronへの機能提案 • Docker Network Plugin開発
- 3. 3Copyright©2018 NTT corp. All Rights Reserved. Container Orchestration Tools
- 4. 4Copyright©2018 NTT corp. All Rights Reserved. CNI vs CNM • Container Network Model (CNM) • Container Network Interface (CNI) CNM IPAM プラグイン CNM Network プラグイン Get/Release Pool Create/Delete Network Create/Delete Endpoint Join/Leave Endpoint CNI プラグイン Add/Remove interface Get/Release IP Address
- 5. 5Copyright©2018 NTT corp. All Rights Reserved. ベアメタルから仮想ネットワーク ファイアウォール ロードバランサ L2スイッチ サーバ ルータ VM Compute node vSwitch VM vRouter vFW Network node vLB vSwitch VMVM
- 6. 6Copyright©2018 NTT corp. All Rights Reserved. コンテナネットワーク Worker Container Container Container Container Container Container Router + FW + LB + Switch
- 7. 7Copyright©2018 NTT corp. All Rights Reserved. コンテナネットワーク技術 Router + FW + LB + Switch • Linux bridge, veth, routing • iptables • OpenVSwitch, VPP
- 8. 8Copyright©2018 NTT corp. All Rights Reserved. 検証環境 version 1.10.0 by kubeadm version 18.03.0-ce version 0.9.1 calico version 3.0.4 cni version 2.0.3
- 9. 9Copyright©2018 NTT corp. All Rights Reserved. flannel • シンプルで簡単な設定でL3ネットワークを利用可能 • k8s APIを使用することで外部DB不要 • Network Policyには未対応
- 10. 10Copyright©2018 NTT corp. All Rights Reserved. ホストインターフェース構成 ens7: 192.168.33.11/24 flannel.1: 10.244.3.0/32 cni0: 10.244.3.1/24 Pod A eth0: 10.244.3.x/24 eth0: 192.168.33.12/24 flannel.1: 10.244.2.0/32 cni0: 10.244.2.1/24 Pod B eth0: 10.244.2.y/24 $ ip a 3: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> inet 192.168.33.11/24 brd 192.168.33.255 scope global ens7 6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> inet 10.244.3.0/32 scope global flannel.1 7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> inet 10.244.3.1/24 scope global cni0
- 11. 11Copyright©2018 NTT corp. All Rights Reserved. vxlan インターフェース $ ip -d link show flannel.1 6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP vxlan id 1 local 192.168.33.11 dev ens7 srcport 0 0 dstport 8472 ens7: 192.168.33.11/24 flannel.1: 10.244.3.0/32 vxlan id 1 dev ens7 cni0: 10.244.3.1/24 Pod A eth0: 10.244.3.x/24 eth0: 192.168.33.12/24 flannel.1: 10.244.2.0/32 vxlan id 1 dev ens7 cni0: 10.244.2.1/24 Pod B eth0: 10.244.2.y/24
- 12. 12Copyright©2018 NTT corp. All Rights Reserved. ルーティング構成 (1 route) $ ip r 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink ens7: 192.168.33.11/24 flannel.1: 10.244.3.0/32 cni0: 10.244.3.1/24 Pod A eth0: 10.244.3.x/24 eth0: 192.168.33.12/24 flannel.1: 10.244.2.0/32 cni0: 10.244.2.1/24 Pod B eth0: 10.244.2.y/24 to: Pod B to: Pod A
- 13. 13Copyright©2018 NTT corp. All Rights Reserved. パケット処理 (1arp, 1FDB entry) ens7: 192.168.33.11/24 flannel.1: 10.244.3.0/32 vxlan id 1 dev ens7 cni0: 10.244.3.1/24 Pod A eth0: 10.244.3.x/24 eth0: 192.168.33.12/24 flannel.1: 10.244.2.0/32 vxlan id 1 dev ens7 mac: 22:f7:97:a0:9c:55 cni0: 10.244.2.1/24 Pod B eth0: 10.244.2.y/24 $ ip n 10.244.2.0 dev flannel.1 lladdr 22:f7:97:a0:9c:55 PERMANENT $ bridge fdb show dev flannel.1 22:f7:97:a0:9c:55 dst 192.168.33.12 self permanent to: Pod B (to: 22:f7:97:a0:9c:55 ) to: Pod B (to: 22:f7:97:a0:9c:55 )
- 14. 14Copyright©2018 NTT corp. All Rights Reserved. vxlan 情報反映契機 // How it works: // Create the vxlan device but don't register for any L2MISS or L3MISS messages // Then, as each remote host is discovered (either on startup or when they are added), do the following // 1) create routing table entry for the remote subnet. It goes via the vxlan device but also specifies a next hop (of the remote flannel host). // 2) Create a static ARP entry for the remote flannel host IP address (and the VTEP MAC) // 3) Create an FDB entry with the VTEP MAC and the public IP of the remote flannel daemon. // // In this scheme the scaling of table entries is linear to the number of remote hosts - 1 route, 1 arp entry and 1 FDB entry per host https://github.com/coreos/flannel/blob/master/backend/vxlan/vxlan.go#L43-L50 • 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink • 10.244.2.0 dev flannel.1 lladdr 22:f7:97:a0:9c:55 PERMANENT • 22:f7:97:a0:9c:55 dst 192.168.33.12 self permanent
- 15. 15Copyright©2018 NTT corp. All Rights Reserved. flannel CNI 適用後の iptables filter 差分 -A FORWARD -s 10.244.0.0/16 -j ACCEPT -A FORWARD -d 10.244.0.0/16 -j ACCEPT ens7: 192.168.33.11/24 flannel.1: 10.244.3.0/32 cni0: 10.244.3.1/24 Pod A eth0: 10.244.3.x/24 eth0: 192.168.33.12/24 flannel.1: 10.244.2.0/32 cni0: 10.244.2.1/24 Pod B eth0: 10.244.2.y/24 to: Pod B to: Pod A
- 16. 16Copyright©2018 NTT corp. All Rights Reserved. Calico • シンプルなL3ルーティングを利用 • BGPを用いてルーティング情報を伝搬可能 • IPIPによるトンネリングにも対応 • Network Policyに対応
- 17. 17Copyright©2018 NTT corp. All Rights Reserved. ホストインタフェース ens7: 192.168.33.11/24 tunl0: 192.168.79.192/32 Pod A eth0: 192.168.79.x/32 eth0: 192.168.33.12/24 tunl0 Pod B $ ip a 5: tunl0@NONE: <NOARP,UP,LOWER_UP> link/ipip 0.0.0.0 brd 0.0.0.0 inet 192.168.79.192/32 brd 192.168.79.192 scope global tunl0 9: calic4324f5b021@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0 calic4324f5b021 eth0: 192.168.179.y/32 caliYYYY
- 18. 18Copyright©2018 NTT corp. All Rights Reserved. ルーティング構成 $ ip r 192.168.179.0/26 via 192.168.33.12 dev tunl0 proto bird onlink 192.168.79.194 dev calic4324f5b021 scope link ens7: 192.168.33.11/24 tunl0: 192.168.79.192/32 Pod A eth0: 192.168.79.194/32 eth0: 192.168.33.12/24 tunl0 Pod B calic4324f5b021 eth0: 192.168.179.y/32 caliYYYY to: Pod B to: Pod A
- 19. 19Copyright©2018 NTT corp. All Rights Reserved. PodA から PodB への iptables filter -A FORWARD -m comment --comment "cali:wUHhoiAYhphO9Mso" -j cali-FORWARD -A cali-FORWARD -m comment --comment "cali:JV9-BRWxjz8He5Ib" -j MARK --set-xmark 0x0/0xe000000 -A cali-FORWARD -o cali+ -m comment --comment "cali:B81FOaQNZymbX9H8" -j cali-to-wl- dispatch -A cali-to-wl-dispatch -o calic4324f5b021 -m comment --comment "cali:VlIiuQZf0oRkWoyL" - g cali-tw-calic4324f5b021 -A cali-tw-calic4324f5b021 -m comment --comment "cali:zjyrnaWCVNz-MaWt" -j MARK --set- xmark 0x0/0x1000000 -A cali-tw-calic4324f5b021 -m comment --comment "cali:uTf5kcRs6DeXSj7Z" -j cali-pri- kns.default -A cali-pri-kns.default -m comment --comment "cali:5yVkkQ7pBcxxkSaE" -j MARK --set- xmark 0x1000000/0x1000000 -A cali-pri-kns.default -m comment --comment "cali:pZi5w5MrTl3DghSD" -m mark --mark 0x1000000/0x1000000 -j RETURN -A cali-tw-calic4324f5b021 -m comment --comment "cali:Z5z02E5-p7S1WwQF" -m comment - -comment "Return if profile accepted" -m mark --mark 0x1000000/0x1000000 -j RETURN -A cali-FORWARD -m comment --comment "cali:I1Ki7aNgQsJFzEpG" -m comment --comment "Policy explicitly accepted packet." -m mark --mark 0x1000000/0x1000000 -j ACCEPT
- 20. 20Copyright©2018 NTT corp. All Rights Reserved. 膨大な iptables ルール $ sudo iptables-save | wc -l 235 PodA起動後のルール数 calico の場合 $ sudo iptables-save | wc -l 85 PodA起動後のルール数 flannel の場合 calico の場合 1 Pod 起動毎16ルール増加する
- 21. 21Copyright©2018 NTT corp. All Rights Reserved. IPIP パケット ens7: 192.168.33.11/24 tunl0: 192.168.79.192/32 Pod A eth0: 192.168.79.194/32 eth0: 192.168.33.12/24 tunl0 Pod B calic4324f5b021 eth0: 192.168.179.y/32 caliYYYY to: Pod B to: Pod A IP 192.168.33.11 > 192.168.33.12: IP 192.168.79.194 > 192.168.179.2: ICMP echo request, id 53, seq 1, length 64 (ipip-proto-4) IP 192.168.33.12 > 192.168.33.11: IP 192.168.179.2 > 192.168.79.194: ICMP echo reply, id 53, seq 1, length 64 (ipip-proto-4)
- 22. 22Copyright©2018 NTT corp. All Rights Reserved. トンネリングプロトコル Pod A eth0: 192.168.1.10 Pod B eth0: 192.168.2.10 eth0: 10.0.1.100 eth0: 10.0.2.200 192.168.1.10 192.168.2.20 知らない to: 10.0.2.200 to: 192.168.2.10 to: 192.168.2.10 to: 192.168.2.10 パフォーマンスボトルネック
- 23. 23Copyright©2018 NTT corp. All Rights Reserved. パフォーマンス改善 • オフロード機能を適切に設定する • VXLANハードウェアオフロード • トンネリングプロトコルを用いない • Flannel: host-gw モード • Calico: ルータへのBGP伝搬 • DPDKやXDPなどのパケット処理高速化技術を用いる
- 24. 24Copyright©2018 NTT corp. All Rights Reserved. Network Policy all deny ポリシー設定後に特定のラベルを持つ Pod のア クセス許可ポリシー作成し Pod(192.168.79.203) 起動 -A cali-pi-_PfqSzIS1AirpjL0oXbg -m comment --comment "cali:8wIHtkclEDHwaHZ2" -m set --match-set cali4- s:AMAB7BNa5u3MmIiwZrblkWt src -j MARK --set-xmark 0x1000000/0x1000000 $ sudo ipset list cali4-s:AMAB7BNa5u3MmIiwZrblkWt Name: cali4-s:AMAB7BNa5u3MmIiwZrblkWt Type: hash:ip Revision: 4 Header: family inet hashsize 1024 maxelem 1048576 Size in memory: 176 References: 3 Members: 192.168.79.203
- 25. 25Copyright©2018 NTT corp. All Rights Reserved. ネットワークと向き合う • tcpdumpしながら膨大なiptablesルールとlinux network設定を読み解く • 実際にハマった問題→ https://github.com/coreos/flannel/commit/5df82dcbcb102 05c419f2a9829c9a50e1e566352 • 実環境、サービス要件から適切なプラグイン、ネットワー ク構成を選択する