Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Container communication on lattice #2

688 views

Published on

Presentation slide for 第31回PaaS勉強会
http://paas.connpass.com/event/25473/

Published in: Technology
  • Be the first to comment

Container communication on lattice #2

  1. 1. Copyright@2016 NTT corp. All Rights Reserved. 2016/2/10 第31回PaaS勉強会 @sinohara Container Communication on lattice #2
  2. 2. Copyright@2016 NTT corp. All Rights Reserved. ⾃⼰紹介 @sinohara NTTでNWの研究開発してます PaaSたのしい 第28回PaaS勉強会でもLTさせてもらいました!
  3. 3. Copyright@2016 NTT corp. All Rights Reserved.
  4. 4. Copyright@2016 NTT corp. All Rights Reserved. What is lattice? • コンテナ管理システム (ルーティング、HM、ロギングetc) • Pivotal社のOSS • Cloud Foundryのdocker-image特化版
  5. 5. Copyright@2016 NTT corp. All Rights Reserved. 今⽇のトピック latticeでコンテナ間通信 • コンテナ間の通信は未サポート • Dockerはコンテナ間通信の仕組みが沢⼭あるが・・ →latticeはdockerではなくgarden-linux上で動く
  6. 6. Copyright@2016 NTT corp. All Rights Reserved. (前回) • 同⼀Cell(VM)内でコンテナ間通信は成功 • Cell跨りは間に合わず・・ Container #1 Container #2 Container #3 Container #4 LATTICE ARCHITECTURE Cell-1Cell-0 eth0 10.0.1.21 w0m6uagc6vub-0 w0m6uagc6vub-1 10.254.0.1 w0m6uagc6vuc-0 w0m6uagc6vuc-1 10.254.0.5 wb-0m6uf716cb10 10.254.0.2 w0m6uf716lku-0 w0m6uf716lku-1 10.254.0.1 wb-0m6uf716cb20 10.254.0.6 w0m6uf716lkv-0 w0m6uf716lkv-1 10.254.0.5 Eth0 10.0.1.193 54.92.69.17854.92.69.172 Region = ap-northeast-1 Virtual Machine Container Network Interfacedescription Network Tunnel wb-0m6uagc6m510 10.254.0.2 wb-0m6uagc6m500 10.254.0.6 kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster kshino/lattice-php-cluster V0.2.5 Cell-brain 54.92.69.42 HTTP_GET 54.92.69.42.xip.io/index.php?ip=10.254.0.5 (php) http_get(http://10.254.0.5/phpinfo.php) 疎通確認方法
  7. 7. Copyright@2016 NTT corp. All Rights Reserved. 今⽇のトピック latticeでコンテナ間通信 • コンテナ間の通信は未サポート • Dockerはコンテナ間通信の仕組みが沢⼭あるが・・ →latticeはdockerではなくgarden-linux上で動く • Cell間やってきました!
  8. 8. Copyright@2016 NTT corp. All Rights Reserved. ⽅針 • L2 over L3でつなぐ • Ubuntuと相性の良いCanonical Fanを使⽤ • latticeのソース読むのはしんどいので 動いてるモノを触って出来る範囲でトライ
  9. 9. Copyright@2016 NTT corp. All Rights Reserved. FanNetworking •同⼀のClass B-NW内でOverlay + Tunneling •コンテナIPのネゴシエーションがVM内で完結 •Overlayは/8占有。Cell毎に/24を割り当て •コンテナのIPの共有は別途必要 •現在v0.3.0 172.16.0.0/16172.16.0.0/16 172.16.3.5 172.16.23.37 10.3.5.x/24 10.23.37.x/24 VM#1 VM#2 Overlay-NW Class-B NW https://wiki.ubuntu.com/FanNetworking https://launchpad.net/ubuntu/+source/ubuntu-fan https://insights.ubuntu.com/2015/06/24/ubuntu-fan-images/ Overlay-NW
  10. 10. Copyright@2016 NTT corp. All Rights Reserved. 流れを追って 説明します
  11. 11. Copyright@2016 NTT corp. All Rights Reserved. Container #1 Container #2 LATTICE ARCHITECTURE Cell-0 eth0(cell0) 10.0.1.21 Veth-host#1 Veth-guest#1 10.254.0.1 Veth-host#2 Veth-guest#2 10.254.0.5 Region = ap-northeast-1 Virtual Machine Container Network Interfacedescription Network Tunnel Vbr#1 10.254.0.2 Vbr#2 10.254.0.6 tutum/apache-php tutum/apache-php lattice=V0.6.0 Fan Container #1 Container #2 Cell-0 eth0(cell0) 10.0.1.21 Veth-host#1 Veth-guest#1 10.254.0.1 Veth-host#2 Veth-guest#2 10.254.0.5 Vbr#1 10.254.0.2 Vbr#2 10.254.0.6 tutum/apache-php tutum/apache-php 10.0.1.0/24 Step 0. Lattice(初期状態)
  12. 12. Copyright@2016 NTT corp. All Rights Reserved. Container #1 Container #2 Cell-0 eth0(cell0) 10.0.1.21 Veth-host#1 Veth-guest#1 10.254.0.1 Veth-host#2 Veth-guest#2 10.254.0.5 Region = ap-northeast-1 Vbr#1 10.254.0.2 Vbr#2 10.254.0.6 tutum/apache-php tutum/apache-php lattice=V0.6.0 fan-br#1 250.1.21.1 tunl0 250.1.21.1 Container #1 Container #2 Cell-0 eth0(cell0) 10.0.1.21 Veth-host#1 Veth-guest#1 10.254.0.1 Veth-host#2 Veth-guest#2 10.254.0.5 Vbr#1 10.254.0.2 Vbr#2 10.254.0.6 tutum/apache-php tutum/apache-php fan-br#1 250.1.21.1 tunl0 250.1.21.1 10.0.1.0/24 L2 over L3 tunnel 250.0.0.0/8 Step 1. FanをCellにセットアップ • カーネル更新(3.19.0-41) • ubuntu-fanのinstall • Fan⽤Overlay-NW作成(250/8) LATTICE ARCHITECTURE Virtual Machine Container Network Interfacedescription Network Tunnel Fan
  13. 13. Copyright@2016 NTT corp. All Rights Reserved. Container #1 Container #2 Cell-0 eth0(cell0) 10.0.1.21 Veth-host#1 Veth-guest#1 10.254.0.1 Veth-host#2 Veth-guest#2 10.254.0.5 Region = ap-northeast-1 Vbr#1 10.254.0.2 Vbr#2 10.254.0.6 tutum/apache-php tutum/apache-php lattice=V0.6.0 fan-br#1 250.1.21.1 fan-veth-h#1 fan-veth-h#2 fan-veth-g#1 250.1.21.2 fan-veth-g#2 250.1.21.3 tunl0 250.1.21.1 Container #1 Container #2 Cell-0 eth0(cell0) 10.0.1.21 Veth-host#1 Veth-guest#1 10.254.0.1 Veth-host#2 Veth-guest#2 10.254.0.5 Vbr#1 10.254.0.2 Vbr#2 10.254.0.6 tutum/apache-php tutum/apache-php fan-br#1 250.1.21.1 fan-veth-h#1 fan-veth-h#2 fan-veth-g#1 250.1.21.2 fan-veth-g#2 250.1.21.3 tunl0 250.1.21.1 10.0.1.0/24 L2 over L3 tunnel 250.0.0.0/8 Step 2. 仮想NIC作成 LATTICE ARCHITECTURE Virtual Machine Container Network Interfacedescription Network Tunnel Fan • netnsで仮想NICをコンテナ内に繋ぎ込み • コンテナ内のルーティングテーブルに250/8の設定追加
  14. 14. Copyright@2016 NTT corp. All Rights Reserved. Container #1 Container #2 Cell-0 eth0(cell0) 10.0.1.21 Veth-host#1 Veth-guest#1 10.254.0.1 Veth-host#2 Veth-guest#2 10.254.0.5 Region = ap-northeast-1 Vbr#1 10.254.0.2 Vbr#2 10.254.0.6 tutum/apache-php tutum/apache-php lattice=V0.6.0 fan-br#1 250.1.21.1 fan-veth-h#1 fan-veth-h#2 fan-veth-g#1 250.1.21.2 fan-veth-g#2 250.1.21.3 tunl0 250.1.21.1 Container #1 Container #2 Cell-0 eth0(cell0) 10.0.1.21 Veth-host#1 Veth-guest#1 10.254.0.1 Veth-host#2 Veth-guest#2 10.254.0.5 Vbr#1 10.254.0.2 Vbr#2 10.254.0.6 tutum/apache-php tutum/apache-php fan-br#1 250.1.21.1 fan-veth-h#1 fan-veth-h#2 fan-veth-g#1 250.1.21.2 fan-veth-g#2 250.1.21.3 tunl0 250.1.21.1 10.0.1.0/24 L2 over L3 tunnel 250.0.0.0/8 完成! • IPinIP protocol(4)→許容 LATTICE ARCHITECTURE Virtual Machine Container Network Interfacedescription Network Tunnel Fan
  15. 15. Copyright@2016 NTT corp. All Rights Reserved. DEMO riak clusterを動かしてみた
  16. 16. Copyright@2016 NTT corp. All Rights Reserved. Cell-0 DEMO概要 Router (lattice-brain) Cell-1 250.0.0.0/8 10.0.0.0/8 sinatra riak sinatra riak clustering • 250/8でriak cluster構築 • 外から250/8に直接アクセス不可 →Sinatraでインタワークし、latticeのエンドポイント からriak clusterを直接操作 http://52.1.2.3.xip.io:8098/ http://250.1.2.2:8098http://250.1.3.4:8098 InterWork InterWork tcp-route 8098:8080tcp-route 8098:8080
  17. 17. Copyright@2016 NTT corp. All Rights Reserved. まとめ Lattice上でのコンテナ間通信(L2 over L3)の紹介 • PoCレベル • netnsとかCellでの作業があります • /8のprivate address(10/8)をlatticeが占有し ちゃってる問題(IANA未使⽤の250/8で代替) riak cluster on latticeのデモ Latticeでクラスタリング、 あなたなら何に使いますか?

×