Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

L3HA-VRRP-20141201

5,455 views

Published on

日本OpenStackユーザ会 第20回勉強会でお話した資料です。

Published in: Technology
  • Sex in your area is here: ❶❶❶ http://bit.ly/2Qu6Caa ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ♥♥♥ http://bit.ly/2Qu6Caa ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

L3HA-VRRP-20141201

  1. 1. 日本OpenStackユーザ会 第20回勉強会 Neutron L3 HA (VRRP) 2014-12-01 Manabu Ori
  2. 2. 自己紹介 ● 名前: 織 学 (おり まなぶ) ● 所属: レッドハット ● Twitter: @orimanabu
  3. 3. 自己紹介 ● 名前: 織 学 (おり まなぶ) ● 所属: レッドハット ● Twitter: @orimanabu ● slideshareに上げた資料たち – POWER + PostgreSQL – POWER + KVM
  4. 4. Neutron L3 agentとは ● OpenStackのネットワーク仮想化で使用するエージェ ントのひとつ – 他にはMetadata Proxy agentとかDHCP agentとか ● テナントネットワークのサブネット間接続、テナント VMと外部ネットワークとの接続等を制御する ● 複数ノードで起動可能 – 仮想ルータはどこか1ノード上で実行される 仮想ルータ #1 仮想ルータ #2 L3 Agent #1 L3 Agent #2 VM VM VM パブリック ネットワーク テナント ネットワーク
  5. 5. これまでのL3 agentの冗長化 ● コントローラーノードによるリスケジューリング – allow_automatic_l3agent_failover = True – Juno以降で使用可能 – https://review.openstack.org/#/c/110893/ – https://bugs.launchpad.net/neutron/+bug/1174591 ● L3 Agent Healthcheck – OpenStackと独立して開発 (by eNovance) – Grizzly, Havana, Icehouseで使用可能 – https://github.com/enovance/neutron-l3-healthcheck ● Pacemaker等でHAクラスター化 フェイルオーバーに 時間がかかる フェイルオーバーに 時間がかかる 設定が大変
  6. 6. L3 HA概要 ● L3 agentの冗長化機能を拡張/ドライバの形で実装 ● 今の実装はkeepalivedを使用 – VRRPを用いて死活監視 – フェイルオーバーはRPC layerから独立 ● テナントごとにひとつVRRPハートビートネットワークを作 成 – ハートビートネットワークも通常のNeutronネットワーク – 物理的にはテナントネットワークを通る ● 全てのテナント分割方式 (VXLAN, VLAN, GRE, ...)で使用可能 ● フェイルオーバー時は、ゲートウェイのIPアドレスおよび Floating IPを仮想IPアドレスとして引き継ぐ
  7. 7. VRRPとは ● VRRP: Virtual Router Redundancy Protocol ● RFC3768 ● 複数のルータから仮想的なルータを構成 – マスタールータ ● 一時点に1台のみ ● 実際にフォワーディング処理を行う – バックアップルータ ● 通常は待機 ● 障害時にマスターになる ● それぞれのルータは、ステータスの確認メッセージを交換するこ とで障害を検知 ● First Hop Redundancy Protocol – サーバ、PC等のデフォルトルートとしてIPアドレス、MACアドレスを 提供
  8. 8. ネットワークノードの構成 - non-HAの場合 - Network Node #1 Network Node #2 eth2 br-ex qg-BBB qr-AAA br-int IP address int-br-tenant phy-br-tenant br-tenant eth1 eth2 br-ex qg-EEE qr-DDD br-int IP address int-br-tenant phy-br-tenant br-tenant eth1 パブリックネットワーク テナントネットワーク IP address IP address qrouter-xxxx qrouter-yyyy
  9. 9. ネットワークノードの構成 - HAの場合 - Network Node #1 Network Node #2 eth2 br-ex qg-BBB br-int qr-AAA qrouter-xxxx qrouter-yyyy ha-CCC int-br-tenant phy-br-tenant br-tenant eth1 eth2 br-ex qg-EEE br-int ha-FFF qr-DDD int-br-tenant phy-br-tenant br-tenant eth1 パブリックネットワーク VRRPハートビート テナントネットワーク VIP VIP VIP VIP Gateway Floating IP Fixed IP keepalived keepalived IP address IP address
  10. 10. 制限事項 ● VRRPハートビート用のネットワークは1つのみ – テナントネットワークを流れる ● テナント当たりの仮想ルータは最大256個 – ハートビートネットワークが1つ かつ VRIDが8ビット ● セッションの引き継ぎは現時点では未サポート ● DVRとの同時使用は現時点では未サポート ● (LB|FW|VPN)aaSとの統合は現時点では未サポート ● 今後の改良予定 – どれがマスターがわかりにくい問題 – 手動フェイルオーバーの方法問題 – L2populationと仲良くなる
  11. 11. Gerrit Topics ● https://review.openstack.org/#/q/topic:bp/l3-h igh-availability,n,z ●
  12. 12. 関連する設定パラメータ 最低限の設定 ● /etc/neutron/neutron.conf – [Default] ● l3_ha = True その他のパラメータ ● /etc/neutron/neutron.conf – [Default] ● max_l3_agents_per_router 3 ● min_l3_agents_per_router 2 ● l3_ha_net_cidr 169.254.192.0/18 ● /etc/neutron/l3_agent.ini – [Default] ● agent_mode legacy ● ha_confs_path $state_path/ha_confs ● ha_vrrp_auth_type PASS ● ha_vrrp_auth_password openstack ● ha_vrrp_advert_int 2
  13. 13. 動き ● neutron router-createすると – ハートビート用のNeutronネットワークを作成 – ハートビート用のインタフェースをルータの Namespace内に作成 – keepalivedを起動 Public Network Management/API Network eth0 eth0 eth2 eth0 eth2 eth0 controller network1 network2 compute1 eth1 eth1 eth1 eth1 Tenant Network
  14. 14. ルータ作成 (1) ● neutron router-createすると... # neutron router-create --tenant-id $(keystone tenant-list | awk '/demo/ {print $2}') router_demo Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | distributed | False | | external_gateway_info | | | ha | True | | id | e7e98ef7-36d5-4544-be07-c83ddf7e67ea | | name | router_demo | | routes | | | status | ACTIVE | | tenant_id | 328ffb467eb24c3595b549516b8acbd0 | +-----------------------+--------------------------------------+
  15. 15. ルータ作成 (2) ● 両ノードのL3 agentで動いている # neutron l3-agent-list-hosting-router router_demo +--------------------------------------+---------------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+---------------+----------------+-------+ | 1c407f95-5a62-4577-a86d-623943ab0283 | juno-network1 | True | :-) | | 8ad4e4df-cd8e-4c6e-bea2-8101a1225019 | juno-network2 | True | :-) | +--------------------------------------+---------------+----------------+-------+ ● ハートビート用のネットワークができている # neutron net-list +--------------------------------------+----------------------------------------------------+----------------------------------| id | name | subnets +--------------------------------------+----------------------------------------------------+----------------------------------| 616a2195-162c-4e09-8afa-a347c56b9c11 | external | 3c705be7-6d51-| 218b2842-1832-4ec6-a979-5efdf01453bf | HA network tenant 328ffb467eb24c3595b549516b8acbd0 | 288d389b-33c9-+--------------------------------------+----------------------------------------------------+----------------------------------
  16. 16. ルータ作成 (3) ● ハートビート用ネットワークの情報 # neutron net-show 218b2842-1832-4ec6-a979-5efdf01453bf +---------------------------+----------------------------------------------------+ | Field | Value | +---------------------------+----------------------------------------------------+ | admin_state_up | True | | id | 218b2842-1832-4ec6-a979-5efdf01453bf | | name | HA network tenant 328ffb467eb24c3595b549516b8acbd0 | | provider:network_type | vlan | | provider:physical_network | physnet-tenant | | provider:segmentation_id | 2001 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 288d389b-33c9-4559-8ec8-98b4faac2274 | | tenant_id | | +---------------------------+----------------------------------------------------+ # neutron subnet-show 288d389b-33c9-4559-8ec8-98b4faac2274 +-------------------+------------------------------------------------------+ | Field | Value | +-------------------+------------------------------------------------------+ | allocation_pools | {"start": "169.254.192.1", "end": "169.254.255.254"} | | cidr | 169.254.192.0/18 | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | | | host_routes | | | id | 288d389b-33c9-4559-8ec8-98b4faac2274 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | HA subnet tenant 328ffb467eb24c3595b549516b8acbd0 | | network_id | 218b2842-1832-4ec6-a979-5efdf01453bf | | tenant_id | | +-------------------+------------------------------------------------------+ ● network_type、physical_networkは テナント用ネットワークと同じ ● 独立したsegmentation_idが振られる ● テナントIDは空 ● /etc/neutron/neutron.confで変更可能 [DEFAULT] l3_ha_net_cidr = 169.254.192.0/18
  17. 17. ルータ作成 (4) ● ハートビート用のポートの確認 # neutron port-list +--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+ | 00ab5988-dedd-421d-921a-61698de65155 | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | fa:16:3e:7c:52:0d | {"subnet_id": "288d389b-33c9-4559-8ec8-98b4faac2274", "ip_address": "169.254.192.2"} | | ef4a5a34-17a1-4008-916a-72ff19424b61 | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | fa:16:3e:16:91:0f | {"subnet_id": "288d389b-33c9-4559-8ec8-98b4faac2274", "ip_address": "169.254.192.1"} | +--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+ # for port_id in $(neutron port-list | awk '/HA port/ {print $2}'); do neutron port-show ${port_id}; done +-----------------------+--------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | juno-network1 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | device_id | 6ef3ebbe-dcd2-4af3-a4d9-def49792f09b | | device_owner | network:router_ha_interface | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "9dc14d95-7588-4a02-b5f8-5436b0669916", "ip_address": "169.254.192.2"} | | id | 208ace4e-077c-4b23-86a3-d67a7d5ca612 | | mac_address | fa:16:3e:99:54:58 | | name | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | | network_id | ca16daee-6272-421a-ba9d-6859a10ffce4 | | security_groups | | | status | ACTIVE | | tenant_id | | +-----------------------+--------------------------------------------------------------------------------------+ +-----------------------+--------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | juno-network2 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | device_id | 6ef3ebbe-dcd2-4af3-a4d9-def49792f09b | | device_owner | network:router_ha_interface | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "9dc14d95-7588-4a02-b5f8-5436b0669916", "ip_address": "169.254.192.1"} | | id | e6a1529c-6a52-418e-8507-7531d343a84a | | mac_address | fa:16:3e:a3:bc:e6 | | name | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | | network_id | ca16daee-6272-421a-ba9d-6859a10ffce4 | | security_groups | | | status | ACTIVE | | tenant_id | | +-----------------------+--------------------------------------------------------------------------------------+
  18. 18. keepalived (1) ● keepalived関連のファイル置き場 # ls -F /var/lib/neutron/ dhcp/ external/ ha_confs/ lock/ metadata_proxy= ● keepalived関連のファイルたち # find /var/lib/neutron/ha_confs/ -type f /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/keepalived.conf /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid-vrrp router id 状態変化に応じて 呼ばれるスクリプト 設定ファイル master or backup
  19. 19. keepalived (2) ● 設定ファイル # cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/keepalived.conf vrrp_sync_group VG_1 { group { VR_1 } notify_master "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh" notify_backup "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh" notify_fault "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh" } vrrp_instance VR_1 { state BACKUP interface ha-ef4a5a34-17 virtual_router_id 1 priority 50 nopreempt advert_int 2 track_interface { ha-ef4a5a34-17 } virtual_ipaddress { 172.16.99.100/24 dev qg-a6ee9669-8f } virtual_ipaddress_excluded { 172.16.99.101/32 dev qg-a6ee9669-8f 192.168.10.1/24 dev qr-3d8666f0-c4 } virtual_routes { 0.0.0.0/0 via 172.16.99.1 dev qg-a6ee9669-8f } } ● パブリックネットワークの ゲートウェイ ● Floating IP ● テナントネットワークの ゲートウェイ
  20. 20. keepalived (3) ● 通知スクリプトたち マスターになったら metadata proxyも起動する # cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh #!/usr/bin/env bash kill -9 $(cat /var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid) echo -n backup > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state # cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh #!/usr/bin/env bash kill -9 $(cat /var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid) echo -n fault > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state # cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh #!/usr/bin/env bash neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --router_id=e7e98ef7-36d5-4544-be07-c83ddf7e67ea --state_path=/var/lib/neutron --metadata_port=9697 --verbose --log-file=neutron-ns-metadata-proxy-e7e98ef7- 36d5-4544-be07-c83ddf7e67ea.log --log-dir=/var/log/neutron echo -n master > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state マスターかバックアップかを stateファイルに書いておく
  21. 21. keepalived (4) ● keepalivedはプロセスが2 個見える – VRRP用に1個 fork(2) する – 親はVRRP用の子供を監視 して、死んでいたら respawnする # pgrep keepalived 2948 5130 # pstree -p -N net [4026531956] systemd(1)─┬─agetty(908) ├─agetty(909) ├─auditd(594)───{auditd}(605) ├─avahi-daemon(624)───avahi-daemon(635) ├─chronyd(674) ├─crond(701) ├─dbus-daemon(680)───{dbus-daemon}(698) ├─httpd(1189)─┬─httpd(1553) │ ├─httpd(1554) │ ├─httpd(1555) │ ├─httpd(1556) │ ├─httpd(1557) │ ├─httpd(1558) │ ├─httpd(1559) │ └─httpd(1560) ├─iprdump(745) ├─iprinit(668) ├─iprupdate(671) ├─irqbalance(644) ├─lvmetad(471) ├─master(1676)─┬─pickup(27341) │ └─qmgr(1690) ├─monitor(704)───ovsdb-server(705) ├─monitor(727)───ovs-vswitchd(729)─┬─{ovs-vswitchd}(732) │ ├─{ovs-vswitchd}(733) │ ├─{ovs-vswitchd}(734) │ └─{ovs-vswitchd}(735) ├─neutron-dhcp-ag(2291) ├─neutron-l3-agen(2290) ├─neutron-metadat(1192)─┬─neutron-metadat(2167) │ └─neutron-metadat(2168) ├─neutron-meterin(1191) ├─neutron-openvsw(2293)───sudo(2378)───neutron-rootwra(2380)───ovsdb-client(2382) ├─nrpe(1198) ├─rsyslogd(626)─┬─{rsyslogd}(646) │ └─{rsyslogd}(647) ├─sshd(1199)───sshd(1842)───bash(1969)───pstree(28359) ├─systemd-journal(470) ├─systemd-logind(679) ├─systemd-udevd(473) └─tuned(642)─┬─{tuned}(774) ├─{tuned}(775) ├─{tuned}(776) └─{tuned}(788) [4026532345] dnsmasq(3834) [4026532262] keepalived(2948)───keepalived(5130) neutron-ns-meta(4043)
  22. 22. マスタールータの確認 ● VIPがついている方がマスター juno-network1 juno-network2 # ip netns list qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea # ip netns list qdhcp-802e69c8-b064-4eb1-90b9-327e712472d8 qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea ip netns exec qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-ef4a5a34-17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:16:91:0f brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-ef4a5a34-17 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe16:910f/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe7a:d916/64 scope link valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever ip netns exec qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-00ab5988-de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7c:52:0d brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-00ab5988-de valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7c:520d/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet 172.16.99.100/24 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet 172.16.99.101/32 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7a:d916/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-3d8666f0-c4 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever [root@juno-network1 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07- c83ddf7e67ea/state; echo backup [ro[root@juno-network2 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07- c83ddf7e67ea/state; echo master ● ここにも書いてある
  23. 23. フェイルオーバー ● 初期状態 – master: juno-network1 – backup: juno-network2 ● keepalivedを殺す [root@juno-network1 ~]# date; killall keepalived Mon Dec 1 12:51:59 JST 2014 [root@juno-controller rdo-juno(keystone_demo)]# nova show vm1 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2014-12-01T03:37:44.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2014-12-01T03:37:01Z | | demo_net network | 192.168.10.100, 172.16.99.101 | | flavor | m1.tiny (1) | | hostId | 826d63fe7c11232775d45dd1edc8bf9db15b01d01ce908c143434750 | | id | da10da5e-6b61-40f7-b5ab-18409b4d4267 | | image | cirros (69d97e8f-24b5-4f0b-8977-7295d8ba8edb) | | key_name | sshkey | | metadata | {} | | name | vm1 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | ACTIVE | | tenant_id | 328ffb467eb24c3595b549516b8acbd0 | | updated | 2014-12-01T03:37:44Z | | user_id | 01c4dc5c8ec0494eaaecb84fd4170556 | +--------------------------------------+----------------------------------------------------------+ [root@juno-controller rdo-juno(keystone_demo)]# nova ssh --login cirros vm1 $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether fa:16:3e:be:02:20 brd ff:ff:ff:ff:ff:ff inet 192.168.10.100/24 brd 192.168.10.255 scope global eth0 inet6 fe80::f816:3eff:febe:220/64 scope link valid_lft forever preferred_lft forever $ ping 172.16.99.11 PING 172.16.99.11 (172.16.99.11): 56 data bytes 64 bytes from 172.16.99.11: seq=0 ttl=63 time=2.659 ms 64 bytes from 172.16.99.11: seq=1 ttl=63 time=0.687 ms 64 bytes from 172.16.99.11: seq=2 ttl=63 time=0.981 ms ^C --- 172.16.99.11 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.687/1.442/2.659 ms $ exit Connection to 172.16.99.101 closed. [root@juno-controller rdo-juno(keystone_demo)]#
  24. 24. フェイルオーバー ● OSのログ juno-network1 12月 01 12:51:59 juno-network1 Keepalived[3668]: Stopping Keepalived v1.2.10 (06/10,2014) 12月 01 12:51:59 juno-network1 Keepalived_vrrp[5025]: VRRP_Instance(VR_1) sending 0 priority juno-network2 12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: VRRP_Instance(VR_1) Transition to MASTER STATE 12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: VRRP_Group(VG_1) Syncing instances to MASTER state 12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: Opening script file /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh 12月 01 12:52:02 juno-network2 Keepalived_vrrp[5038]: VRRP_Instance(VR_1) Entering MASTER STATE
  25. 25. フェイルオーバー ● VMへの疎通 [Mon Dec 1 12:51:52 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=417 ttl=62 time=1.13 ms [Mon Dec 1 12:51:53 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=418 ttl=62 time=1.12 ms [Mon Dec 1 12:51:54 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=419 ttl=62 time=1.41 ms [Mon Dec 1 12:51:55 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=420 ttl=62 time=1.24 ms [Mon Dec 1 12:51:56 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=421 ttl=62 time=0.617 ms [Mon Dec 1 12:51:57 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=422 ttl=62 time=0.855 ms [Mon Dec 1 12:51:58 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=423 ttl=62 time=1.39 ms [Mon Dec 1 12:51:59 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=424 ttl=62 time=1.20 ms [Mon Dec 1 12:52:03 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=428 ttl=62 time=4.30 ms [Mon Dec 1 12:52:04 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=429 ttl=62 time=1.63 ms [Mon Dec 1 12:52:05 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=430 ttl=62 time=1.16 ms [Mon Dec 1 12:52:06 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=431 ttl=62 time=0.952 ms [Mon Dec 1 12:52:07 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=432 ttl=62 time=1.25 ms [Mon Dec 1 12:52:08 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=433 ttl=62 time=0.906 ms [Mon Dec 1 12:52:09 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=434 ttl=62 time=1.26 ms [Mon Dec 1 12:52:10 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=435 ttl=62 time=1.22 ms
  26. 26. フェイルオーバー前 ● ip netns juno-network1 juno-network2 [root@juno-network1 ~]# ip netns exec $(ip netns list | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-ef4a5a34-17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:16:91:0f brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-ef4a5a34-17 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe16:910f/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet 172.16.99.100/24 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet 172.16.99.101/32 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7a:d916/64 scope link valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-3d8666f0-c4 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever [root@juno-network1 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state master [root@juno-network2 ~]# ip netns exec $(ip netns list | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-00ab5988-de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7c:52:0d brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-00ab5988-de valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7c:520d/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe7a:d916/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever [root@juno-network2 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state backup
  27. 27. フェイルオーバー後 ● ip netns juno-network1 juno-network2 [root@juno-network1 ~]# ip netns exec $(ip netns list | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-ef4a5a34-17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:16:91:0f brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-ef4a5a34-17 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe16:910f/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe7a:d916/64 scope link valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever [root@juno-network2 ~]# ip netns exec $(ip netns list | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-00ab5988-de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7c:52:0d brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-00ab5988-de valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7c:520d/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet 172.16.99.100/24 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet 172.16.99.101/32 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7a:d916/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-3d8666f0-c4 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever
  28. 28. フェイルオーバー ● VRRP 12:51:51.991662 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:51:53.992827 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:51:55.993186 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:51:57.994432 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:51:59.674908 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 2s, length 20 12:52:00.481158 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:52:02.483882 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:52:04.484732 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:52:06.485926 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:52:08.485989 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20
  29. 29. フェイルオーバー ● Gratuitous ARP 12:52:02.476000 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476075 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476121 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476133 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476144 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477325 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477376 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477403 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477429 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477439 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:51:59.679351 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679373 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:51:59.679377 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679380 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:51:59.679383 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679385 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:51:59.679388 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679391 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:51:59.679393 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679396 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680231 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680307 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680313 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680315 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680317 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680319 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680321 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680322 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680324 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680326 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28
  30. 30. フェイルオーバー ● network nodes (external) 12:52:02.475961 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476059 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.476114 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476127 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.476138 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476150 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.476155 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476161 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.476175 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476181 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:03.331280 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 428, length 64 12:52:03.334090 ARP, Request who-has 172.16.99.254 tell 172.16.99.101, length 28 12:52:03.334767 ARP, Reply 172.16.99.254 is-at 52:54:00:9e:a3:85 (oui Unknown), length 28 12:52:03.335070 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 428, length 64 12:52:04.332680 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 429, length 64 12:52:04.333804 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 429, length 64 12:52:05.334435 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 430, length 64 12:52:05.335160 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 430, length 64 12:52:06.335625 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 431, length 64 12:52:06.336258 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 431, length 64 12:52:07.335902 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 432, length 64 12:52:07.336705 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 432, length 64 12:52:07.477275 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477364 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.477397 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477424 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.477433 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477446 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.477458 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477462 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.477466 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477475 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:08.337331 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 433, length 64 12:52:08.337924 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 433, length 64 12:52:09.337900 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 434, length 64 12:52:09.338739 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 434, length 64 12:52:10.339405 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 435, length 64 12:52:10.340197 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 435, length 64 juno-network1 12:51:57.328686 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 422, length 64 12:51:57.329325 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 422, length 64 12:51:58.329837 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 423, length 64 12:51:58.330791 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 423, length 64 12:51:59.331426 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 424, length 64 12:51:59.332194 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 424, length 64 12:52:00.332890 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 425, length 64 12:52:01.332965 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 426, length 64 12:52:02.332935 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 427, length 64 12:52:02.478520 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.478549 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.479044 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.479059 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.479062 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.479064 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.479067 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.479069 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.479071 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.479073 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:03.335874 ARP, Request who-has 172.16.99.254 tell 172.16.99.101, length 28 12:52:07.479212 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479867 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.479877 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479879 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.479881 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479882 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.479884 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479885 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.479887 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479889 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 juno-network2
  31. 31. フェイルオーバー ● network nodes (tenant) juno-network1 juno-network2 12:51:59.675765 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s) 12:52:02.476000 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.476075 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.476121 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.476133 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.476144 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:03.331701 ARP, Request who-has 192.168.10.100 tell 192.168.10.1, length 28 12:52:03.332823 ARP, Reply 192.168.10.100 is-at fa:16:3e:be:02:20 (oui Unknown), length 28 12:52:03.333103 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 428, length 64 12:52:03.333832 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 428, length 64 12:52:04.332750 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 429, length 64 12:52:04.333764 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 429, length 64 12:52:04.403080 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s) 12:52:05.334499 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 430, length 64 12:52:05.335128 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 430, length 64 12:52:06.335673 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 431, length 64 12:52:06.336219 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 431, length 64 12:52:07.335970 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 432, length 64 12:52:07.336659 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 432, length 64 12:52:07.477325 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.477376 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.477403 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.477429 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.477439 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:08.337381 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 433, length 64 12:52:08.337903 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 433, length 64 12:52:09.337972 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 434, length 64 12:52:09.338713 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 434, length 64 12:52:10.339497 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 435, length 64 12:52:10.340170 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 435, length 64 12:51:57.329293 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 422, length 64 12:51:58.329928 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 423, length 64 12:51:58.330740 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 423, length 64 12:51:59.331478 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 424, length 64 12:51:59.332152 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 424, length 64 12:51:59.677264 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s) 12:52:02.478590 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.478605 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.478625 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.478631 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.478636 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:03.333519 ARP, Request who-has 192.168.10.100 tell 192.168.10.1, length 28 12:52:04.404617 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s) 12:52:07.479383 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.479396 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.479401 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.479404 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.479408 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28
  32. 32. 素朴な疑問 (1) ● Q: non-HA構成で作ったルータをHA構成に変更 できる? ● A: できませんでした # neutron router-create --tenant-id $(keystone tenant-list | awk '/demo/ {print $2}') --ha False router_demo Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | distributed | False | | external_gateway_info | | | ha | False | | id | f747119c-6183-4ae4-b6a5-397628a3cc4a | | name | router_demo | | routes | | | status | ACTIVE | | tenant_id | 328ffb467eb24c3595b549516b8acbd0 | +-----------------------+--------------------------------------+ # neutron router-list +--------------------------------------+-------------+-----------------------+-------------+-------+ | id | name | external_gateway_info | distributed | ha | +--------------------------------------+-------------+-----------------------+-------------+-------+ | f747119c-6183-4ae4-b6a5-397628a3cc4a | router_demo | null | False | False | +--------------------------------------+-------------+-----------------------+-------------+-------+ [root@juno-controller rdo-juno(keystone_admin)]# neutron router-update router_demo --ha True Bad Request (HTTP 400) (Request-ID: req-f7f3a8db-cb4e-4f2e-b3e1-ab99809fb15c) 2014-11-30 23:19:41.715 3526 INFO neutron.wsgi [-] (3526) accepted ('10.0.1.111', 54480) 2014-11-30 23:19:41.776 3526 INFO neutron.wsgi [req-f2181436-4a6c-4ab8-b63d-f12c5770d6f2 None] 10.0.1.111 - - [30/Nov/2014 23:19:41] "GET /v2.0/routers.json?fields=id&name=router_demo HTTP/1.1" 200 250 0.059557 2014-11-30 23:19:41.785 3526 ERROR neutron.api.v2.resource [req-f7f3a8db-cb4e-4f2e-b3e1-ab99809fb15c None] update failed 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource Traceback (most recent call last): 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 87, in resource 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource result = method(request=request, **args) 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 501, in update 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource allow_bulk=self._allow_bulk) 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 623, in prepare_request_body 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource raise webob.exc.HTTPBadRequest(msg) 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute ha 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource
  33. 33. 素朴な疑問 (2) ● Q: マスタールータはどっち? ● A: VIPがついている方 ● でもたまに動きがあやしい... ● この辺りから総合的に判断するのが確実 – VIPがついている – /var/lib/neutron/ROUTER_ID/stateに "master" と書かれて いる – keepalivedのプロセスが見える
  34. 34. 素朴な疑問 ● Q: セッションの引き継ぎ? ● A: 今後conntrackdと組み合わせてやる予定 – Wikiにはconntrackdの設定テンプレートとかが 載っていていかにもできそうですが、まだできませ ん ● https://wiki.openstack.org/wiki/Neutron/L3_High_Avai lability_VRRP – Specには "Phase 2でやるよ" と書いてあります ● https://github.com/openstack/neutron-specs/blob/maste r/specs/juno/l3-high-availability.rst
  35. 35. 素朴な疑問 ● HAルータを2個作ると? [root@juno-network1 ~]# for ns in $(ip netns list); do echo "=> ${ns}"; ip netns exec ${ns} ip a; done => qrouter-767318c2-8dfa-4713-9018-27ef072383dd 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 12: ha-12860419-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:65:92:ed brd ff:ff:ff:ff:ff:ff inet 169.254.192.4/18 brd 169.254.255.255 scope global ha-12860419-7f valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe65:92ed/64 scope link valid_lft forever preferred_lft forever 13: qg-6214b83a-d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:bc:8d:2f brd ff:ff:ff:ff:ff:ff inet 172.16.99.101/24 scope global qg-6214b83a-d7 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:febc:8d2f/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 14: qr-56c2e065-77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:66:8d:63 brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-56c2e065-77 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe66:8d63/64 scope link valid_lft forever preferred_lft forever => qrouter-b0402579-0a6e-42c3-8d3b-33af9a51d3cf 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-51456562-ca: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7b:10:1d brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-51456562-ca valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7b:101d/64 scope link valid_lft forever preferred_lft forever 10: qg-5f3287c0-e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:e7:7d:5c brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fee7:7d5c/64 scope link valid_lft forever preferred_lft forever 11: qr-48ea3761-b0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:ab:52:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:feab:52f0/64 scope link valid_lft forever preferred_lft forever [root@juno-network2 ~]# for ns in $(ip netns list); do echo "=> ${ns}"; ip netns exec ${ns} ip a; done => qrouter-767318c2-8dfa-4713-9018-27ef072383dd 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 12: ha-fe1f12b3-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:ae:78:91 brd ff:ff:ff:ff:ff:ff inet 169.254.192.3/18 brd 169.254.255.255 scope global ha-fe1f12b3-8c valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feae:7891/64 scope link valid_lft forever preferred_lft forever 13: qg-6214b83a-d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:bc:8d:2f brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:febc:8d2f/64 scope link valid_lft forever preferred_lft forever 14: qr-56c2e065-77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:66:8d:63 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe66:8d63/64 scope link valid_lft forever preferred_lft forever => qrouter-b0402579-0a6e-42c3-8d3b-33af9a51d3cf 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-1dff8969-fc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:db:a4:68 brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-1dff8969-fc valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fedb:a468/64 scope link valid_lft forever preferred_lft forever 10: qg-5f3287c0-e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:e7:7d:5c brd ff:ff:ff:ff:ff:ff inet 172.16.99.100/24 scope global qg-5f3287c0-e0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fee7:7d5c/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 11: qr-48ea3761-b0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:ab:52:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-48ea3761-b0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feab:52f0/64 scope link valid_lft forever preferred_lft forever
  36. 36. Packstackのハマりどころ ● Cinder – Bug 1164359 - KeyError: 'CONFIG_CINDER_VOLUMES_SIZE' ● https://bugzilla.redhat.com/show_bug.cgi?id=1164359 – Fix LVM Cinder Volume Creation ● https://review.openstack.org/#/c/131803/ – Avoid KeyError: 'CONFIG_CINDER_VOLUMES_SIZE' error ● https://review.openstack.org/#/c/136766/ ● mongodb – Bug 1167888 - [CENTOS7] packstack all in one install fails at MongoDB Service[mongodb]: Execution of '/usr/bin/systemctl start mongod' returned 1: Job for mongod.service failed ● https://bugzilla.redhat.com/show_bug.cgi?id=1167888 – mongodb.pp error when installing RDO on CentOS 7 ● https://ask.openstack.org/en/question/54015/mongodbpp-error-when-installing-rdo -on-centos-7/
  37. 37. SELinux問題 ● SELinux=enforcingの場合... – HA Routerを起動するとエラーになります 2014-11-29 15:28:11.800 2269 ERROR neutron.agent.l3_agent [-] Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter- 002a56ba-beda-43ad-a5d6-adff4dc1a725', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/002a56ba-beda- 43ad-a5d6-adff4dc1a725/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6- adff4dc1a725.pid', '-r', '/var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6-adff4dc1a725.pid-vrrp'] Exit code: 99 Stdout: '' Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip netns exec qrouter-002a56ba-beda-43ad-a5d6- adff4dc1a725 keepalived -P -f /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6- adff4dc1a725/keepalived.conf -p /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6-adff4dc1a725.pid -r /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6adff4dc1a725.pid-vrrp (no filter matched)n' "no filter matched" と言われている → 一見rootwrap関連の不具合にみえる → だが実は...
  38. 38. SELinux問題 ● SELinux関連が原因だったのでした – keepalivedの起動に失敗している – permissiveにすると問題なし # ausearch -m avc ---- time->Sat Nov 29 15:28:11 2014 type=SYSCALL msg=audit(1417242491.790:1992): arch=c000003e syscall=21 success=no exit=-13 a0=1b17a00 a1=1 a2=7fc4466fcf88 a3=7fff3728eb20 items=0 ppid=3390 pid=3391 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="neutron-rootwra" exe="/usr/bin/python2.7" subj=system_u:system_r:neutron_t:s0 key=(null) type=AVC msg=audit(1417242491.790:1992): avc: denied { execute } for pid=3391 comm="neutron-rootwra" name="keepalived" dev="dm-1" ino=9038320 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:keepalived_exec_t:s0 tclass=file (snip) avc: access vector cache
  39. 39. 参考文献 ● Blueprint – https://blueprints.launchpad.net/neutron/+spec/l3-high-availability ● Spec – https://github.com/openstack/neutron-specs/blob/master/specs/juno/l3-high-availability .rst ● Neutron/L3 High Availability VRRP – https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP ● Atlanta Summit: High Availability in Neutron – Getting the L3 Agent Right – https://www.youtube.com/watch?v=8kBPlIBq0Dc – http://techs.enovance.com/wp-content/uploads/2014/05/Neutron-L3-HA-Summit-eNova nce-template.pdf ● Paris Summit: Neutron Network Node High Availability – https://www.youtube.com/watch?v=go4fOYOUkmE – https://assafmuller.files.wordpress.com/2014/11/l3-ha.pdf ● Layer 3 High Availability – http://assafmuller.com/2014/08/16/layer-3-high-availability/ ● Keepalived Documentation – http://www.keepalived.org/documentation.html

×