Your SlideShare is downloading. ×
CloudStack_usersgroup_19_nagoya_nakaya_20140704_pub
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

CloudStack_usersgroup_19_nagoya_nakaya_20140704_pub

1,225

Published on

Published in: Software, Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,225
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
18
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Apache CloudStack 4.3 Virtual Router Deep Drive 2014/7/4 第19回 CloudStackユーザー会 in 名古屋 Satoru Nakaya(@giraffeforestg)____ http://giraffeforestg.blog.fc2.com/
  • 2. 自己紹介 × 中谷 悟 / Satoru Nakaya × 岐阜県在住 × 大学情報基盤 や オープンソースクラウド を担当 × 自宅SAN友の会 × VMware Certified Advanced Professional × CCA for Citrix XenServer 本日はよろしくお願いします。 2
  • 3. お約束 × 本資料の情報を利⽤することによって⽣じるいかなる損害についても責 任を負うものではありません。 × 発⾔は個⼈の⾒解であり所属する組織の公式⾒解ではありません。 3
  • 4. 注意 今回,発表時間25分に対して スライド111枚となっております。 途中で時間切れ可能性大です。 または超早口になります。 どうぞご了承ください。 ※1スライド 10秒程度ならば最後までいける… 4
  • 5. Virtual Router VR 仮想ルーター ソフトウェアルーター 5
  • 6. 1.概要 2.内部構造 3.性能 4.カスタマイズ 5.新機能 6
  • 7. 7 1.概要 2.内部構造 3.性能 4.カスタマイズ 5.新機能
  • 8. CloudStack UI/Virtual Router 8
  • 9. Virtual Router/Virtual Machine 9
  • 10. Virtual Router/Virtual Machine 10 Hypervisor Hypervisor CloudStack Management Server Storage Virtual Router
  • 11. User VM Instance Guest Network Public Network Virtual Router Advanced Network 11
  • 12. User VM Instance Guest Network Virtual Router Basic Network 12
  • 13. Network Service 13
  • 14. Network Offering 14
  • 15. Network Offering 15
  • 16. Network Offering 16
  • 17. User VM Instance Guest Network Public Network Virtual Router DHCP / DNS / Firewall / NAT / Load Balancer … Network Service 17
  • 18. External devices as network service providers 18
  • 19. User VM Instance External devices as network service providers NetScaler/F5 Load Balancer JuniperSRX Firewall/NAT Virtual Router DHCP/DNS 19
  • 20. External devices as network service providers 20
  • 21. System Offering 21
  • 22. System Offering 22
  • 23. Virtual Router Scale up (CPU:500Mhz→2000MHz, Mem:128MB→2048MB, Net:100Mbps→10000Mbps) Scale up 23
  • 24. Virtual Router High Availability (VRRP) MasterBackup High Availability 24
  • 25. VR VPC(Virtual Private Cloud) 25 Network1 (Web) Network2 (AP) Network3 (DB) 外部ネットワーク 他データセンター Site to Site VPN (IPSEC) VLAN Routing Static Route
  • 26. 26 1.概要 2.内部構造 3.性能 4.カスタマイズ 5.新機能
  • 27. Virtual Router/SSH Login 27
  • 28. Hypervisor(XenServer) から # ssh -i /root/.ssh/id_rsa.cloud リンクローカルアドレス -p 3922 Linux r-45-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Wed Jan 15 00:27:48 2014 from 10.0.2.2 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. root@r-45-VM:~# 28
  • 29. Network Interface # ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 02:00:40:e3:00:02 brd ff:ff:ff:ff:ff:ff inet 10.1.1.1/24 brd 10.1.1.255 scope global eth0 inet6 fe80::40ff:fee3:2/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 0e:00:a9:fe:03:18 brd ff:ff:ff:ff:ff:ff inet 169.254.3.24/16 brd 169.254.255.255 scope global eth1 inet6 fe80::c00:a9ff:fefe:318/64 scope link valid_lft forever preferred_lft forever 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 06:66:44:00:00:18 brd ff:ff:ff:ff:ff:ff inet 192.168.11.130/24 brd 192.168.11.255 scope global eth2 inet6 fe80::466:44ff:fe00:18/64 scope link valid_lft forever preferred_lft forever Guest Network Link Local Public Network 29
  • 30. Routing Table # ip route show default via 192.168.11.254 dev eth2 10.1.1.0/24 dev eth0 proto kernel scope link src 10.1.1.1 169.254.0.0/16 dev eth1 proto kernel scope link src 169.254.3.24 192.168.11.0/24 dev eth2 proto kernel scope link src 192.168.11.130 30
  • 31. Firewall 31
  • 32. Firewall # iptables -nL -v Chain INPUT (policy DROP 443 packets, 29549 bytes) pkts bytes target prot opt in out source destination 1880 159K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * * 0.0.0.0/0 224.0.0.18 0 0 ACCEPT all -- * * 0.0.0.0/0 225.0.0.50 0 0 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 584 68556 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 692 52648 ACCEPT all -- eth2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 16 1344 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 1 576 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 216 14234 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 19 1140 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3922 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80 0 0 ACCEPT tcp -- eth0 * 10.1.1.0/24 0.0.0.0/0 state NEW tcp dpt:8080 Chain FORWARD (policy DROP 276 packets, 16560 bytes) pkts bytes target prot opt in out source destination 276 16560 NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state NEW 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 276 16560 FW_OUTBOUND all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT 1338 packets, 176K bytes) pkts bytes target prot opt in out source destination 1379 183K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 Chain FW_OUTBOUND (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED Chain NETWORK_STATS (3 references) pkts bytes target prot opt in out source destination 276 16560 all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 224 35166 tcp -- !eth0 eth2 0.0.0.0/0 0.0.0.0/0 224 11648 tcp -- eth2 !eth0 0.0.0.0/0 0.0.0.0/0 32
  • 33. Firewall iptables について少しお勉強 テーブル ・filter パケットの通過/遮断 ・nat アドレス変換 ・mangle パケットのフィールドを変換(TOS等) ・row 33
  • 34. User VM Instance Guest Network Public Network SourceNAT 34
  • 35. SourceNAT # iptables -nL -v -t nat Chain PREROUTING (policy ACCEPT 1847 packets, 148K bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 400 packets, 26564 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 78 packets, 5577 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 1 packets, 576 bytes) pkts bytes target prot opt in out source destination 77 5001 SNAT all -- * eth2 0.0.0.0/0 0.0.0.0/0 to:192.168.11.130 35
  • 36. User VM Instance Guest Network Public Network StaticNAT 36
  • 37. StaticNAT 37
  • 38. StaticNAT # iptables –nL -v -t nat Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes) pkts bytes target prot opt in out source destination 0 0 DNAT all -- eth2 * 0.0.0.0/0 192.168.11.131 to:10.1.1.236 0 0 DNAT all -- eth0 * 0.0.0.0/0 192.168.11.131 to:10.1.1.236 Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 SNAT all -- * eth2 10.1.1.236 0.0.0.0/0 to:192.168.11.131 0 0 SNAT all -- * eth2 0.0.0.0/0 0.0.0.0/0 to:192.168.11.130 0 0 SNAT all -- * eth0 10.1.1.0/24 10.1.1.236 to:10.1.1.1 38
  • 39. User VM Instance Guest Network Public Network Firewall 39
  • 40. User VM Instance Guest Network Public Network Firewall(Ingress rules) 40
  • 41. Firewall(Ingress rules) 41
  • 42. Firewall(Ingress rules) ルール追加前 # iptables -nL -v -t mangle Chain PREROUTING (policy ACCEPT 164 packets, 19188 bytes) pkts bytes target prot opt in out source destination 87 6751 VPN_192.168.11.130 all -- * * 0.0.0.0/0 192.168.11.130 0 0 FIREWALL_192.168.11.130 all -- * * 0.0.0.0/0 192.168.11.130 289 33156 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED CONNMARK restore Chain INPUT (policy ACCEPT 210 packets, 23683 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 18 packets, 1080 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 187 packets, 26084 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 187 packets, 26084 bytes) pkts bytes target prot opt in out source destination 0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68 CHECKSUM fill Chain FIREWALL_192.168.11.130 (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 Chain VPN_192.168.11.130 (1 references) pkts bytes target prot opt in out source destination 87 6751 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 42
  • 43. Firewall(Ingress rules) ルール追加後 # iptables -nL -v -t mangle Chain PREROUTING (policy ACCEPT 12 packets, 856 bytes) pkts bytes target prot opt in out source destination 251 17836 VPN_192.168.11.130 all -- * * 0.0.0.0/0 192.168.11.130 0 0 FIREWALL_192.168.11.130 all -- * * 0.0.0.0/0 192.168.11.130 402 46776 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED CONNMARK restore Chain INPUT (policy ACCEPT 13 packets, 900 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 1 packets, 60 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 9 packets, 1016 bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 9 packets, 1016 bytes) pkts bytes target prot opt in out source destination 0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68 CHECKSUM fill Chain FIREWALL_192.168.11.130 (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 RETURN tcp -- * * 172.20.0.0/16 0.0.0.0/0 tcp dpt:20000 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 Chain VPN_192.168.11.130 (1 references) pkts bytes target prot opt in out source destination 251 17836 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 43
  • 44. User VM Instance Guest Network Public Network Firewall(Egress rules) 44
  • 45. Firewall(Egress rules) ルール追加前 # iptables -nL -v -t filter Chain INPUT (policy DROP 443 packets, 29549 bytes) pkts bytes target prot opt in out source destination 1880 159K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * * 0.0.0.0/0 224.0.0.18 0 0 ACCEPT all -- * * 0.0.0.0/0 225.0.0.50 0 0 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 584 68556 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 692 52648 ACCEPT all -- eth2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 16 1344 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 1 576 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 216 14234 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 19 1140 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3922 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80 0 0 ACCEPT tcp -- eth0 * 10.1.1.0/24 0.0.0.0/0 state NEW tcp dpt:8080 Chain FORWARD (policy DROP 276 packets, 16560 bytes) pkts bytes target prot opt in out source destination 276 16560 NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state NEW 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 276 16560 FW_OUTBOUND all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 45
  • 46. Firewall(Egress rules) ルール追加前 # iptables -nL -v -t filter Chain OUTPUT (policy ACCEPT 1338 packets, 176K bytes) pkts bytes target prot opt in out source destination 1379 183K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 Chain FW_OUTBOUND (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED Chain NETWORK_STATS (3 references) pkts bytes target prot opt in out source destination 276 16560 all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 224 35166 tcp -- !eth0 eth2 0.0.0.0/0 0.0.0.0/0 224 11648 tcp -- eth2 !eth0 0.0.0.0/0 0.0.0.0/0 46
  • 47. Firewall(Egress rules) ルール追加後 # iptables -nL -v -t filter Chain INPUT (policy DROP 6 packets, 280 bytes) pkts bytes target prot opt in out source destination 1496 113K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * * 0.0.0.0/0 224.0.0.18 0 0 ACCEPT all -- * * 0.0.0.0/0 225.0.0.50 0 0 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 802 70672 ACCEPT all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 487 34100 ACCEPT all -- eth2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 11 924 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 127 8334 ACCEPT udp -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 15 900 ACCEPT tcp -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3922 0 0 ACCEPT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80 0 0 ACCEPT tcp -- eth0 * 10.1.1.0/24 0.0.0.0/0 state NEW tcp dpt:8080 Chain FORWARD (policy DROP 5 packets, 300 bytes) pkts bytes target prot opt in out source destination 174 10440 NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- eth0 eth1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state NEW 0 0 ACCEPT all -- eth0 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 0 ACCEPT all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 174 10440 FW_OUTBOUND all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 47
  • 48. Firewall(Egress rules) ルール追加後 # iptables -nL -v -t filter Chain OUTPUT (policy ACCEPT 432 packets, 365K bytes) pkts bytes target prot opt in out source destination 1587 831K NETWORK_STATS all -- * * 0.0.0.0/0 0.0.0.0/0 Chain FW_EGRESS_RULES (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT tcp -- * * 10.1.1.100 0.0.0.0/0 tcp dpt:11111 Chain FW_OUTBOUND (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 5 300 FW_EGRESS_RULES all -- * * 0.0.0.0/0 0.0.0.0/0 Chain NETWORK_STATS (3 references) pkts bytes target prot opt in out source destination 174 10440 all -- eth0 eth2 0.0.0.0/0 0.0.0.0/0 0 0 all -- eth2 eth0 0.0.0.0/0 0.0.0.0/0 230 36664 tcp -- !eth0 eth2 0.0.0.0/0 0.0.0.0/0 230 11960 tcp -- eth2 !eth0 0.0.0.0/0 0.0.0.0/0 48
  • 49. DNS/DHCP dnsmasq 軽量なDNSサーバ(フォワーダ/キャッシュ)とDHCPサーバ http://www.thekelleys.org.uk/dnsmasq/doc.html 49
  • 50. DNS/DHCP # grep -v -e '#' -e '^$' /etc/dnsmasq.conf domain-needed bogus-priv resolv-file=/etc/dnsmasq-resolv.conf local=/cs2cloud.internal/ interface=eth0 except-interface=eth1 except-interface=eth2 except-interface=lo listen-address=10.1.1.1 no-dhcp-interface=eth1 no-dhcp-interface=eth2 bind-interfaces expand-hosts domain=cs2cloud.internal domain=cs2cloud.internal domain=cs2cloud.internal dhcp-range=10.1.1.1,static dhcp-hostsfile=/etc/dhcphosts.txt dhcp-option=15,"cs2cloud.internal" dhcp-option=vendor:MSFT,2,1i dhcp-lease-max=2100 domain=cs2cloud.internal log-facility=/var/log/dnsmasq.log conf-dir=/etc/dnsmasq.d dhcp-option=option:router,10.1.1.1 dhcp-option=6,10.1.1.1,172.16.0.38 dhcp-client-update dhcp-optsfile=/etc/dhcpopts.txt 50
  • 51. DNS/DHCP # cat /etc/dhcphosts.txt 02:00:7b:ed:00:01,set:10_1_1_236,10.1.1.236,vm1,infinite 02:00:62:62:00:03,set:10_1_1_162,10.1.1.162,vm2,infinite # cat /etc/dnsmasq-resolv.conf nameserver 172.16.0.38 51
  • 52. Load balancer HAProxy http://haproxy.1wt.eu/ 52
  • 53. Load balancer # cat /etc/haproxy/haproxy.cfg global log 127.0.0.1:3914 local0 info chroot /var/lib/haproxy user haproxy group haproxy daemon defaults log global mode tcp option dontlognull retries 3 option redispatch option forwardfor stats enable stats uri /admin?stats stats realm Haproxy¥ Statistics stats auth admin1:AdMiN123 option forceclose timeout connect 5000 timeout client 50000 timeout server 50000 listen cloud-default 0.0.0.0:35999 option transparent/ 53
  • 54. Load balancer 54
  • 55. Load balancer # cat /etc/haproxy/haproxy.cfg global log 127.0.0.1:3914 local0 info chroot /var/lib/haproxy user haproxy group haproxy daemon : listen cloud-default 0.0.0.0:35999 option transparent/ listen 192_168_11_132-80 192.168.11.132:80 balance roundrobin server 192_168_11_132-80_0 10.1.1.236:80 check server 192_168_11_132-80_1 10.1.1.162:80 check 55
  • 56. Correspondence table of network services and applications Network Services Applications description Firewall iptables administration tools for packet filtering and NATSource NAT Static NAT Port Forwording DHCP dnsmasq Small caching DNS proxy and DHCP/TFTP serverDNS User Data apache Apache HTTP Server Load Balancer haproxy fast and reliable load balancing reverse proxy VPN xl2tpd layer 2 tunneling protocol implementation openswan Internet Key Exchange daemon Redundant Router conntrackd Connection tracking daemon keepalived Failover and monitoring daemon for LVS clusters 56
  • 57. 設定反映スクリプト CloudStack Database格納情報 (ファイアウォールルールや払い出したIP等)を元に SSHで(CloudStack Management Server → XenServer → VR) という経路でスクリプトがキックされ仮想ルータの各種設定を実⾏。 ls /root bumpup_priority.sh firewallRule_egress.sh monitorServices.py clearUsageRules.sh firewall_rule.sh reconfigLB.sh createIpAlias.sh firewall.sh redundant_router deleteIpAlias.sh func.sh userdata.py dnsmasq.sh hv-kvp-daemon_3.1_amd64.deb userdata.sh edithosts.sh loadbalancer.sh 57
  • 58. 1.概要 2.内部構造 3.性能 4.カスタマイズ 5.新機能 58
  • 59. 59 <注意!> ざっくり計測しているので参考程度にみてね。 私の環境では,こうなったというぐらい。 <caution!> I have measured roughly. Performance will vary depending on the environment.
  • 60. Network Performance 60 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit Hardware specification CPU:AMD Opteron Processor 3250 HE 2.5GHz 4コア Memory:6GB HDD:7200rpm SATA NIC:Broadcom BCM57780 1G Virtual Router System Offerings CPU:500MHz,Memory:128MB,Nwtork Limit:10Gbps Network performance measurement tool:nuttcp-6.1.2 L2スイッチ(1G) Catalyst 3550-12G CentOS6.4 64bit systemvm64template- 2014-01-14- master-xen.vhd
  • 61. Network Performance(P→P) 61 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) 測定結果はユーザ会でのみ公開
  • 62. Network Performance(P→VM) 62 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) 測定結果はユーザ会でのみ公開
  • 63. Network Performance(P→VR→VM) 63 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) StaticNAT 測定結果はユーザ会でのみ公開
  • 64. Network Performance(P→VR→VM) 64 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) Port Transfer 測定結果はユーザ会でのみ公開
  • 65. Network Performance(VM→VR→P) 65 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) SourceNAT 測定結果はユーザ会でのみ公開
  • 66. Network Performance(P→VR) 66 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) # service iptables-persistent flush # apt-get install nuttcp # nuttcp -S System Offerings CPU:2000 MHz Memory:2048 MB Network Limit: 10000Mbps 測定結果はユーザ会でのみ公開
  • 67. Network Performance(VM→VM) 67 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) 測定結果はユーザ会でのみ公開
  • 68. Network Performance(VM→VR→VR→VM) 68 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) StaticNAT SourceNAT 測定結果はユーザ会でのみ公開
  • 69. 69 CentOS 6.4 64bit
  • 70. Network Performance(P→VM→VM) 70 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) # ip addr add 192.168.11.245/24 brd 192.168.11.255 dev eth0 # iptables -t nat -A PREROUTING -d 192.168.11.245 -j DNAT --to 10.1.1.101 # iptables -t nat -A POSTROUTING -s 10.1.1.101 -j SNAT --to 192.168.11.245 CentOS6.4 64bit StaticNAT 測定結果はユーザ会でのみ公開
  • 71. 71 vyos-1.0.4-i586-virt
  • 72. Config 72 vyos@vyos:~$ show configuration commands set interfaces ethernet eth0 address '192.168.11.244/24' set interfaces ethernet eth0 address '192.168.11.243/24' set interfaces ethernet eth0 hw-id '72:b4:04:56:a8:4e' set interfaces ethernet eth1 address '10.1.1.70/24' set interfaces ethernet eth1 hw-id '96:58:b1:13:51:40' set interfaces loopback 'lo' set nat destination rule 10 destination address '192.168.11.243' set nat destination rule 10 inbound-interface 'eth0' set nat destination rule 10 protocol 'all' set nat destination rule 10 translation address '10.1.1.101' set nat source rule 10 outbound-interface 'eth0' set nat source rule 10 protocol 'all' set nat source rule 10 source address '10.1.1.101' set nat source rule 10 translation address '192.168.11.243' set service ssh listen-address '192.168.11.244' :
  • 73. Network Performance(P→VM→VM) 73 Hypervisor XenServer 6.2SP1 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) Vyos-1.0.4 StaticNAT 測定結果はユーザ会でのみ公開
  • 74. 74 Juniper Networks JUNOS 12.1X46-D10.2 +VMware vSphere
  • 75. Network Performance 75 Hypervisor XenServer 6.2SP1 VMware ESX5 CentOS6.4 64bit L2スイッチ(1G) Catalyst 3550-12G Hardware specification CPU:AMD Opteron Processor 3250 HE 2.5GHz 4コア Memory:6GB HDD:7200rpm SATA NIC:Broadcom BCM57780 1G Hardware specification (VMware vSphere 5) CPU:Intel Xeon X5260 3.33GHz 2コア Memory:16GB HDD:15000rpm SAS NIC:Broadcom BCM5708 1G
  • 76. Network Performance(P→VM→VM) 76 Hypervisor XenServer 6.2SP1 VMware ESX5 CentOS6.4 64bit L2スイッチ(1G) StaticNAT Juiper Firefly 測定結果はユーザ会でのみ公開
  • 77. 77 CentOS 6.4 64bit +VMware vSphere
  • 78. Network Performance(P→VM→VM) 78 Hypervisor XenServer 6.2SP1 VMware ESX5 CentOS6.4 64bit L2スイッチ(1G) StaticNAT CentOS6.4 64bit 測定結果はユーザ会でのみ公開
  • 79. Network Performance(P→VM→VM) 79 Hypervisor VMware ESX5 CentOS6.4 64bit CentOS6.4 64bit L2スイッチ(1G) CentOS6.4 64bit StaticNAT 測定結果はユーザ会でのみ公開
  • 80. Network Performance まとめ 80 個⼈の所有物(自宅)でハードの条件が同一ではない為, 非常にざっくりとした推測ですが ・仮想ルーターが動作する物理サーバには高速CPUが必要 →CPU 2.2GHz と CPU 3.33GHz でDNAT処理において差がでた。 ・仮想ルーターが動作する物理サーバ/プライマリストレージには 高速ディスクが必要 →SATA 7200 rpm と SAS 15000rpm でDNAT処理において差がでた。 ・スペックが高い物理サーバを⽤意すれば仮想ルーター(ソフト処理) でも1Gレベルであれば理論値に近いスループットを出せる。 ※10Gではどうなるのか? ・CloudStackの仮想ルーターはVyOSやFireflyと性能差はなく優秀。 ・同一サーバの仮想ネットワーク内通信は高速。 ・ハイパーバイザの性能差は? ・今回,準仮想化のドライバは不使⽤。
  • 81. 81 1.概要 2.内部構造 3.性能 4.カスタマイズ 5.新機能
  • 82. 仮想ルータの作成・起動の流れ 1)SystemVMのテンプレートから仮想ルータを作成 テンプレートはセカンダリストレージに格納されている。 2)パッチファイル:systemvm.isoを仮想ルータに適応 /etc/init.d/cloud-early-config 3)スクリプトで各種設定反映 82
  • 83. 作成後の仮想ルータの設定を直接編集すると? ・仮想ルータの起動時に ファイアウォールやDHCPの設定ファイル等の CloudStackが管理しているファイルはリセットされ再設定される。 すべての設定がリセットされるわけではない。 なお,インストールしたアプリケーションは維持される。 ・仮想ルータ再作成(削除→作成)時は変更した情報は失われる。 ・仮想ルータ作成毎に毎回再設定するのは面倒。 83
  • 84. 仮想ルータの作成・起動の流れ 1)SystemVMのテンプレートから仮想ルータを作成 テンプレートはセカンダリストレージに格納されている。 2)パッチファイル:systemvm.isoを仮想ルータに適応 /etc/init.d/cloud-early-config 3)スクリプトで各種設定反映 上記の1)か2)で変更を加える。 84
  • 85. ※Failed to import となる場合はDHCPでIPが取得できないのが原因。 importウィザードのIP設定でIPを固定すればOK SystemVMテンプレート SystemVMのテンプレートを XenCenter等でインポートし変更を加える。 ※CloudStackの管理対象外のプール/ホストで実施しな いとCloudStackにインポートしたVMを削除されるので 注意。 85
  • 86. SystemVMテンプレート 86
  • 87. SystemVMテンプレート SystemVMのテンプレート自体のカスタマイズは 大がかりなので, 今回は⽐較的容易に実施できるsystemvm.isoのカ スタマイズを紹介。 87
  • 88. systemvm.iso ファイルはどこにあるのか? 1) CloudStack Management Server /usr/share/cloudstack-common/vms/systemvm.iso 2) XenServer(CloudStack Management Serverから配布される) /opt/xensource/packages/iso/systemvm.iso ISOファイルの中身を確認する。 mkdir /tmp/1 mount -t iso9660 -o loop /usr/share/cloudstack-common/vms/systemvm.iso /tmp/1 ls /tmp/1 authorized_keys cloud-scripts.tgz systemvm.zip 88
  • 89. systemvm.iso/cloud-scripts.tgz etc/ etc/apache2/ etc/apache2/sites-available/ etc/cron.daily/ etc/default/ etc/haproxy/ etc/init.d/ etc/ipsec.d/ etc/iptables/ etc/modprobe.d/ etc/ppp/ etc/profile.d/ etc/ssh/ etc/xl2tpd/ opt/ opt/cloud/ opt/cloud/bin/ root/ root/.ssh/ root/redundant_router/ usr/ usr/sbin/ var/ var/www/ var/www/html/ var/www/html/latest/ var/www/html/userdata/ etc/apache2/httpd.conf etc/apache2/ports.conf etc/apache2/sites-available/default etc/apache2/sites-available/default-ssl etc/apache2/vhostexample.conf etc/cloud-nic.rules etc/cron.daily/cloud-cleanup etc/default/cloud etc/default/cloud-passwd-srvr etc/dnsmasq.conf.tmpl etc/haproxy/haproxy.cfg etc/init.d/cloud etc/init.d/cloud-early-config etc/init.d/cloud-passwd-srvr etc/init.d/postinit etc/ipsec.conf etc/ipsec.d/l2tp.conf etc/ipsec.secrets 89
  • 90. systemvm.iso/cloud-scripts.tgz etc/iptables/iptables-consoleproxy etc/iptables/iptables-elbvm etc/iptables/iptables-ilbvm etc/iptables/iptables-router etc/iptables/iptables-secstorage etc/iptables/iptables-vpcrouter etc/iptables/rt_tables_init etc/iptables/rules etc/logrotate.d/apache2 etc/logrotate.d/cloud etc/logrotate.d/dnsmasq etc/logrotate.d/haproxy etc/logrotate.d/ppp etc/logrotate.d/rsyslog etc/modprobe.d/aesni_intel etc/ppp/options.xl2tpd etc/profile.d/cloud.sh etc/rc.local etc/rsyslog.conf etc/ssh/sshd_config etc/sysctl.conf etc/vpcdnsmasq.conf etc/xl2tpd/xl2tpd.conf opt/cloud/bin/checkbatchs2svpn.sh opt/cloud/bin/checks2svpn.sh opt/cloud/bin/cloud-nic.sh opt/cloud/bin/get_template_version.sh opt/cloud/bin/ilb.sh opt/cloud/bin/ipassoc.sh opt/cloud/bin/ipsectunnel.sh opt/cloud/bin/monitor_service.sh opt/cloud/bin/netusage.sh opt/cloud/bin/passwd_server opt/cloud/bin/passwd_server_ip opt/cloud/bin/patchsystemvm.sh opt/cloud/bin/savepassword.sh opt/cloud/bin/serve_password.sh opt/cloud/bin/vmdata.py 90
  • 91. systemvm.iso/cloud-scripts.tgz opt/cloud/bin/vpc_acl.sh opt/cloud/bin/vpc_func.sh opt/cloud/bin/vpc_guestnw.sh opt/cloud/bin/vpc_ipassoc.sh opt/cloud/bin/vpc_loadbalancer.sh opt/cloud/bin/vpc_netusage.sh opt/cloud/bin/vpc_passwd_server opt/cloud/bin/vpc_portforwarding.sh opt/cloud/bin/vpc_privateGateway.sh opt/cloud/bin/vpc_privategw_acl.sh opt/cloud/bin/vpc_snat.sh opt/cloud/bin/vpc_staticnat.sh opt/cloud/bin/vpc_staticroute.sh opt/cloud/bin/vpn_l2tp.sh root/.ssh/authorized_keys root/bumpup_priority.sh root/clearUsageRules.sh root/createIpAlias.sh root/deleteIpAlias.sh root/dnsmasq.sh root/edithosts.sh root/firewall.sh root/firewallRule_egress.sh root/firewall_rule.sh root/func.sh root/loadbalancer.sh root/monitorServices.py root/reconfigLB.sh root/redundant_router/arping_gateways.sh.templ root/redundant_router/backup.sh.templ root/redundant_router/check_bumpup.sh root/redundant_router/check_heartbeat.sh.templ root/redundant_router/checkrouter.sh.templ root/redundant_router/conntrackd.conf.templ root/redundant_router/disable_pubip.sh root/redundant_router/enable_pubip.sh.templ root/redundant_router/fault.sh.templ root/redundant_router/heartbeat.sh.templ root/redundant_router/keepalived.conf.templ root/redundant_router/master.sh.templ root/redundant_router/primary-backup.sh.templ root/redundant_router/services.sh root/userdata.py root/userdata.sh 91
  • 92. systemvm.iso/cloud-scripts.tgz usr/sbin/xe-daemon usr/sbin/xe-linux-distribution usr/sbin/xe-update-guest-attrs var/www/html/latest/.htaccess var/www/html/userdata/.htaccess etc/logrotate.d/ etc/logrotate.conf etc/logrotate.d/apache2 etc/logrotate.d/cloud etc/logrotate.d/dnsmasq etc/logrotate.d/haproxy etc/logrotate.d/ppp etc/logrotate.d/rsyslog 92
  • 93. systemvm.iso/systemvm.zip cloud-agent-4.3.0.jar cloud-core-4.3.0.jar cloud-api-4.3.0.jar cloud-utils-4.3.0.jar cloud-framework-managed-context-4.3.0.jar slf4j-api-1.7.5.jar javax.inject-1.jar spring-context-3.2.4.RELEASE.jar spring-aop-3.2.4.RELEASE.jar aopalliance-1.0.jar spring-beans-3.2.4.RELEASE.jar spring-core-3.2.4.RELEASE.jar commons-logging-1.1.1.jar spring-expression-3.2.4.RELEASE.jar aspectjweaver-1.7.0.jar log4j-1.2.16.jar slf4j-log4j12-1.7.5.jar cglib-nodep-2.2.2.jar commons-codec-1.6.jar bcprov-jdk16-1.46.jar jsch-0.1.42.jar jasypt-1.9.0.jar trilead-ssh2-build213-svnkit-1.3-patch.jar aws-java-sdk-1.3.22.jar httpclient-4.2.1.jar httpcore-4.2.1.jar jackson-core-asl-1.8.9.jar jackson-mapper-asl-1.8.9.jar apache-log4j-extras-1.1.jar ejb-api-3.0.jar java-ipv6-0.10.jar commons-configuration-1.8.jar commons-lang-2.6.jar reflections-0.9.8.jar guava-14.0-rc1.jar javassist-3.12.1.GA.jar dom4j-1.6.1.jar esapi-2.0.1.jar commons-beanutils-core-1.7.0.jar commons-collections-3.2.jar commons-fileupload-1.2.jar xom-1.1.jar xercesImpl-2.6.2.jar xalan-2.7.0.jar jaxen-1.1-beta-8.jar jdom-1.0.jar 93
  • 94. systemvm.iso/systemvm.zip bsh-core-2.0b4.jar antisamy-1.4.3.jar batik-css-1.7.jar batik-ext-1.7.jar batik-util-1.7.jar nekohtml-1.9.12.jar commons-httpclient-3.1.jar commons-net-3.3.jar gson-1.7.2.jar cloud-framework-config-4.3.0.jar cloud-framework-db-4.3.0.jar ehcache-core-2.6.6.jar javax.persistence-2.0.0.jar commons-dbcp-1.4.jar commons-pool-1.6.jar mysql-connector-java-5.1.21.jar cloud-engine-api-4.3.0.jar cxf-bundle-jaxrs-2.7.0.jar woodstox-core-asl-4.1.4.jar stax2-api-3.1.1.jar xmlschema-core-2.0.3.jar geronimo-javamail_1.4_spec-1.7.1.jar wsdl4j-1.4.jar jetty-continuation-8.1.7.v20120910.jar jetty-http-8.1.7.v20120910.jar jetty-io-8.1.7.v20120910.jar jetty-util-8.1.7.v20120910.jar jetty-security-8.1.7.v20120910.jar geronimo-servlet_3.0_spec-1.0.jar javax.ws.rs-api-2.0-m10.jar cloud-framework-rest-4.3.0.jar jackson-module-jaxb-annotations-2.1.1.jar jackson-core-2.1.1.jar jackson-databind-2.1.1.jar jackson-annotations-2.1.1.jar jackson-jaxrs-json-provider-2.1.1.jar cloud-framework-ipc-4.3.0.jar commons-io-1.4.jar commons-daemon-1.0.10.jar cloud-secondary-storage-4.3.0.jar cloud-server-4.3.0.jar spring-web-3.2.4.RELEASE.jar cloud-framework-cluster-4.3.0.jar cloud-framework-jobs-4.3.0.jar xstream-1.3.1.jar xpp3_min-1.1.4c.jar mail-1.4.jar 94
  • 95. systemvm.iso/systemvm.zip activation-1.1.jar jstl-1.2.jar cloud-engine-schema-4.3.0.jar cloud-framework-events-4.3.0.jar cloud-engine-components-api-4.3.0.jar cloud-console-proxy-4.3.0.jar cloudstack-service-console-proxy-rdpclient-4.3.0.jar tomcat-embed-core-7.0.30.jar scripts/ scripts/storage/ scripts/storage/secondary/ scripts/storage/secondary/create_privatetemplate_fro m_snapshot_xen.sh scripts/storage/secondary/createvolume.sh scripts/storage/secondary/createtmplt.sh scripts/storage/secondary/installIso.sh scripts/storage/secondary/listvolume.sh scripts/storage/secondary/listvmtmplt.sh scripts/storage/secondary/cloud-install-sys-tmplt scripts/storage/secondary/swift run.bat ssvm-check.sh run-proxy.sh run.sh secstorage.sh _run.sh config_ssl.sh config_auth.sh consoleproxy.sh ipfirewall.sh conf/ conf/consoleproxy.properties conf/agent.properties conf/log4j-cloud.xml 95
  • 96. systemvm.iso/systemvm.zip images/ images/shrink_button.gif images/stop_button.gif images/grid_headerbg.gif images/clr_button_hover.gif images/minimize_button.gif images/cad.gif images/notready.jpg images/bright-green.png images/dot.cur images/shrink_button_hover.gif images/back.gif images/right.png images/left.png images/gray-green.png images/play_button_hover.gif images/minimize_button_hover.gif images/right2.png images/winlog.png images/clr_button.gif images/play_button.gif images/cannotconnect.jpg images/stop_button_hover.gif js/ js/ajaxviewer.js js/ajaxkeys.js js/cloud.logger.js js/jquery.js js/handler.js ui/ ui/viewer-connect-failed.ftl ui/viewer-update.ftl ui/viewer-bad-sid.ftl ui/viewer.ftl css/ css/logger.css css/ajaxviewer.css certs/ certs/realhostip.key certs/realhostip.crt certs/localhost.crt certs/localhost.key certs/realhostip.keystore 96
  • 97. systemvm.iso/cloud-scripts.tgzのカスタマイズ カスタマイズ内容: 1)シスログ転送を設定し,シスログサーバにログを送る。 2)任意のファイルをetcに配置する。 tar zxvf cloud-scripts.tgz vi etc/rsyslog.conf : *.* @@シスログサーバIPアドレス:514 : touch etc/additional-file 97
  • 98. systemvm.iso/cloud-scripts.tgzのカスタマイズ systemvm.isoを再作成する。 tar zcvf cloud-scripts.tgz etc opt root usr var cp -p /tmp/3/cloud-scripts.tgz /tmp/2/ ls /tmp/2 authorized_keys cloud-scripts.tgz systemvm.zip mkisofs -r -o systemvm.iso /tmp/2 各サーバのsystemvm.isoを置き換える。 ・CloudStack Management Server /usr/share/cloudstack-common/vms/systemvm.iso ・XenServer /opt/xensource/packages/iso/systemvm.iso 98
  • 99. 設定変更が反映されているか確認する 仮想ルータをCloudStack UIから再起動する。 →仮想ルータ起動時にsystemvm.isoの内容が仮想ルータに反映される。 ls /etc/ : acpi additional-file adduser.conf : cat /etc/rsyslog.conf # /etc/rsyslog.conf Configuration file for rsyslog. # : local0.* -/var/log/haproxy.log *.* @@172.16.0.102:514 : 99
  • 100. ログ転送 仮想ルータからシスログサーバにログが転送されることを確認。 tail -f /var/log/messages Jun 15 07:22:08 r-45-VM kernel: imklog 5.8.11, log source = /proc/kmsg started. Jun 15 07:22:08 r-45-VM rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2668" x-info="http://www.rsyslog.com"] start Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Initializing cgroup subsys cpuset Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Initializing cgroup subsys cpu Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-15) ) #1 SMP Debian 3.2.41-2 Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Command line: root=UUID=89eb6b0a-5f5a-47cc-8bd1-371987c5700e ro debian-installer=en_US quiet -- quiet console=hvc0%template=domP%name=r-45- VM%eth2ip=192.168.11.130%eth2mask=255.255.255.0%gateway=192.168.11.254%eth0ip=10.1.1.1%eth0mask=255.255.255.0%domain=cs2clo ud.internal%dhcprange=10.1.1.1%eth1ip=169.254.2.140%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=172.16.0.38 Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Disabled fast string operations Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] ACPI in unprivileged domain disabled Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Released 0 pages of unused memory Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Set 0 page(s) to 1-1 mapping Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] BIOS-provided physical RAM map: Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] Xen: 0000000000100000 - 0000000008000000 (usable) Jun 15 07:22:08 r-45-VM kernel: [ 0.000000] NX (Execute Disable) protection: active : 100
  • 101. ログ転送 Jun 15 07:22:19 r-45-VM cloud: vpn_l2tp.sh: created VPN chain in PREROUTING mangle Jun 15 07:22:20 r-45-VM ipsec_setup: Starting Openswan IPsec U2.6.37-g955aaafb-dirty/K3.2.0-4-amd64... Jun 15 07:22:20 r-45-VM ipsec_setup: Using NETKEY(XFRM) stack Jun 15 07:22:20 r-45-VM kernel: [ 36.514030] Initializing XFRM netlink socket Jun 15 07:22:20 r-45-VM ipsec_setup: multiple ip addresses, using 192.168.11.130 on eth2 Jun 15 07:22:20 r-45-VM ipsec_setup: ...Openswan IPsec started Jun 15 07:22:20 r-45-VM cloud: vpn_l2tp.sh: waiting ipsec start... Jun 15 07:22:20 r-45-VM pluto: adjusting ipsec.d to /etc/ipsec.d Jun 15 07:22:20 r-45-VM ipsec__plutorun: adjusting ipsec.d to /etc/ipsec.d Jun 15 07:22:20 r-45-VM ipsec__plutorun: 002 added connection description "L2TP-PSK" Jun 15 07:22:21 r-45-VM xl2tpd[4137]: setsockopt recvref[30]: Protocol not available Jun 15 07:22:21 r-45-VM xl2tpd[4137]: This binary does not support kernel L2TP. Jun 15 07:22:21 r-45-VM xl2tpd[4138]: xl2tpd version xl2tpd-1.3.1 started on r-45-VM PID:4138 Jun 15 07:22:21 r-45-VM xl2tpd[4138]: Written by Mark Spencer, Copyright (C) 1998, Adtran, Inc. Jun 15 07:22:21 r-45-VM xl2tpd[4138]: Forked by Scott Balmos and David Stipp, (C) 2001 Jun 15 07:22:21 r-45-VM xl2tpd[4138]: Inherited by Jeff McAdams, (C) 2002 Jun 15 07:22:21 r-45-VM xl2tpd[4138]: Forked again by Xelerance (www.xelerance.com) (C) 2006 Jun 15 07:22:21 r-45-VM xl2tpd[4138]: Listening on IP address 0.0.0.0, port 1701 Jun 15 07:22:21 r-45-VM cloud: edithosts: releasing 10.1.1.236 Jun 15 07:22:21 r-45-VM cloud: edithosts: released 10.1.1.236 Jun 15 07:22:21 r-45-VM cloud: edithosts: update 02:00:7b:ed:00:01 10.1.1.236 vm1 to hosts Jun 15 07:22:24 r-45-VM ntpd[2875]: Listen normally on 10 eth2 192.168.11.132 UDP 123 Jun 15 07:22:24 r-45-VM ntpd[2875]: peers refreshed 101
  • 102. 102 1.概要 2.内部構造 4.性能 4.カスタマイズ 5.新機能
  • 103. Apache Cloudstack 4.3 Design Documents Alert publishing via Web ROOT admin API Contrail network plugin Database High Availability (DB HA) Dynamic Compute Offering FS Hyper-V 2012 (3.0) Support Improve SystemVm upgrades LDAP user provisioning Linux native VXLAN support on KVM hypervisor Marvin Refactor Migration of NFS Secondary Storage to Object Store Modularize Spring Monitoring VR services Palo Alto Firewall Integration Pluggable VM snapshot related operations Remote Access VPN for VPC Report CPU sockets Root volume metering Site-to-site VPN VR-to-VR functional spec SSL Termination Support Update UI visual appearance 103
  • 104. Monitoring VR services https://cwiki.apache.org/confluence/display/CLOUDSTACK/Monitoring+VR+services Introduction: Virtual router has running services which needs to run always until cloudsack disable it.In VR if some service goes down currently there is no mechanism to alert the admin and take action on the crashed services. This feature is about monitoring the services rendered by the VR. Goal for this feature is to monitor all the VR services and ensure they are running through the lifetime of VR On service failure a) Restart the service b) Generate an alert and event indicating failure This monitoring VR services has two tasks. 1.monitoring services in VR 2.sending alerts from router to external receivers 104
  • 105. Monitoring VR services crontab -l #monitoringConfig SHELL=/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin */3 * * * * /usr/bin/python /root/monitorServices.py more /root/monitorServices.py #!/usr/bin/python # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file : def getConfig( config_file_path = "/etc/monitor.conf" ): """ Reads the process configuration from the config file. Config file contains the processes to be monitored. 105
  • 106. Monitoring VR services more /etc/monitor.conf #Monitor services config [dhcp] processname=dnsmasq servicename=dnsmasq pidfile=/var/run/dnsmasq/dnsmasq.pid [loadbalancing] processname=haproxy servicename=haproxy pidfile=/var/run/haproxy.pid [ssh] processname=sshd servicename=ssh pidfile=/var/run/sshd.pid [webserver] processname=apache2 servicename=apache2 pidfile=/var/run/apache2.pid 106
  • 107. Apache Cloudstack 4.4 Design Documents Baremetal Advanced Networking Support Cloudstack network-element plugin to orchestrate Juniper's switches (for L2 services) Cloudstack Windowsfication Configuring load balancing rules for VM nic secondary ips GPU and vGPU support for CloudStack Guest VMs Granular SCSI Controller support in CloudStack over VMware deployments Hyper-V support features in 4.4 In-memory event bus IPv6 in VPC Router KVM Support For Multiple Template Formats LXC 2.0 OVS distributed routing and network ACL Proposal - Ability to add new guest OS mappings PVLAN support for CloudStack deployment over Nexus 1000v in VMware environment Region level VPC and guest network spanning multiple zones Root Resize Support Storage OverProvisioning as Per Primary Basis Support OVA files containing multiple disks Virtual Router aggregated command execution Virtual Router Service Failure Alerting 107
  • 108. To close Virtual Router 1.概要 VM , Network Service , External devices 2.内部構造 Debian Linux , OSS , Scripts 3.性能 4.カスタマイズ Reset , テンプレート,systemvm.iso 5.新機能 108
  • 109. 参考文献/Reference CloudStack Administration Documentation Managing Networks and Traffic http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/networking_and_traffic.html How to Customize CloudStack Virtual Machines https://support.citrix.com/article/CTX133441 Apache Cloudstack 4.2 Design Documents Build Your Own SystemVM Templates https://cwiki.apache.org/confluence/display/CLOUDSTACK/Build+Your+Own+SystemVM+Templates 109
  • 110. 参考文献/Reference CloudStack仮想ルータの謎に迫る / @MayumiK0 http://www.slideshare.net/samemoon/cloud-stackadventcalendar-2012121201-15600230 CloudStackのアーキテクチャ / Kimihiko Kitase http://www.slideshare.net/kkitase/cloudstack-architecture-19886203 Virtual Router in CloudStack 4.4 / Sheng Yang http://www.youtube.com/watch?v=0lxaYOjvghQ http://events.linuxfoundation.org/sites/events/files/slides/VR_4_4%20.pdf http://www.netfilter.org http://vyos.net http://www.juniper.net/jp/jp/products- services/security/firefly-perimeter/ 110
  • 111. 111 ありがとうございました Thank you so much.

×