A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2

Naoto MATSUMOTO
Naoto MATSUMOTOSenior Researcher at Naoto MATSUMOTO
29 Nov, 2012

     SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO
VXLAN TESTING Overview
 1) Linux bridge-based VXLAN is developing by Stephen Hemminger & et al.
 http://linux-network-plumber.blogspot.jp/2012/09/just-published-linux-kernel.html
               +config VXLAN
               +    tristate "Virtual eXtensible Local Area Network (VXLAN)"
               +    depends on EXPERIMENTAL
               +    ---help---
               +                This allows one to create vxlan virtual interfaces that provide
               +                Layer 2 Networks over Layer 3 Networks. VXLAN is often used
               +                to tunnel virtual network infrastructure in virtualized environments.
               +                For more information see:
               +                  http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-02


 2) Infiniband Adapter is the fastest of all for datacenter networking.


                                                              Dual-port Infiniband FDR 56Gbit/s
                                                                               Host Bus Adapter




       VXLAN + IP over Infiniband Network is AWESOME!.
Buildup to the New Kernel & etc
  1) Install OFED for Infiniband Networking
  # wget https://www.openfabrics.org/downloads/OFED/ofed-1.5.4/OFED-1.5.4.1.tgz
  # tar xzvf ./OFED-1.5.4.1.tgz
  # ./OFED-1.5.4.1/install.pl   /*Install OFED software */
  # sync; sync; sync; reboot;

  2) Linux Kernel Update
  # cd /usr/local/src
  # wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.7-rc7.tar.bz2
  # tar xvfj linux-3.7-rc7.tar.bz2
  # cd linux-3.7-rc7
  # cp /boot/config-2.6.32-220.23.1.el6.x86_64 ./.config
  # make oldconfig
  # vi .config
  CONFIG_VXLAN=m          /* Add New Line */
  # make; make modules_install ; make install
  # sync; sync; sync; reboot

  3) iproute2 Update
  # yum install db4-devel.x86_64
  # git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
  # cd iproute2; ./configure ; make ; make install
                                            Source: SAKURA Internet Research Center. 11/2012: Project THORN
VXLAN Configuration

# uname -a
Linux VXLAN-A 3.7.0-rc7 #1 SMP ....                                            56Gbit/s IPoIB
# ip addr add 10.1.1.1/24 dev ib1                                              IP over Infiniband Network
# ip route add 224.0.0.0/4 dev ib1
# ip link set up dev ib1
# ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1       VXLAN Network
# ip addr add 192.168.99.1/24 dev vxlan99
# ip link set up dev vxlan99
# iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */



# uname -a
Linux VXLAN-B 3.7.0-rc7 #1 SMP ....
# ip addr add 10.1.1.2/24 dev ib1
# ip route add 224.0.0.0/4 dev ib1
# ip link set up dev ib1
# ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1
# ip addr add 192.168.99.2/24 dev vxlan99
# ip link set up dev vxlan99
# iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */
                                                 Source: SAKURA Internet Research Center. 11/2012: Project THORN
VXLAN performance check

VXLAN-A# ip link show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc
                                                                             56Gbit/s IPoIB
link/ether b2:cc:a5:b8:74:95 brd ff:ff:ff:ff:ff:ff                           IP over Infiniband Network

VXLAN-A# iperf -c 192.168.99.2 -d
:                                                                            VXLAN Network
[ ID] Interval   Transfer  Bandwidth
[ 3] 0.0-10.0 sec 29.7 GBytes 25.5 Gbits/sec




VXLAN-B# ip link show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc
link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff

VXLAN-B# iperf -s & vnstat -l -i ib1
Monitoring ib1... (press CTRL-C to stop)

rx:   18.49 Gbit/s 53431 p/s     tx:   22.82 Gbit/s 58258 p/s

                                               Source: SAKURA Internet Research Center. 11/2012: Project THORN
Traffic Analysis using Wireshark




                                              Outer Header




                                              Inner Header




                   Source: SAKURA Internet Research Center. 11/2012: Project THORN
Debug TIPS for tshark
  1) VXLAN Traffic Monitoring
  # tshark -pni ib1
  Capturing on ib1
   0.000000      10.1.1.1 -> 239.0.0.99 UDP Source port: 38367 Destination port: 8472
   0.001072      10.1.1.2 -> 10.1.1.1  UDP Source port: 39994 Destination port: 8472

  2) IGMPv3 Join/Releave Monitoring
  # tshark -i ib1 -VV
  Frame 1 (56 bytes on wire, 56 bytes captured)
  Internet Protocol, Src: 10.1.1.1 (10.1.1.1), Dst: 224.0.0.22 (224.0.0.22)
  Internet Group Management Protocol
     [IGMP Version: 3]
     Type: Membership Report (0x22)
     Header checksum: 0xea9a [correct]
     Num Group Records: 1
     Group Record : 239.0.0.99 Change To Exclude Mode
        Record Type: Change To Exclude Mode (4)
        Aux Data Len: 0
        Num Src: 0
        Multicast Address: 239.0.0.99 (239.0.0.99)


                                             Source: SAKURA Internet Research Center. 11/2012: Project THORN
Debug TIPS for iproute2
  1) VXLAN Link status
  # ip -d link show dev vxlan99
  5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN
     link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff
     vxlan id 5001 group 239.0.0.99 dev ib1 port 32768 61000 ageing 300

  2) VXLAN IP Address information
  # ip -d addr show dev vxlan99
  5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN
     link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff
     inet 192.168.99.1/24 scope global vxlan99
     inet6 fe80::48a6:cbff:fe85:fb2b/64 scope link
        valid_lft forever preferred_lft forever

  3) VXLAN Network Table Lookup
  # netstat -nr
  Kernel IP routing table
  Destination    Gateway   Genmask        Flags   MSS Window irtt Iface
  10.1.1.0        0.0.0.0 255.255.255.0   U       00      0 ib1
  192.168.99.0 0.0.0.0    255.255.255.0   U       00      0 vxlan99
  224.0.0.0       0.0.0.0 240.0.0.0       U       00      0 ib1
                                      Source: SAKURA Internet Research Center. 11/2012: Project THORN
Thank you for your interest.
1 of 9

Recommended

Large Scale L2TPv3 Overlay Networking with OSPFv3(DRAFT) by
Large Scale L2TPv3 Overlay Networking with OSPFv3(DRAFT)Large Scale L2TPv3 Overlay Networking with OSPFv3(DRAFT)
Large Scale L2TPv3 Overlay Networking with OSPFv3(DRAFT)Naoto MATSUMOTO
2.1K views11 slides
DevOpsDays Amsterdam Cosmic workshop by
DevOpsDays Amsterdam Cosmic workshopDevOpsDays Amsterdam Cosmic workshop
DevOpsDays Amsterdam Cosmic workshopRemi Bergsma
14K views49 slides
Operating CloudStack: the easy way (automation!) by
Operating CloudStack: the easy way (automation!)Operating CloudStack: the easy way (automation!)
Operating CloudStack: the easy way (automation!)Remi Bergsma
15.1K views40 slides
【検証してみた】いま話題のVMware on IBM Cloud SoftLayer 配布版 by
【検証してみた】いま話題のVMware on IBM Cloud SoftLayer 配布版【検証してみた】いま話題のVMware on IBM Cloud SoftLayer 配布版
【検証してみた】いま話題のVMware on IBM Cloud SoftLayer 配布版Hayama Kyouhei
8K views81 slides
How VXLAN works on Linux by
How VXLAN works on LinuxHow VXLAN works on Linux
How VXLAN works on LinuxEtsuji Nakai
26.7K views22 slides
【共通版】 IBM Cloud (SoftLayer) 最新動向情報 2017年11月版 v1.0 by
【共通版】 IBM Cloud (SoftLayer) 最新動向情報 2017年11月版 v1.0【共通版】 IBM Cloud (SoftLayer) 最新動向情報 2017年11月版 v1.0
【共通版】 IBM Cloud (SoftLayer) 最新動向情報 2017年11月版 v1.0Kazuhiko Isaji
19.9K views67 slides

More Related Content

More from Naoto MATSUMOTO

5Gの見える化 by
5Gの見える化5Gの見える化
5Gの見える化Naoto MATSUMOTO
393 views2 slides
2023年以降のサーバークラスタリング設計(メモ) by
2023年以降のサーバークラスタリング設計(メモ)2023年以降のサーバークラスタリング設計(メモ)
2023年以降のサーバークラスタリング設計(メモ)Naoto MATSUMOTO
266 views2 slides
防災を考慮した水中調査の一考察 by
防災を考慮した水中調査の一考察防災を考慮した水中調査の一考察
防災を考慮した水中調査の一考察Naoto MATSUMOTO
368 views14 slides
旅するパケットの見える化 by
旅するパケットの見える化旅するパケットの見える化
旅するパケットの見える化Naoto MATSUMOTO
280 views2 slides
LTE-M/NB IoTを試してみる nRF9160/Thingy:91 by
LTE-M/NB IoTを試してみる nRF9160/Thingy:91LTE-M/NB IoTを試してみる nRF9160/Thingy:91
LTE-M/NB IoTを試してみる nRF9160/Thingy:91Naoto MATSUMOTO
431 views2 slides
災害時における無線モニタリングによる社会インフラの見える化 by
災害時における無線モニタリングによる社会インフラの見える化災害時における無線モニタリングによる社会インフラの見える化
災害時における無線モニタリングによる社会インフラの見える化Naoto MATSUMOTO
692 views10 slides

More from Naoto MATSUMOTO(20)

2023年以降のサーバークラスタリング設計(メモ) by Naoto MATSUMOTO
2023年以降のサーバークラスタリング設計(メモ)2023年以降のサーバークラスタリング設計(メモ)
2023年以降のサーバークラスタリング設計(メモ)
Naoto MATSUMOTO266 views
防災を考慮した水中調査の一考察 by Naoto MATSUMOTO
防災を考慮した水中調査の一考察防災を考慮した水中調査の一考察
防災を考慮した水中調査の一考察
Naoto MATSUMOTO368 views
旅するパケットの見える化 by Naoto MATSUMOTO
旅するパケットの見える化旅するパケットの見える化
旅するパケットの見える化
Naoto MATSUMOTO280 views
LTE-M/NB IoTを試してみる nRF9160/Thingy:91 by Naoto MATSUMOTO
LTE-M/NB IoTを試してみる nRF9160/Thingy:91LTE-M/NB IoTを試してみる nRF9160/Thingy:91
LTE-M/NB IoTを試してみる nRF9160/Thingy:91
Naoto MATSUMOTO431 views
災害時における無線モニタリングによる社会インフラの見える化 by Naoto MATSUMOTO
災害時における無線モニタリングによる社会インフラの見える化災害時における無線モニタリングによる社会インフラの見える化
災害時における無線モニタリングによる社会インフラの見える化
Naoto MATSUMOTO692 views
BeautifulSoup / selenium Deep dive by Naoto MATSUMOTO
BeautifulSoup / selenium Deep diveBeautifulSoup / selenium Deep dive
BeautifulSoup / selenium Deep dive
Naoto MATSUMOTO212 views
全国におけるCOVID-19対策の見える化 ~宿泊業の場合~ by Naoto MATSUMOTO
全国におけるCOVID-19対策の見える化 ~宿泊業の場合~全国におけるCOVID-19対策の見える化 ~宿泊業の場合~
全国におけるCOVID-19対策の見える化 ~宿泊業の場合~
Naoto MATSUMOTO238 views
我が国の電波の使用状況/携帯電話向け割当 (2019年3月1日現在) by Naoto MATSUMOTO
我が国の電波の使用状況/携帯電話向け割当 (2019年3月1日現在)我が国の電波の使用状況/携帯電話向け割当 (2019年3月1日現在)
我が国の電波の使用状況/携帯電話向け割当 (2019年3月1日現在)
Naoto MATSUMOTO153 views
私たちに訪れる(かもしれない)未来と計算機によるモノコトの見える化 by Naoto MATSUMOTO
私たちに訪れる(かもしれない)未来と計算機によるモノコトの見える化私たちに訪れる(かもしれない)未来と計算機によるモノコトの見える化
私たちに訪れる(かもしれない)未来と計算機によるモノコトの見える化
Naoto MATSUMOTO135 views
仮想化環境におけるバイナリー・ポータビリティの考察 (WebAssemblyの場合) by Naoto MATSUMOTO
仮想化環境におけるバイナリー・ポータビリティの考察 (WebAssemblyの場合)仮想化環境におけるバイナリー・ポータビリティの考察 (WebAssemblyの場合)
仮想化環境におけるバイナリー・ポータビリティの考察 (WebAssemblyの場合)
Naoto MATSUMOTO163 views
インターネットを用いたニア・リアルタイムでの災害観測の考察 by Naoto MATSUMOTO
インターネットを用いたニア・リアルタイムでの災害観測の考察インターネットを用いたニア・リアルタイムでの災害観測の考察
インターネットを用いたニア・リアルタイムでの災害観測の考察
Naoto MATSUMOTO132 views

Recently uploaded

Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...ShapeBlue
184 views12 slides
Business Analyst Series 2023 - Week 4 Session 7 by
Business Analyst Series 2023 -  Week 4 Session 7Business Analyst Series 2023 -  Week 4 Session 7
Business Analyst Series 2023 - Week 4 Session 7DianaGray10
139 views31 slides
Generative AI: Shifting the AI Landscape by
Generative AI: Shifting the AI LandscapeGenerative AI: Shifting the AI Landscape
Generative AI: Shifting the AI LandscapeDeakin University
53 views55 slides
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...Bernd Ruecker
54 views69 slides
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates by
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesShapeBlue
252 views15 slides
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue by
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueShapeBlue
203 views54 slides

Recently uploaded(20)

Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by ShapeBlue
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
ShapeBlue184 views
Business Analyst Series 2023 - Week 4 Session 7 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 7Business Analyst Series 2023 -  Week 4 Session 7
Business Analyst Series 2023 - Week 4 Session 7
DianaGray10139 views
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker54 views
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates by ShapeBlue
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates
ShapeBlue252 views
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue by ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
ShapeBlue203 views
Digital Personal Data Protection (DPDP) Practical Approach For CISOs by Priyanka Aash
Digital Personal Data Protection (DPDP) Practical Approach For CISOsDigital Personal Data Protection (DPDP) Practical Approach For CISOs
Digital Personal Data Protection (DPDP) Practical Approach For CISOs
Priyanka Aash158 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson160 views
Why and How CloudStack at weSystems - Stephan Bienek - weSystems by ShapeBlue
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystems
ShapeBlue238 views
DRBD Deep Dive - Philipp Reisner - LINBIT by ShapeBlue
DRBD Deep Dive - Philipp Reisner - LINBITDRBD Deep Dive - Philipp Reisner - LINBIT
DRBD Deep Dive - Philipp Reisner - LINBIT
ShapeBlue180 views
The Power of Heat Decarbonisation Plans in the Built Environment by IES VE
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built Environment
IES VE79 views
State of the Union - Rohit Yadav - Apache CloudStack by ShapeBlue
State of the Union - Rohit Yadav - Apache CloudStackState of the Union - Rohit Yadav - Apache CloudStack
State of the Union - Rohit Yadav - Apache CloudStack
ShapeBlue297 views
The Role of Patterns in the Era of Large Language Models by Yunyao Li
The Role of Patterns in the Era of Large Language ModelsThe Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language Models
Yunyao Li85 views
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R... by ShapeBlue
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
ShapeBlue173 views
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ... by ShapeBlue
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...
ShapeBlue186 views
Webinar : Desperately Seeking Transformation - Part 2: Insights from leading... by The Digital Insurer
Webinar : Desperately Seeking Transformation - Part 2:  Insights from leading...Webinar : Desperately Seeking Transformation - Part 2:  Insights from leading...
Webinar : Desperately Seeking Transformation - Part 2: Insights from leading...
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T by ShapeBlue
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&TCloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
ShapeBlue152 views

A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2

  • 1. 29 Nov, 2012 SAKURA Internet Research Center Senior Researcher / Naoto MATSUMOTO
  • 2. VXLAN TESTING Overview 1) Linux bridge-based VXLAN is developing by Stephen Hemminger & et al. http://linux-network-plumber.blogspot.jp/2012/09/just-published-linux-kernel.html +config VXLAN + tristate "Virtual eXtensible Local Area Network (VXLAN)" + depends on EXPERIMENTAL + ---help--- + This allows one to create vxlan virtual interfaces that provide + Layer 2 Networks over Layer 3 Networks. VXLAN is often used + to tunnel virtual network infrastructure in virtualized environments. + For more information see: + http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-02 2) Infiniband Adapter is the fastest of all for datacenter networking. Dual-port Infiniband FDR 56Gbit/s Host Bus Adapter VXLAN + IP over Infiniband Network is AWESOME!.
  • 3. Buildup to the New Kernel & etc 1) Install OFED for Infiniband Networking # wget https://www.openfabrics.org/downloads/OFED/ofed-1.5.4/OFED-1.5.4.1.tgz # tar xzvf ./OFED-1.5.4.1.tgz # ./OFED-1.5.4.1/install.pl /*Install OFED software */ # sync; sync; sync; reboot; 2) Linux Kernel Update # cd /usr/local/src # wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.7-rc7.tar.bz2 # tar xvfj linux-3.7-rc7.tar.bz2 # cd linux-3.7-rc7 # cp /boot/config-2.6.32-220.23.1.el6.x86_64 ./.config # make oldconfig # vi .config CONFIG_VXLAN=m /* Add New Line */ # make; make modules_install ; make install # sync; sync; sync; reboot 3) iproute2 Update # yum install db4-devel.x86_64 # git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git # cd iproute2; ./configure ; make ; make install Source: SAKURA Internet Research Center. 11/2012: Project THORN
  • 4. VXLAN Configuration # uname -a Linux VXLAN-A 3.7.0-rc7 #1 SMP .... 56Gbit/s IPoIB # ip addr add 10.1.1.1/24 dev ib1 IP over Infiniband Network # ip route add 224.0.0.0/4 dev ib1 # ip link set up dev ib1 # ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1 VXLAN Network # ip addr add 192.168.99.1/24 dev vxlan99 # ip link set up dev vxlan99 # iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */ # uname -a Linux VXLAN-B 3.7.0-rc7 #1 SMP .... # ip addr add 10.1.1.2/24 dev ib1 # ip route add 224.0.0.0/4 dev ib1 # ip link set up dev ib1 # ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1 # ip addr add 192.168.99.2/24 dev vxlan99 # ip link set up dev vxlan99 # iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */ Source: SAKURA Internet Research Center. 11/2012: Project THORN
  • 5. VXLAN performance check VXLAN-A# ip link show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc 56Gbit/s IPoIB link/ether b2:cc:a5:b8:74:95 brd ff:ff:ff:ff:ff:ff IP over Infiniband Network VXLAN-A# iperf -c 192.168.99.2 -d : VXLAN Network [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 29.7 GBytes 25.5 Gbits/sec VXLAN-B# ip link show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff VXLAN-B# iperf -s & vnstat -l -i ib1 Monitoring ib1... (press CTRL-C to stop) rx: 18.49 Gbit/s 53431 p/s tx: 22.82 Gbit/s 58258 p/s Source: SAKURA Internet Research Center. 11/2012: Project THORN
  • 6. Traffic Analysis using Wireshark Outer Header Inner Header Source: SAKURA Internet Research Center. 11/2012: Project THORN
  • 7. Debug TIPS for tshark 1) VXLAN Traffic Monitoring # tshark -pni ib1 Capturing on ib1 0.000000 10.1.1.1 -> 239.0.0.99 UDP Source port: 38367 Destination port: 8472 0.001072 10.1.1.2 -> 10.1.1.1 UDP Source port: 39994 Destination port: 8472 2) IGMPv3 Join/Releave Monitoring # tshark -i ib1 -VV Frame 1 (56 bytes on wire, 56 bytes captured) Internet Protocol, Src: 10.1.1.1 (10.1.1.1), Dst: 224.0.0.22 (224.0.0.22) Internet Group Management Protocol [IGMP Version: 3] Type: Membership Report (0x22) Header checksum: 0xea9a [correct] Num Group Records: 1 Group Record : 239.0.0.99 Change To Exclude Mode Record Type: Change To Exclude Mode (4) Aux Data Len: 0 Num Src: 0 Multicast Address: 239.0.0.99 (239.0.0.99) Source: SAKURA Internet Research Center. 11/2012: Project THORN
  • 8. Debug TIPS for iproute2 1) VXLAN Link status # ip -d link show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff vxlan id 5001 group 239.0.0.99 dev ib1 port 32768 61000 ageing 300 2) VXLAN IP Address information # ip -d addr show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff inet 192.168.99.1/24 scope global vxlan99 inet6 fe80::48a6:cbff:fe85:fb2b/64 scope link valid_lft forever preferred_lft forever 3) VXLAN Network Table Lookup # netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.1.1.0 0.0.0.0 255.255.255.0 U 00 0 ib1 192.168.99.0 0.0.0.0 255.255.255.0 U 00 0 vxlan99 224.0.0.0 0.0.0.0 240.0.0.0 U 00 0 ib1 Source: SAKURA Internet Research Center. 11/2012: Project THORN
  • 9. Thank you for your interest.