This document provides an overview of multi-path VPN technologies. It discusses using Linux bridge, Rapid STP, virtual Ethernet NICs, and tunneling protocols like OpenVPN and L2TPv3 to enable multi-path VPNs across multiple cloud providers. It also covers related topics like performance benchmarking and tuning the Linux kernel for improved throughput.

1. (C)Copyright 1996-2010 SAKURA Internet Inc.
5 Sep, 2013
SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO
日本学術振興会産学協力研究委員会 インターネット技術第 163 委員会 (ITRC)
地域間インタークラウド分科会 (RICC)
第3回地域間インタークラウドワークショップ

2. Multi-Path VPN Overview
Bridge + IEEE 802.1w(Rapid STP)
Virtual Ethernet NIC
Process Process Process
Vyatta VM
IaaS-B IaaS-C
IaaS-A
IaaS-D
Source: SAKURA Internet Research Center. 08/2012: Project THORN
Tunnel TunnelTunnel
Vyatta VM
Process Process Process
Vyatta VM
Process Process Process
Vyatta VM
Process Process Process

3. How does it works?
Bridge + IEEE 802.1w(Rapid STP)
Virtual Ethernet NIC
Process Process Process
Tunnel Tunnel Tunnel
Vyatta VM
vyattaVM# sudo rstpctl showport br0
* vtun1 8001 Fwd 8000-027a0b4f3269 8...3269 8001 D
* vtun2 8001 Fwd 8000-027a0b4f3269 8...3269 8002 D
:
veth1 8004 Fwd 8000-027a0b4f3269 8...3269 8004 D
Tunnel Interface + Linux Bridge + IEEE 802.1w (Rapid STP) Daemon
Source: SAKURA Internet Research Center. 08/2012: Project THORN

4. Packages for Multi-Path VPN
Bridge + IEEE 802.1w
(Rapid STP)
Virtual Ethernet NIC
Process Process Process
Tunnel Tunnel Tunnel
VyattaCore 6.4
- Tunneling Protocol:
OpenVPN,GRETAP,L2TPv3 or VXLAN ..etc
- Bridging Daemon: bridge-utils
- Rapid STP Daemon: rstpd
http://git.kernel.org/?p=linux/kernel/git/shemminger/rstp.git
- Virtual Interface: iproute2
http://kernel.org/pub/linux/utils/net/iproute2/
Source: SAKURA Internet Research Center. 08/2012: Project THORN

5. How to Build on Vyatta
configure
delete system package repository community
set system package repository oxnard components main
set system package repository oxnard distribution oxnard
set system package repository oxnard url http://packages.vyatta.com/vyatta-dev/oxnard/unstable
commit
save
reboot
sudo full-upgrade -k
configure
set system package repository squeeze components main
set system package repository squeeze distribution squeeze
set system package repository squeeze url http://cdn.debian.net/debian
commit
save
sudo apt-get update
sudo aptitude install module-assistant
sudo aptitude install linux-vyatta-kbuild
cd /lib/modules/3.0.23-1-586-vyatta-virt/
sudo ln -s /usr/src/linux-image/debian/build/build-i386-none-586-vyatta-virt/ build
sudo aptitude install rpm zlib1g-dev zlib1g-dbg byacc bison flex
sudo module-assistant prepare Source: SAKURA Internet Research Center. 08/2012: Project THORN

6. Install rstpd and switch-setup
- Downloading latest master from git
http://git.kernel.org/?p=linux/kernel/git/shemminger/rstp.git
cd rstp ; sudo make; sudo make install
- L2TPv3 pseudo-wire Config
sudo ip l2tp add tunnel tunnel_id 1 peer_tunnel_id 1 udp_sport 5001 udp_dport 5001 ¥
encap udp local X.X.X.X remote Y.Y.Y.Y
sudo ip l2tp add session tunnel_id 1 session_id 1 peer_session_id 1
:
sudo ifconfig l2tpeth0 0
sudo ifconfig l2tpeth0 up up
- Virtual Ehternet Config
ip link add type veth
sudo ifconfig veth0 up up ; sudo ifconfig veth1 up up
- Bridge/Switch Config
sudo brctl addbr br0
sudo brctl addif br0 veth1
sudo brctl addif br0 l2tpeth0
sudo rstpd
sudo rstpctl rstp br0 on
Bridge + IEEE 802.1w(Rapid STP)
Virtual Ethernet NIC
Process Process Process
Tunnel Tunnel Tunnel
Vyatta VM

7. Linux Performance Tuning
- Linux kernel parameter Tuning
su
sysctl -w net.core.netdev_max_backlog=500000
sysctl -w net.core.rmem_max=67108864
sysctl -w net.core.wmem_max=67108864
sysctl -w net.core.rmem_default=67108864
sysctl -w net.core.wmem_default=67108864
sysctl -w net.ipv4.tcp_mem="67108864 67108864 67108864″
echo 1000000000 > /proc/sys/kernel/shmmax
echo 400 > /proc/sys/vm/nr_hugepages
- Linux RPS(Recive Packet Steering) Tuning
echo 65535 >/proc/sys/net/core/rps_sock_flow_entries
echo 002 > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo 200 > /sys/class/net/l2tpeth0/queues/rx-0/rps_cpus
:
etc
echo 65535 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
echo 65535 > /sys/class/net/l2tpeth0/queues/rx-0/rps_flow_cnt
:
etc Source: SAKURA Internet Research Center. 08/2012: Project THORN

8. Bridge/RSTP Aging Timer
- Linux Bridge STP Timer Config
brctl sethello br0 1
brctl setfd br0 4
brctl setmaxage br0 6
brctl setageing br0 10
- Linux RSTP Timer Config
rstpctl sethello br0 1
rstpctl setmaxage br0 6
rstpctl setfdelay br0 4
Bridge + IEEE 802.1w
(Rapid STP)
Virtual Ethernet NIC
Process Process Process
Tunnel Tunnel Tunnel
VyattaCore 6.4
Source: SAKURA Internet Research Center. 08/2012: Project THORN

9. Performance Benchmark HowTo
- Linux in-kernel packet generator tool
su
modprobe pktgen
echo "rem_device_all" > /proc/net/pktgen/kpktgend_0
echo "add_device eth0" > /proc/net/pktgen/kpktgend_0
echo "max_before_softirq 50000" > /proc/net/pktgen/kpktgend_0
echo "count 0" > /proc/net/pktgen/eth0
echo "clone_skb 1000000" > /proc/net/pktgen/eth0
echo "pkt_size 64" > /proc/net/pktgen/eth0
echo "delay 400" > /proc/net/pktgen/eth0
echo "dst X.X.X.X" > /proc/net/pktgen/eth0
echo "dst_mac aa:18:39:6e:fc:3c" > /proc/net/pktgen/eth0
echo "start" > /proc/net/pktgen/pgctrl
Bridge + IEEE 802.1w(Rapid STP)
Virtual Ethernet NIC
Process Process Process
Tunnel Tunnel Tunnel
Vyatta VM
Source: SAKURA Internet Research Center. 08/2012: Project THORN

10. Tunneling Performance Analysis
[System: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM]
[Traffic Generater: *pktgen, **iperf. Traffic Monitor: vnstat]
VXLAN on UNKOWN-DEVICE (IPoEth) 4.79 Gbit/s**
Mellanox ConnectX3 10GbE-NIC (IPoEth) 9.93 Gbit/s*
Mellanox ConnectX3 40GbE-NIC (IPoEth) 27.45 Gbit/s*
Mellanox ConnectX3 FDR10-HCA (IPoIB) 29.56 Gbit/s**
(Gbit/sec) (Packet/sec)
820Kpps (MTU 1,500)
882Kpps (MTU 1,500)
2.28Mpps (MTU 1,500)
61Kpps (MTU 65,520)
High is Faster Low is Better
OpenVPN on FDR10-HCA (IPoIB) 0.24 Gbit/s*
GRETAP on FDR10-HCA (IPoIB) 5.41 Gbit/s*
L2TPv3 on FDR10-HCA (IPoIB) 10.65 Gbit/s*
N/A
N/A
N/A
Source: SAKURA Internet Research Center. 07/2012: Project THORN

11. Another way to Multi-Path VPN
set interfaces tunnel tun0 address 10.0.0.99/24
set interfaces tunnel tun0 encapsulation gre-multipoint
set interfaces tunnel tun0 local‐ip 192.168.2.99
set interfaces tunnel tun0 multicast enable
set interfaces tunnel tun0 nhrp authentication pre‐shared‐secret NET10
set interfaces tunnel tun0 nhrp holding‐time 300
set interfaces tunnel tun0 nhrp multicast parameters dynamic
set interfaces tunnel tun0 nhrp redirect
set interfaces tunnel tun0 parameters ip key 1
set protocols static route 192.168.1.0/24 next‐hop 10.0.0.1
set protocols static route 192.168.2.0/24 next‐hop 10.0.0.2

13. What is PBR?
1) PBR-LB (Policy Based Router–Load Balancing) is forcusing Server Scale-Out Tech.
- Breaking L2DSR(Layer 2 Direct Server Return) Limitation.
- without NAT.
- without DSCP modify (e.g. Layer 3 Direct Server Return mechanism).
- Add Tunneling Capabiltiy.
- Do not add/modify Server kernel modules.
- using Policy Based Routing Functionality on Newest Software Router.

14. PBR-LB BASIC Configuration.
vyatta$ configuration
# set policy route SRC-PORT-SLB rule 11 destination address A.A.A.A
# set policy route SRC-PORT-SLB rule 11 protocol tcp_udp
# set policy route SRC-PORT-SLB rule 11 set table 11
# set policy route SRC-PORT-SLB rule 11 source port 1-10000
# set protocols static table 10 route 0.0.0.0/0 next-hop 10.0.0.1
# set interfaces ethernet eth0 policy route SRC-PORT-SLB
# commit
# save
*Reference: VYATTA, INC. Policy Based Routing REFERENCE GUIDE. (6.5R1 v01)

15. Internet
Layer 3 Direct Server Return
SERVER
eth0:10.0.0.1/32
lo:A.A.A.A/32
SRC PORT DST ADDR NEXTHOP-TABLE
1-10,000 A.A.A.A 0.0.0.0/0 -> 10.0.0.1
10,001-20,000 A.A.A.A 0.0.0.0/0 -> 10.0.0.2
20,001-30,000 A.A.A.A 0.0.0.0/0 -> 10.0.0.3
Policy Router
(VyattaCore 6.5R1 on x86 Commodity Hardware)
SOURCE: SAKURA Internet Research Center. 02/2013 Project THORN.
*Reference: L3DSR – Overcoming Layer 2 Limitations of Direct Server Return Load Balancing. Jan Schaumann, Systems Architect (NANOG51),
Policy Based Routing REFERENCE GUIDE. (6.5R1 v01), VYATTA, INC.

16. in more detail.
$ show version
Version: VC6.5R1
Description: Vyatta Core 6.5 R1
:
$ show policy route
$ show policy route statistics
...etc

17. One more Thing: Enabling 40GbE
1) Using Pre-installed kernel modeuls for Mellanox 40GbE-NIC(mlx4_core,en)
2) Load 40GbE-NIC kernel module on /etc/modules
$ show version
Version: VC6.5R1
Description: Vyatta Core 6.5 R1
$ sudo vi /etc/modules
mlx4_en
$ sync; sync; sync; reboot
© 2013 Mellanox Technologies. All Rights Reserved.

18. 40GbE-NIC Status Check
$ show interfaces ethernet eth1 physical
Settings for eth1:
Supported ports: [ TP ]
:
Speed: 40000Mb/s
Duplex: Full
Port: Twisted Pair
:
Link detected: yes
driver: mlx4_en
version: 2.0 (Dec 2011)
firmware-version: 2.10.800
bus-info: 0000:01:00.0
© 2013 Mellanox Technologies. All Rights Reserved.

19. 40GbE-NIC Option Check
$ sudo ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: on
© 2013 Mellanox Technologies. All Rights Reserved.

20. 40GbE-NIC module Check
$ sudo modinfo mlx4_en
filename: /lib/modules/3.3.8-1-586-vyatta/.../mellanox/mlx4/mlx4_en.ko
version: 2.0 (Dec 2011)
license: Dual BSD/GPL
description: Mellanox ConnectX HCA Ethernet driver
author: Liran Liss, Yevgeny Petrilin
srcversion: 142AB5E074EB35EDB627247
depends: mlx4_core
intree: Y
vermagic: 3.3.8-1-586-vyatta SMP mod_unload modversions 586
parm: inline_thold:threshold for using inline data (int)
parm: udp_rss:Enable RSS for incomming UDP traffic or disabled (0)...
parm: pfctx:Priority based Flow Control policy on TX[7:0]. Per prio...
parm: pfcrx:Priority based Flow Control policy on RX[7:0]. Per prio...

21. Thanks for your interest.
SAKURA Internet Research Center.