The document discusses using OpenNHRP to enable multicloud connectivity across hybrid cloud deployments. It provides instructions for installing and configuring OpenNHRP on Ubuntu to set up a dynamic multipoint VPN (DMVPN) with one hub and two spoke nodes in different cloud environments. The configuration allows the spoke nodes to connect directly via an encrypted GRE tunnel without traversing the hub, providing optimized traffic flow across clouds.

1. Multicloud connectivity using OpenNHRP
Openstack Summit Lightning talk
May 24, 2018
Sridar Kandaswamy, Bob Melander, Shweta Padubidri
Cisco Cloud CTO group

2. Agenda
 Problem description
 Site to SiteVPNs vs MultipointVPNs
 OpenNHRP Packages/Installation/Configuration
 Scenarios
 Conclusion

3. Hybrid cloud deployments

4. Evolution to multiple clouds

5. VPN options
Point-to-point IPSec tunnels
 Tunnel interface growth on hub
 One per spoke
 All traffic transit the hub
 Reconfiguration of hub everytime
new site is added/removed

6. VPN options
Point-to-point IPSec tunnels
 Tunnel interface growth on hub
 One per spoke
 All traffic transit the hub
 Reconfiguration of hub everytime
new site is added/removed
Dynamic Multi-pointVPN (DMVPN)
 Multi-point GRE (mGRE)
 Next Hop Resolution Protocol (NHRP)
 Redirect
 Automatic creation of tunnels
 Cisco solution

7. Open source implementations
 OpenNHRP
 Quagga NHRP plugin (version >=1.2.0)
 VyOS

8. OpenNHRP - Installation steps (Ubuntu)
# Install IKEv2 stuff + IPSec
sudo apt-get install racoon
sudo apt-get install ipsec-tools
# Install development packages and dependencies for OpenNHRP
sudo apt-get install build-essential
sudo apt-get install libc-ares-dev
sudo apt-get install pkg-config
# Fetch OpenNHRP, compile and install it
# Download & extract tarball from https://sourceforge.net/projects/opennhrp
tar xjvf opennhrp-0.14.1.tar.bz2
cd opennhrp-0.14.1
make
sudo make install

9. Configuration steps
 Create mGRE interface
 Configure IKE
 Racoon
 Configure IPsec SA
 ipsec-tools
 Configure OpenNHRP
 Enable routing
 Static routes or
 Dynamic routing, e.g., BGP
 Configure firewall / security groups
 UDP port 500 — ISAKMP as source and
destination
 UDP port 4500 — NAT-T as a destination
 IP protocol 50 — ESP
 IP protocol 51 — AH (if using AH)
 IP protocol 47 — GRE
 Ports for routing protocol

10. VMVM
VM
AWS
VPC
AZURE
VNET
Openstac
k
network
HUB
SPOKE SPOKE
InternetDMVPN 10.100.0.0/24
eth0
10.20.0.96 (Floating IP: 18.216.240.85)
mpgre1
10.100.0.2
mpgre0
10.100.0.3
eth0
10.10.0.4 (Public IP: 104.42.54.201)
eth0
10.0.2.246 (Elastic IP: 13.58.97.150)
mpgre0
10.100.0.1
10.0.2.0/24 10.10.0.0/24
10.20.0.0/24
Configuration example topology
CIDR next-hop
0.0.0.0/0 10.0.2.1
10.10.0.0/24 10.100.0.3
10.20.0.0/24 10.100.0.1
CIDR next-hop
0.0.0.0/0 10.20.0.1
10.0.3.0/24 10.100.0.2
10.10.0.0/24 10.100.0.3
CIDR next-hop
0.0.0.0/0 10.10.0.1
10.0.2.0/24 10.100.0.2
10.20.0.0/24 10.100.0.1

11. Common configuration (mGRE + IPsec)
# Create multi-point GRE interface
sudo ip tunnel add mpgre0 mode gre key 98701234 ttl 64
sudo ip addr add 10.100.0.1/24 dev mpgre0
sudo ip link set mpgre0 up
# Define pre-shared key for IKE negotiation
sudo bash -c 'cat << EOF >> /etc/racoon/psk.txt
demo@openstack-summit.com rocky-summit
EOF'
Change address to
10.100.0.2/24 and
10.100.0.3/24, respectively
for the spoke nodes
Select a GRE key for
the DMVPN

12. Common configuration (IKE + IPsec)
# Generate configuration for IKE (racoon daemon)
sudo bash -c 'cat << EOF >> /etc/racoon/racoon.conf
path pre_shared_key "/etc/racoon/psk.txt";
remote anonymous {
exchange_mode aggressive;
lifetime time 24 hour;
my_identifier user_fqdn "demo@openstack-summit.com";
nat_traversal on;
script "/etc/opennhrp/racoon-ph1dead.sh" phase1_dead;
proposal {
encryption_algorithm aes;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group 2;
}
}

13. Common configuration (IKE + IPsec)
sainfo anonymous {
pfs_group 2;
lifetime time 12 hour;
encryption_algorithm aes, 3des, blowfish 448, rijndael;
authentication_algorithm hmac_sha1, hmac_md5;
compression_algorithm deflate;
}
EOF'
# Generate configuration for IPSec
sudo bash -c 'cat << EOF >> /etc/ipsec-tools.conf
spdflush;
# Encrypt all traffic in ESP transport mode
spdadd 0.0.0.0/0 0.0.0.0/0 gre -P out ipsec esp/transport//require;
spdadd 0.0.0.0/0 0.0.0.0/0 gre -P in ipsec esp/transport//require;
EOF'

14. Hub configuration (OpenNHRP daemon)
# Generate configuration for OpenNHRP hub
sudo bash -c 'cat << EOF >> /etc/opennhrp/opennhrp.conf
interface mpgre0
holding-time 3600
# make this a hub node
multicast dynamic
# send redirects to enable spoke-to-spoke communication
redirect
interface eth0
# Create shortcut routes for subnets on this interface
shortcut-destinationEOF'

15. Spoke configuration (OpenNHRP daemon)
# Generate configuration for OpenNHRP spoke
sudo bash -c 'cat << EOF >> /etc/opennhrp/opennhrp.conf
interface mpgre0
# register with hub
map 10.100.0.1/24 18.216.240.85 register
holding-time 3600
# act as a spoke
multicast nhs
# Enable shortcut routes
shortcut
# dynamically create tunnels to other spokes
redirect
interface eth0
# Create shortcut routes for subnets on this interface
shortcut-destination

16. Configuration (static routing)
# Enable IPv4 forwarding
sudo sysctl -w net.ipv4.ip_forward=1
# Set static routes (for hub node)
sudo ip route add 10.0.2.0/24 nexthop via 10.100.0.2
sudo ip route add 10.10.0.0/24 nexthop via 10.100.0.3
# Set static routes (for spoke node in Azure)
sudo ip route add 10.20.0.0/24 nexthop via 10.100.0.1
sudo ip route add 10.0.2.0/24 nexthop via 10.100.0.2
# Set static routes (for spoke node in AWS)
sudo ip route add 10.20.0.0/24 nexthop via 10.100.0.1
sudo ip route add 10.10.0.0/24 nexthop via 10.100.0.3

17. Start services
# Restart setkey and racoon services
sudo service setkey restart
sudo service racoon restart
# Start opennhrp service
sudo opennhrp -v

18. VMVM
VM
AWS
VPC
AZURE
VNET
Openstac
k
network
HUB
SPOKE SPOKE
InternetDMVPN 10.100.0.0/24
eth0
10.20.0.96 (Floating IP: 18.216.240.85)
mpgre1
10.100.0.2
mpgre0
10.100.0.3
eth0
10.10.0.4 (Public IP: 104.42.54.201)
eth0
10.0.2.246 (Elastic IP: 13.58.97.150)
mpgre0
10.100.0.1
10.0.2.0/24 10.10.0.0/24
10.20.0.0/24
After start
Statically established
spoke-hub GRE/IPsec tunnel
Statically established
spoke-hub GRE/IPsec tunnel

19. AzureVM OpenNHRP spoke daemon output
ubuntu@vm-B:~$ sudo /usr/sbin/opennhrp –v
opennhrp[115081]: OpenNHRP upstream/0.14.1-10-ge6ad153 startingopennhrp
[115081]: Adding static 10.100.0.1/24 nbma 18.216.240.85 dev mpgre0
...
opennhrp[115081]: Filter code installed (25 opcodes)Create link from 10.100.0.3
(10.10.0.4) to 10.100.0.1 (18.216.240.85)
opennhrp[115081]: [10.100.0.1] Peer up script: success
opennhrp[115081]: NL-ARP(mpgre0) 10.100.0.1 is-at 18.216.240.85
opennhrp[115081]: Sending Registration Request to 10.100.0.1 (my mtu=0)
opennhrp[115081]: Sending packet 3, from: 10.100.0.3 (nbma 10.10.0.4), to: 10.100.0.1
(nbma 18.216.240.85)
...
opennhrp[115081]: Received Registration Reply from 10.100.0.1: success

20. AzureVM security associations
ubuntu@vm-B:~$ sudo racoonctl show-sa isakmp
Destination Cookies Created
18.216.240.85.4500 3311c118100b7621:171faacc5400718f 2018-05-21 23:11:19
ubuntu@vm-B:~$ sudo racoonctl show-sa esp
10.10.0.4[4500] 18.216.240.85[4500] esp-udp mode=transport spi=81812847(0x04e05d6f)
reqid=0(0x00000000)
E: aes-cbc 6508127b ecb0d4fb 200e1fe2 361d20db
A: hmac-sha1 89e10ffe e0931d43 54c204e6 0e14a563 0db9a31c
seq=0x00000000 replay=4 flags=0x00000000 state=mature
created: May 21 23:13:26 2018 current: May 21 23:14:26 2018
...
18.216.240.85[4500] 10.10.0.4[4500] esp-udp mode=transport spi=255020070(0x0f334c26)
reqid=0(0x00000000)
E: aes-cbc 12460a05 95547561 f0de30e2 6fd022cb
A: hmac-sha1 057344d4 1b1e8216 9d7bba03 db73d09b 0753a9ef seq=0x00000000 replay=4
flags=0x00000000 state=mature
created: May 21 23:13:26 2018 current: May 21 23:14:26 2018
...

21. AzureVM OpenNHRP DB
ubuntu@vm-B:~$ sudo opennhrpctl show
Status: ok
...
Interface: mpgre0
Type: localProtocol-Address: 10.100.0.3/32
Flags: up
Interface: mpgre0
Type: static
Protocol-Address: 10.100.0.1/24
NBMA-Address: 18.216.240.85
Flags: up

22. Traceroute 1: Azure spoke to AWS spoke
ubuntu@vm-B:~$ traceroute 10.0.2.246
traceroute to 10.0.2.246 (10.0.2.246), 30 hops max, 60 byte packets
1 10.100.0.1 (10.100.0.1) 78.781 ms 78.724 ms 78.696 ms
2 10.0.2.246 (10.0.2.246) 79.920 ms 79.901 ms 79.879 ms.

23. VMVM
VM
AWS
VPC
AZURE
VNET
Openstac
k
network
HUB
SPOKE SPOKE
InternetDMVPN 10.100.0.0/24
eth0
10.20.0.96 (Floating IP: 18.216.240.85)
mpgre1
10.100.0.2
mpgre0
10.100.0.3
eth0
10.10.0.4 (Public IP: 104.42.54.201)
eth0
10.0.2.246 (Elastic IP: 13.58.97.150)
mpgre0
10.100.0.1
10.0.2.0/24 10.10.0.0/24
10.20.0.0/24
What happened behind the scenes?
Dynamically
established
spoke-spoke
GRE/IPsec tunnel

24. AzureVM OpenNHRP spoke daemon output
...
opennhrp[115081]: Traffic Indication from proto src 10.100.0.1; about packet to
10.0.2.246
...
opennhrp[115081]: Received Resolution Reply 10.0.2.246/24 is at proto 10.100.0.2 nbma
10.0.2.246
opennhrp[115081]: NAT detected: really at proto 10.100.0.2 nbma 13.58.97.150
...
Create link from 10.100.0.3 (10.10.0.4) to 10.100.0.2 (13.58.97.150)
...
opennhrp[115081]: Adding shortcut-route 10.0.2.0/24 nexthop 10.100.0.2 dev mpgre0
expires_in 120:00
...
VPN connexion established
Phase 2 established : 10.10.0.4[4500] -> 13.58.97.150[4500]
opennhrp[115081]: [10.100.0.2] Peer up script: success
...

25. AzureVM security associations
ubuntu@vm-B:~$ sudo racoonctl show-sa isakmp
Destination Cookies Created
18.216.240.85.4500 3311c118100b7621:171faacc5400718f 2018-05-21 23:11:19
13.58.97.150.4500 f4b90dd523121cf7:87f7e521609f9858 2018-05-21 23:24:39
ubuntu@vm-B:~$ sudo racoonctl show-sa esp
10.10.0.4[4500] 13.58.97.150[4500] esp-udp mode=transport spi=139361052(0x084e7b1c)
reqid=0(0x00000000)
E: aes-cbc 551010c4 130011ad 72d37c5a 1bcce0a1
A: hmac-sha1 91010e1e 0a1bafec 17e7fed9 aa980e0a f0e385b3
seq=0x00000000 replay=4 flags=0x00000000 state=mature
created: May 21 23:24:40 2018
...
13.58.97.150[4500] 10.10.0.4[4500] esp-udp mode=transport spi=221059027(0x0d2d17d3)
reqid=0(0x00000000)
E: aes-cbc e1369de2 8b922555 b9888aa2 56bc628a
A: hmac-sha1 50f5abd9 41608f3a 11ae798b bf67c61c 21fc7e46
seq=0x00000000 replay=4 flags=0x00000000 state=mature
created: May 21 23:24:40 2018
...

26. AzureVM OpenNHRP DB
ubuntu@vm-B:~$ sudo opennhrpctl show
Status: ok
...
Interface: mpgre0
Type: localProtocol-Address: 10.100.0.3/32
Flags: up
Interface: mpgre0
Type: cached
Protocol-Address: 10.100.0.2/32
NBMA-Address: 13.58.97.150
NBMA-NAT-OA-Address: 10.0.2.246
Flags: up
Expires-In: 52:38
Interface: mpgre0
Type: static
Protocol-Address: 10.100.0.1/24
NBMA-Address: 18.216.240.85
Flags: up

27. Traceroute 2: Azure spoke to AWS spoke
ubuntu@vm-B:~$ traceroute 10.0.2.246
traceroute to 10.0.2.246 (10.0.2.246), 30 hops max, 60 byte packets
1 10.0.2.246 (10.0.2.246) 78.031 ms 77.977 ms 77.948 ms

28. Conclusion
 Open source DMVPN possible for multi-cloud use cases
 Before you deploy in production:TEST EXTENSIVELY for
 Stability
 Performance
 Compatibility with vendor implemenations
 Caveats
 Packages for popular distros
 Documentation

29. References and FutureWork
 Links
 https://sourceforge.net/projects/opennhrp
 http://savannah.nongnu.org/projects/quagga/
 https://wiki.vyos.net/wiki/DMVPN
 Shannon McFarland
 Multicloud Networking – Connecting OpenStack Private Clouds to Public
Clouds, (Tuesday 22, 9:00am - 9:40am)
 Possible extensions toVPNaaS
 Slides available at: https://www.slideshare.net/BobMelander/