Vyatta Subscription Edition 6.5 R1 Testing and Analysis

RA-111812-02 Vyatta Subscription Edition 6.5 R1 Test Plan and Results November 2012

Vyatta Subscription Edition 6.5 R1
Test Plan and Results

Report RA-111812-02
November 2012

Testing Initiated Independently by Router Analysis, Inc
Copyright © 2012 Router Analysis, Inc.


Testing Dates: October 10th – 30th 2012
Report Updated: November 20th 2012
Report Author: Steven Noble

Vyatta Subscription Edition 6.5 R1
Key Points
Performance Summary:
Vyatta is able to forward 100% Line-Rate

test system

Performance degrades as features are o 4.2 Mpps at 64 bytes.
added. o Line-Rate at 375 bytes
Vyatta handles QoS with no issues,
-
tion interface is more than 200% over-
subscribed.
o Eight Tenants – Two Gigabit Uplinks
Vyatta can be run directly on commodity
hardware or in a virtual machine
o Line-Rate IMIX between Two Ports
o uRPF/ACLs have no impact on performance

Test System
Configuration:
Hardware:
Supermicro X9SAE-V
Intel I7-3770 / 32GB ECC RAM
Four Intel I340-T2 NICs (8 Total
Interfaces)
Intel 520 Series 240GB SSD
Cost: ~$1600 US

VM Configuration:
VMware 5.1.0 Hypervisor
2 Or 4vCPUs, 4GB Of RAM
Direct Access To Uplink Ports
Via VMDirectpath

Router Analysis Contact Information
info@routeranalysis.com IMIX

Testing Initiated Independently by Router Analysis, Inc
Copyright © 2012 Router Analysis, Inc.

!

Table of Contents

Table of Contents ................................................................................ 3!
Table of Figures ................................................................................... 3!
Results Summary ................................................................................. 4!
Test Lab Setup .................................................................................... 5!
Tests Carried Out ................................................................................. 6!
Testing Results Reviewed .................................................................... 7!
Appendix A – Test Cases ................................................................... 12!
Appendix B – Sample Test Result Data Tables................................... 14!
About Router Analysis ........................................................................ 15!

Table of Figures

Figure 1 - Vyatta 6.5 Deployment Scenarios. ..................................... 4
Figure 2 – Vyatta 6.5 R1 VM Setup ..................................................... 5
Figure 3 - Vyatta 6.5 R1 IPv4 and IPv6 IMIX Forwarding .................... 7
Figure 4 - ACL + uRPF Results ........................................................... 8
Figure 5 - QoS Loss On TOS 0 at 57% Line Rate ............................... 9
Figure 6 - QoS Loss on TOS 0 at 57% Line Rate ................................ 9
Figure 7 - At 69% TOS 4 is affected ................................................. 10
Figure 8 - 39% Loss On TOS 4 ......................................................... 10
Figure 9 - 500k BGP Routes ............................................................. 11

3 Copyright © 2012 Router Analysis
!

!

Results Summary
Vyatta Subscription Edition 6.5 R1 (Vyatta SE 6.5) performed better than
expected when tested in the Router Analysis test lab. Tests were run on both Bare
Metal (directly on the server) and in a Virtual Machine (under VMWare ESXi 5.1).

Featureless IPv4 forwarding performance for 64 byte packets crossed
4Mpps after modifications were made to the host server used for the test. Initial
non-optimal setup tests showed 1.7Mpps.

Enabling features such as ACLs and uRPF dropped the routers forwarding
performance for 64 byte packets down to 3Mpps for IPv4.

QoS functioned as expected, protecting streams as configured.

Below are the deployment scenarios that were tested by Router Analysis.
Testing included two different VM setups, one using two vCPUs, one using four
vCPUs and a Bare Metal setup using the test lab setup. The following graph is the
throughput of each system while running IMIX traffic over it.

Figure 1 - Vyatta 6.5 Deployment Scenarios.

!
!

!

Test Lab Setup
The Router Analysis test lab utilizes specially built systems from IXSystems
to host software based systems. The hardware specs are the following:

SuperMicro X9SAE-V with a single Intel I7-3770 CPU, 32GB ECC RAM,
240GB Intel 520 Series SSD and 4 Intel I340-T2 GbE NICs.

Setting up the system to optimally forward packets is a key part of working
with any software forwarding router. The choice of network cards was key to
getting the results included in this test.

Vyatta SE 6.5 was configured to use all eight GbE interfaces. Tests were
done with 4GB RAM on Bare Metal but Vyatta never used more than 1GB of ram
during any tests.

When testing under VMWare, Vyatta SE 6.5 was given 4GB RAM, 2vCPUs,
vSwitch access to six interfaces and DirectPath access to two interfaces. A total of
eight tenant systems were emulated including two on the same VMWare host and
six via links to the tester.

To generate data for the 4vCPU test, four VMs were used along with four
GbE interfaces for tenants. The test used four GbE interfaces connected via
DirectPath for uplinks.

Figure 2 – Vyatta 6.5 R1 VM Setup

!

!

Tests Carried Out
• IP forwarding rate
o Minimum Packet Size (64 bytes IPv4, 82 bytes IPv6)
! With ACLs (permit any any)
! With uRPF
! With QoS
o Packet IMIX + Features
! IPv4
! IPv6
o 1500 Byte Packets + Features
• QoS Functionality
o QoS
• OSPF
o 5000 Routes
! One Neighbor
! Five Neighbors
• BGP
o 500,000 Routes
! One Neighbor
! Five Neighbors
!

!

!

Testing Results Reviewed
IPv4 and IPv6 IMIX Forwarding Results with Features

3000000

2500000

2000000

2vCPU VM
1500000
4vCPU VM
Bare Metal
1000000

500000

0
IPv4 Base
IPv4 ACL
IPv6 Base
IPv6 ACL

Figure 3 - Vyatta 6.5 R1 IPv4 and IPv6 IMIX Forwarding

The main takeaway from the results is that Vyatta SE 6.5 performs well both
in a Bare Metal environment and as a Virtualized Multi-Tenant Router.

While testing, the lowest performance appeared while executing the “IPv6
forwarding with ACLs, uRPF, and QoS IMIX” test case. Even then the performance
was reasonable at 2.25Mpps.

The biggest impact within virtualization was using the standard vSwitch to
connect hardware interfaces. Future tests may include using the distributed
vSwitch or specialized hardware to see how much impact the different vSwitch
types have.
!

!

!
Base Test Cases:

IPv4/IPv6 IMIX Forwarding + Features Tests

For these tests we follow the following testing concepts to find the best-case
performance of the system. Testing is done using IMIX traffic on bare-metal.

Step 1: Configure 8G of ports and send traffic in a full mesh – 2.58Mpps.

Step 2: Configure a simple permit all ACL on the router – No Change.

Step 3: Configure the ACL inbound on one interface – 2.481Mpps.

Step 4: Configure the ACL outbound on one interface – No Change.

Step 5: Configure the ACL in/out on all interfaces – 2.38Mpps.

Step 6: Configure traffic to run at about 2.38Mpps.

Step 7: Configure uRPF on one interface – No Change.

Step 8: Configure uRPF on a second interface – No Change.

Step 9: Configure uRPF on the rest of the interfaces – No Change.

Result: uRPF has no impact on traffic.

!
Figure 4 - ACL + uRPF Results

!
!

!

!
QoS Testing and Results:

From our tests we have determined that QoS works as expected on the
system. In a simple test involving four GbE links (a three to one oversubscription),
three different TOS bits were set (0, 4 and 7) and traffic was sent balanced (33.3%
per TOS) from each GbE towards a single GbE port outbound.

The router was configured so that TOS 0 was protected up to 15% of line
rate, TOS 4 was protected up to 15% of line rate and TOS 7 was protected up to
70% of line rate.

When running the test the following results were seen:

< 100% all streams went through without issue
> 100% TOS 0 started to be dropped
~ 140% TOS 4 started to be dropped
~ 190% TOS 7 started to be dropped

Step 1: Send less than 33% line-rate from each of the three ports

Result: Up to almost 33% line-rate from the 3 ports (< 100% outbound on
other interface) no loss was seen. Above 33% we saw loss for the TOS 0 stream
but not the TOS 4 or 7 streams.

Step 2: Send 50% line-rate from the TOS 0 port (113% total)

Figure 5 - QoS Loss On TOS 0 at 57% Line Rate

Result: At above 34% we start to see TOS 0 traffic drop as shown in the
magnified section below where we are sending 50% traffic on the TOS 0 link:

Figure 6 - QoS Loss on TOS 0 at 57% Line Rate

!

!
Step 3: Up the traffic to about 69% line-rate per port (207% total)

Figure 7 - At 69% TOS 4 is affected

As shown in this magnified section we see about 38% Loss on TOS 4 when
sending 69% line-rate per port (207% line-rate outbound)

Figure 8 - 39% Loss On TOS 4

Step 4: Raise the traffic on each stream to above 88.5% traffic loss is seen
on all streams.
!
! Result: Loss is seen on all 3 QoS traffic profiles.

Final Result: QoS works as expected.

!

!

!

!
OSPF/BGP TESTING AND RESULTS:

For OSPF and BGP we setup the following tests:

Test 1: Send 5000 OSPF Routes in on a single interface and verify routes
are forwarding correctly using traffic.

Test 2: Send 5000 OSPF Routes in on five interfaces and verify routes are
forwarding correctly using traffic.

Test 3: Send 500k BGP Routes in on a single interface and verify routes are

Test 4: Send 500k BGP Routes in on five interfaces and verify routes are

Results:

In all cases forwarding worked as expected.

!
Figure!9!)!500k!BGP!Routes!

!

!

!

Appendix A – Test Cases
The following two test cases represent a skeleton format of the set of tests
performed in the Router Analysis lab.

Test Case One – IP Forwarding

Synopsis:

When testing systems that have a set PPS or Forwarding Bandwidth that is lower
than the combined interfaces available; it is necessary to validate the best-case
base-forwarding rate. Once the rate has been determined, it is possible to judge
other test cases against the best-case forwarding rate.

Goal of Test:

To confirm best-case IPv4 or IPv6 forwarding rate using the minimum and
maximum packet size accepted by the system.

Procedure:

Configure the minimum number of links between the tester and the system under
test (SUT) to provide either a.) More than the maximum PPS or b.) More than the
maximum forwarding bandwidth as claimed by the vendor.

For example:

Vyatta SE 6.5 performance is based on the hardware it is installed on. To test the
system you would start with the same number of interfaces as cores you have in
the system.

Data to Collect:

Maximum PPS for IPv4, IPv6 and a 50/50 mix of IPv4 and IPv6
Maximum traffic forwarding for IPv4, IPv6 and a 50/50 mix of IPv4 and IPv6

!

!

!

Test Case Two – QoS Testing

Synopsis:
!
This is a simple QoS test based on information from the IXIA BlackBook. We have
modified it slightly to utilize more than 2 ports.

Goal of Test:

Determine if QoS functions as expected by creating three different TOS valued
streams, 0, 4 and 7. On the router configure a traffic policy to prefer 7 over 4 and 4
over 0.

Procedure:

Using IxNetwork create three streams with TOS levels of 0, 4 and 7 on three
interfaces going towards a single interface.

On the outbound interface configure a QoS policy to prefer TOS 7 over TOS 4 and
TOS 4 over TOS 0.

Run traffic and slowly raise it until you see packets drop for 0, then 4 and then 7.

Additive Tests:

Add in background traffic from your earlier test where you saw packet loss due to
features. Confirm that the QoS policy is still working as expected.

Data to Collect:

Packet loss data for each stream at different traffic levels.
Screen Captures of the IxNetwork screen during the tests.

!

!

!

Appendix B – Sample Test Result Data Tables
The following tables represent a sample of the data collected during testing in the
Router Analysis Lab.

IPv4 Forwarding Results Data

Bare-Metal Eight GbE Interfaces

Packet Size (bytes) Aggregate No Drop Rate (pps) Aggregate TX Rate (Gbps)
64 4017857 2.057
128 3885135 3.978
256 3079710 6.307
512 1879699 7.699
1024 957854 7.846
1280 769231 7.876
1518 650195 7.895

Under VMWare 5.1 Two GbE Interfaces

Packet Size (bytes) Aggregate Aggregate Aggregate
Throughput (pps) Throughput (Mbps) Throughput %
64 500000 256 16.8
128 394321 404 23.3
256 253512 519 28
512 461254 1889 98
1024 239463 1962 100
1280 192307 1969 100
1518 162548 1974 100

!

!

!

About Router Analysis
With a world-class fully independent testing lab, Router Analysis provides
unbiased information to its clients.

Router Analysis provides the following services:

Network Equipment Testing
Consultation and Advisement
Test Plan Creation
On-Site Support
Network Design Consultation, Verification and Installation
… And More.

Contact Router Analysis by phone at +1 888 SONN COM (766-6266) or by
email at info@routeranalysis.com for a quote, more information or any questions.

!

Vyatta Subscription Edition 6.5 R1 Testing and Analysis

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Vyatta Subscription Edition 6.5 R1 Testing and Analysis

Similar to Vyatta Subscription Edition 6.5 R1 Testing and Analysis (20)

Recently uploaded

Recently uploaded (20)

Vyatta Subscription Edition 6.5 R1 Testing and Analysis