Pankur Agarwal and Muthurajan (M Jay) Jayakumar offered a hands-on lab for Network Service Benchmarking (NSB). NSB extends the yardstick framework to do VNF characterization and benchmarking in three different execution environments - bare metal i.e., native Linux environment, standalone virtual environment, and managed virtualized environment (e.g., OpenStack, etc.). This is part 1 - the introduction.

1. NetworkServiceBenchmarking

2. 2
DatacenterSolutionsGroup
WHYNETWORKSERVICE
Benchmarking?
General Lack of
Telco Grade
Conformance/
Benchmarks
Unclear Network
Workload
Dimensions and
Stress Vectors
Missing System
Level Capacity
Requirements
Network Workload
Scalability/ Agility
Implications on NFV
Operators lack
comprehensive information
for TCO models to plan,
procure and deploy NFV

3. 3
DatacenterSolutionsGroup
NETWORKSERVICE BenchmarkingQuestions
How will a Network
Service perform on a
specified
infrastructure?
How will specific
performance metrics
for a Network Service
change when various
infrastructure features
are introduced or re-
provisioned?
How will specific
performance metrics
for a Network Service
change with realistic
and dynamic traffic
workloads?
How do MANO
components impact
performance with
policy based
networking?
How will Network
Services change in a
SFC with other
Network Services
sharing platform
resources?
How do commercial
VNFs compare to other
VNFs providing similar
functionality for key
Network Service
metrics?

4. 4
DatacenterSolutionsGroup
1 2 3
Dimension network
workloads and stress
vectors for different
market segments
Deliver reference VNFs built
to operational capacities for
the different network
segments
Develop a common
reference set of benchmarks
with test infrastructure for
VNF interoperability
4 5
Contribute test suites to
measure system level
requirements for different
network workloads
including test methodology
and KPIs
Upstream/downstream
adoption within open
source communities
NSBSolvestheIndustryProblem:FIVESTEPS

5. 5
DatacenterSolutionsGroup
NSBMethodology:Vnfperformancebenchmarking
*VIL – VNF Infrastructure Libraries includes DPDK Optimized NFVi with OVS-DPDK
* Not a commercial product, Not Vendor/Proprietary Workloads
SHVServerPlatform
Hypervisor
VNF1
vPE
TestHarness
OpenvSwitch
VNF2
vEPC
VNF3
vFW
OptimizedNFVi
VIL*Enhancements
Fd.io
• Developed Open Source approximations of
Telco grade VNF’s using optimized VNF +
NFVi Infrastructure libraries
• VNFs: vCG-NAT, vACL, vPE, vFW, vSAE-GW
• Performance Characterization of Sample
Traffic Flows using open source Test
Harness
Facilitate Deterministic and Repeatable
benchmarking on Industry SHV Servers

6. 6
DatacenterSolutionsGroup
NSBMethodology:Vnfperformancebenchmarking
VNF performance
benchmarking
Native Linux environment
Standalone Virtualized
environment
Managed virtualized
environment (e.g.
OpenStack)
Evaluate both scale-
up and scale-out
performance data
VNFs performance graphs
for both scale-up and
scale-out in all three
environments
Collect KPIs:
Network KPIs, VNF
KPIs and NFVi KPIs
Test Infrastructure:
Standard test
framework for all 3
environments

7. 7
DatacenterSolutionsGroup
NSBTestHarness
Based on OPNFV
Yardstick Open
Source code
Enhanced with TREX
Open Source Traffic
generator,
Integration with Ixia,
Spirent traffic gen
Enhanced with Intel
developed features for
VNF benchmarking
Plug-ins to interface VNF’s to
test harness for
benchmarking
Features additional
to Yardstick
Test Harness open
sourced back into
OPNFV Yardstick

8. 8
DatacenterSolutionsGroup
NSB–OpenSourceandStandardSSTRATEGY
Test
Specifications
and Methodology
Open Source
ImplementationsOpenSource
Projects
Standards
Organizations
NSB
UpstreamContributionsand
DownstreamConsumption
Test Harness
Reference VNF’s
Test Methodology and KPI’s
SampleVNF:
vACL, vCG-NAT, vFW, vPE, vSAE-GW
Test Methodology
and KPI’s
Yardstick: Test Methodology and KPI’s
Test Harness
Service Providers, OEMs, ISV’s, OSV’s, SI’s

9. 9
DatacenterSolutionsGroup
SUT Management
and Orchestration
VIM
Benchmark Control Application
Ve-Vnfm
Nf-Vi
BMConfigAPI
KPIAPI
VNFM
NFVO
VNFI
Network Service
Under Test
Network Infrastructure
Under Test
OpenO
OSM
Tacker
Open Stack
MANO
Os-Ma
BM Deploy API
BM Provision APIVSPERFTrafficGenAPI
SUT
FastPathQM
ACL
VNFC
FW
VNFC
NAT
VNFC
Traffic
Verifier
Yardstick Test Framework
VSPERF Test Framework
CI Framework
DPDK, FDIO, OVS, PNF
e.g. VPP, IP pipeline
Traffic Profiles
Test Cases
Test Results
Functest APIs
NSB API Definitions
1. Minimum API
2. NS Extensions
Test Dashboard
BM Config API
BM KPI API
NSB Logical Model
With MANO
NSB Physical Model
No MANO
VSPERF Traffic Gen API
NSB-Network
Services
Benchmarking
Framework
ROADMAP
WITHMANO
Traffic
Generator

10. 10