The document outlines the system testing and integration test strategy for the Brahmaputra project, detailing the testing ecosystem, coverage, and resources required for Release B. It emphasizes the need for automated test suites, comprehensive test case collection, and a centralized testing dashboard to manage various testing projects and dependencies. The strategy includes considerations for system-level performance, virtualized network functions, and multiple testing tools to ensure consistency and robustness.

1. System Testing & Integration
Test Strategy for Brahmaputra
Ana Cunha (Ericsson)
Morgan Richomme (Orange)
Trevor Cooper (Intel)

2. Agenda
• Testing ecosystem
• Test coverage
• Release-B Test plans

3. OPNFV Test Ecosystem
Community Labs
Functest (OPNFV Platform Functionality – VNFI & VIM)
MongoDB Database
Traffic profiles
Test Cases
Test Results
Test Community (Test and Performance Sub-Group)
Test Infrastructure (Pharos)
Linux Foundation Lab
Yardstick (Infrastructure Verification)
Qtip (Platform performance)VSPERF (Vswitch performance)
CPERF (Controller Performance)Bottlenecks (System limits)
STORPERF(Storage Performance)
Policy Test (Network Policies)
Test Dashboard KPIs
Dovetail (OPNFV
Compliance & Certification)
Test Framework
Test Tools Test Tools

4. Tools and Process
• Definitions & Test Coverage per domain/component
• Traffic Profiles
• Dashboard
Testing coordination
• Release B test requests
• Test Project Dependencies
Resource Planning
• Test Lab Requirements
• Test Project Lab Resources
OPNFV Test Activities
Weekly Test and Performance Meetings https://wiki.opnfv.org/meetings/test

5. Components
Test coverage – Domains and Components
Domains
System Level
Performance
OAM
Security
Robustness
Component,
Subsystem,
e2e, User/VNF
Component,
Subsystem,
e2e
Component,
Subsystem,
e2e
Component,
Subsystem,
e2e, User/VNF
Installation,
patch, upgrade,
Backup, recovery
OpenStack
ODL
KVM
OVS
Ceph
VNF
NIC
Memory
CPU
Storage
FUNCTEST,
YARDSTICK
FUNCTEST, VSPERF
QTIP, STORPERF,
YARDSTICK
YARDSTICK
From Testing wiki: Test Coverage
FUNCTEST
FUNCTEST
YARDSTICK
VSPERF, YARDSTICK
FUNCTEST,YARDSTICK
FUNCTEST,YARDSTICK
QTIP
QTIP, YARDSTICK
QTIP, YARDSTICK
QTIP, STORPERF, YARDSTICK

6. Test Coverage – ETSI-NFV Domains
NetworkStorageCompute
Hardware Resources
Virtualization Layer
Virtual
Compute
Virtual
Storage
Virtual
Network
NFV Infrastructure (NFVI)
VNF VNF VNF VNF VNF
Virtualized Network Functions (VNFs)
NFV
Management
and
Orchestration
From ETSI-NFV: GS NFV 002

7. Test Coverage – ETSI-NFV Domains
VSPERF, QTIP, STORPERF, YARDSTICK
Hardware Resources
YARDSTICK
VSPERF, QTIP, STORPERF, YARDSTICK
NFV Infrastructure (NFVI)
VNF VNF VNF VNF VNF
Virtualized Network Functions (VNFs)
FUNCTEST,
YARDSTICK
YARDSTICK
FUNCTEST
Note: Interfaces defined by ETSI-NFV are not shown in the diagram

8. • Lots of feature projects
• More test projects
• More installers
• More configurations
3 levels of dependencies
• installers
• labs
• feature projects
Release B
More challenging !

9. Release B: Ensure consistency
Lots of Feature projects
CI++: the test suite must be
automated and run on as many labs
as possible
Lots of configurations,
installers,…
Test collection API to collect:
- the test cases
- the test results
Lots of test projects
Companion test projects
e.g. VPN/Yardstick, Doctor/Functest

10. • An centralized view of all the test cases
• A user guide & and installation guide per Test project
• A developer Guide for OPNFV frameworks
• A Testing dashboard
Test Deliverables for Release B