The document outlines the Okinawa Open Laboratory's (OOL) mission to promote advanced ICT technologies, notably software-defined networking (SDN) and cloud computing, in support of regional industrial development. It details the development of a VNF (Virtual Network Function) automation testing system to address the challenges of VNF testing, such as performance variability and lack of standardization. The document also highlights the collaborative efforts of various stakeholders to enhance VNF interoperability and test automation capabilities in the emerging market.

1. June 20–23, 2016 | Berlin, Germany

2. Automated Platform for
Testing VNF Performance and Interoperability with
Variable Flavors
Takashi Torii NEC Corporation
Tatsuya Tohyama, Okinawa Open Laboratory/OKIT

3. Introduction of
Okinawa Open Laboratory(OOL)

4. Vision: Okinawa Smart Hub
• Okinawa Prefectural Government vision and policy for industrial development
and promotion
• Attract companies, talents and knowledge from other prefectures in Japan and overseas,
especially Asian countries, and promote new value creation in Okinawa
• Form major international ICT hub in Asia

5. Okinawa Smart Hub and OOL
• Contribute to realizing Okinawa Smart Hub concept
• Based in Okinawa with support from Okinawa Prefectural Government, attracting leading
companies, organizations, educational institutions, and research bodies from other
prefectures in Japan and overseas, especially Asia, and promoting international R&D
activities
• Contributes to promoting concentration of ICT industries and advancement of regional
companies
SDN technology research and
standardization base
OSS cloud development community
Leading Linux and OSS organization
OSS SDN development project
Okinawa Open Laboratory
Practical-use research base that integrates
cloud technology and SDN technology
International
R&D organization
Leading companies and organizations
from other prefectures in Japan
and overseas, especially Asia
Technology
cooperation
Cooperation, attraction
Activity support
・Support for R&D activities
・Promotion of cloud DC facility use

6. Mission
• Research and development activities for practical application and dissemination of
advanced Information Communications Technologies (ICT), especially next-generation
ICT platform technologies, such as
• Software-Defined Networking (SDN)
• Cloud Computing using Open Source Software (OSS, especially Open Stack)
• Open Innovations
• Collaboration, Joint Work
• Bridging the gaps between users and providers

7. Activity Model
• OOL connects OSS development communities and market via use cases and
reference models to expand user base of next-generation ICT platform
technologies in cooperation with MEMBER and partners
• OOL Value Chain: Use Case à Verification à Ref. Model à Promotion
• Feedback Loops: (1) OOL ó Market, (2) OOL ó OSS Dev. Communities
OSS Bodies (Community/Consortium)
Develop
Use Case Verify Promote
Issues
/Needs
Apply
/Use
OOL and its Members
Market (Providers and End Users)
Expand User Base
Reference
Model
Testbed (OpenStack with SDN)

8. Member List
Board
(5)
Member
(29)
Fellow
(14)
Total (48)

9. Project Formations
Full Time Engineers
Organizer
Companies
Research Institute
Academia
• Planning projects and goals
• Communicate with companies
• Communicate with Boards, Staffs and
Stakeholders (Government etc)
• Manage the activities
• Promote and collaborate
Staffs Board

10. Dev Project Example: OF-Patch
• Patch panel function implemented using OpenFlow
technology
OFPM
User
Change network
topology by
remote operation
OF-Patch
GUI
OFC
OFS Patch by flow
Host1 Host2 Host3

11. OPNFV Test Lab
• VNF Test Automation
• Service Function
Chaining
• Joint POC with oversea
members
Internet
Jump Server
POD#1
(Fuel)
POD#2
(RDO)

12. VNF Test Automation Project

13. Background
• Market of VNF is now emerging
• ETSI proposed NFV architecture
• Open Source VNF
• VyOS (Vyatta）
• Open vSwitch
…etc
• Vender VNF
• Cisco CSR1000V
• Juniper FireFly, vSRX
• Brocade Vyatta Core
…etc
• But we don‘t have test automation platform of VNF

14. Background
• Problem of VNF testing
• There is no standard
• Performance and stability is depend on flavors
• Many test patterns are required than H/W appliance
• Impossible to cover all test patterns by manual
• Risk of human error or miss configuration
• Configurations are depend on vendors
• Need support from venders or skilled engineers in all phase
• To solve these problems, we have developed VNF automation
testing system

15. How to test VNF?
• Workflow
• Deploy
• Deploy VNF on VIM with flavor
• Initialize VNF
• Configuration
• Configure and confirm
• Check the network connectivity with other devices
• Run
• Subscribe test patterns and run on traffic generator.
• Check VNF status and behavior
• Reporting
• Writing report based on result

16. Test workflow
Test
Deploy
VNF
Initialize
Initialize
VNF
Config
Delete
VNF
Log
Archive
Analysis
Report
Check

17. Test workflow
• Benefit of automation
• Engineer can focus on troubleshooting
• Engineers who don’t have much skills can operate the tests
• High repeatability
Repeat
Test
Deploy
VNF
Initialize
Initialize
VNF
Config
Delete
VNF
Log
Archive
Analysis
ReportCheck

18. Architecture
PoC User
Orchestrator
Web-Portal
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr
(Deploy of VNF appliance)
NW Device Ctr
(Insert/delete of configure)
Tester Ctr
(Control of IXIA)

19. Architecture
PoC User
Orchestrator
Web-Portal
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr
(Deploy of VNF appliance)
NW Device Ctr
(Insert/delete of configure)
Tester Ctr
(Control of IXIA)

20. Web Portal
• Set the test parameters
• Select the test scenario from drop down lists
Orchestrator
Web-Portal
REST API
PoC User
Test Parameter

21. Architecture
PoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr
(Deploy of VNF appliance)
NW Device Ctr
(Insert/delete of configure)
Tester Ctr
(Control of IXIA)
Web-Portal

22. Orchestrator
• Controller management
• Controls each components based on the configuration of test scenarios.
• Creating test scenario
• Create test scenarios by GUI
• Reporting
• Output PDF format
Scenario generate tool

23. Architecture
PoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr
(Deploy of VNF appliance)
NW Device Ctr
(Insert/delete of configure)
Tester Ctr
(Control of IXIA)
Web-Portal

24. Architecture
PoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr
(Deploy of VNF appliance)
NW Device Ctr
(Insert/delete of configure)
Tester Ctr
(Control of IXIA)
Web-Portal

25. Virtual Appliance Controller
• Multi virtualized platform control
• Deploy virtual appliance on selected virtualized platform.
Orchestrator
REST-API
REST-Message
OpenStack Link
Virtual Appliance Controller
REST-API REST-API

26. Architecture
PoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr
(Deploy of VNF appliance)
NW Device Ctr
(Insert/delete of configure)
Tester Ctr
(Control of IXIA)
Web-Portal

27. NW Device Controller
• Multi vendor control
• Provides the ability to control multiple vendor VNF
Orchestrator
NW Device Controller
REST-API
Telnet/SSH/Netconf
REST-Message
Parameter Ex
・Vender Name
・Vender Version
・Test scenario
other…

28. Architecture
PoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr
(Deploy of VNF appliance)
NW Device Ctr
(Insert/delete of configure)
Tester Ctr
(Control of IXIA)
Web-Portal

29. Tester Controller
• Generating Tcl script
• Based on the parameters set by REST API from Orchestrator
• Ixia Driver
• Load Tcl Script by IXIAAPI.
Orchestrator
Tester Controller
REST-Message
Parameter Ex
・Interface Information
・Packet Size(Short,Long,Imix)
・Frame rate
other...
Test Traffic
IXIA API
REST API
Tcl Script

30. Test workflow
Repeat
Virtual Appliance Controller
Tester ControllerNW Device Controller
Orchestrator
Test
Deploy
VNF
Initialize
Initialize
VNF
Config
Delete
VNF
Log
Archive
Analysis
Report
Check

31. Our environment
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600

32. Our environment for interoperability
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600

33. Our environment for interoperability
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
VNF appliance test
environment
VNF appliance test
environment

34. Our environment for interoperability
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
VNF appliance for
interoperability testing
VNF appliance for
interoperability testing

35. Our environment for interoperability
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600

36. Our environment for Traffic Test
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600

37. Our environment for Traffic Test
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
VNF appliance test
environment
VNF appliance test
environment

38. Our environment for Traffic Test
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
Traffic Generator

39. Our environment for Traffic Test
VyOS
vSwitch
vnic0
vSwitch
InteroperabilityL2SW
TrafficTestL2SW
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600
vnic0
vSwitch
vnic1
VyOS
Juniper
Firefly
Cisco
CSR1000v
Vyatta
5600

40. Demonstration (Movie)

41. Result of test automation
• Enables multiple patterns of test runs which are difficult by manual
Automate the combination test of 2736 patterns by 40 hours
【CSR1000v,Firefly,Vyatta,VyOS】VNF appliance
【VMware ESXi, OpenStack】VIM
【CPU,Mem,HDD,VNIC】
【CSR1000v,Firefly,Vyatta,VyOS】Interoperability with reference router
【VMware ESXi, OpenStack】VIM of reference router
【BGP:5 Pattern,OSPF:7 Pattern】
【Test Pattern:3 Pattern ,FrameRate:6 Pattern】
Flavor of VNF Appliance
Routing protocol
Traffic test pattern

42. Current Activities
• We have just joined OPNFV, and deploy Community Lab.
• Refactoring to align NFV Architecture of ETSI
• Using TOSCA based orchestrator (we are considering Tacker)
• Collaborate with Functest

43. TesterTester
Collaborate with Functest
Web Portal
Functest
Web Portal
VNF
Tester
VNFM
VNF
VNF
Functest

44. Any Questions?