Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Summit 16: Automated Platform for Testing VNF Performance and Interoperability with Variable Flavors

1,554 views

Published on

VNF sizing is one of the big issues for users, integrators and also venders/providers of VNF. In legacy system, network functions are integrated in the systems with carefully sizing by estimated performance metrics, but in NFV, performance and metrics of VNF are depends on flavors. To decide which flavor is suitable for requirements, we have to know the basic data of each metrics with variable flavors. But there are many parameters, so automatic testing is necessary. We had developed the test automation system of VNF that can automatically conducting the sequence of tests with variable parameters such as number of core, memory size, interface speed, etc. We already created the test cases that evaluate BGP interoperability of virtual router and performance test of Cisco, Juniper, Brocade, and so on. We think the activity is similar to Yardstick, so we would like to collaborate with them. In this talk, we summarize our activities and development result, and introducing our latest plan to collaborate with OPNFV.

Published in: Technology
  • Be the first to comment

Summit 16: Automated Platform for Testing VNF Performance and Interoperability with Variable Flavors

  1. 1. June 20–23, 2016 | Berlin, Germany
  2. 2. Automated Platform for Testing VNF Performance and Interoperability with Variable Flavors Takashi Torii NEC Corporation Tatsuya Tohyama, Okinawa Open Laboratory/OKIT
  3. 3. Introduction of Okinawa Open Laboratory(OOL)
  4. 4. Vision: Okinawa Smart Hub • Okinawa Prefectural Government vision and policy for industrial development and promotion • Attract companies, talents and knowledge from other prefectures in Japan and overseas, especially Asian countries, and promote new value creation in Okinawa • Form major international ICT hub in Asia
  5. 5. Okinawa Smart Hub and OOL • Contribute to realizing Okinawa Smart Hub concept • Based in Okinawa with support from Okinawa Prefectural Government, attracting leading companies, organizations, educational institutions, and research bodies from other prefectures in Japan and overseas, especially Asia, and promoting international R&D activities • Contributes to promoting concentration of ICT industries and advancement of regional companies SDN technology research and standardization base OSS cloud development community Leading Linux and OSS organization OSS SDN development project Okinawa Open Laboratory Practical-use research base that integrates cloud technology and SDN technology International R&D organization Leading companies and organizations from other prefectures in Japan and overseas, especially Asia Technology cooperation Cooperation, attraction Activity support ・Support for R&D activities ・Promotion of cloud DC facility use
  6. 6. Mission • Research and development activities for practical application and dissemination of advanced Information Communications Technologies (ICT), especially next-generation ICT platform technologies, such as • Software-Defined Networking (SDN) • Cloud Computing using Open Source Software (OSS, especially Open Stack) • Open Innovations • Collaboration, Joint Work • Bridging the gaps between users and providers
  7. 7. Activity Model • OOL connects OSS development communities and market via use cases and reference models to expand user base of next-generation ICT platform technologies in cooperation with MEMBER and partners • OOL Value Chain: Use Case à Verification à Ref. Model à Promotion • Feedback Loops: (1) OOL ó Market, (2) OOL ó OSS Dev. Communities OSS Bodies (Community/Consortium) Develop Use Case Verify Promote Issues /Needs Apply /Use OOL and its Members Market (Providers and End Users) Expand User Base Reference Model Testbed (OpenStack with SDN)
  8. 8. Member List Board (5) Member (29) Fellow (14) Total (48)
  9. 9. Project Formations Full Time Engineers Organizer Companies Research Institute Academia • Planning projects and goals • Communicate with companies • Communicate with Boards, Staffs and Stakeholders (Government etc) • Manage the activities • Promote and collaborate Staffs Board
  10. 10. Dev Project Example: OF-Patch • Patch panel function implemented using OpenFlow technology OFPM User Change network topology by remote operation OF-Patch GUI OFC OFS Patch by flow Host1 Host2 Host3
  11. 11. OPNFV Test Lab • VNF Test Automation • Service Function Chaining • Joint POC with oversea members Internet Jump Server POD#1 (Fuel) POD#2 (RDO)
  12. 12. VNF Test Automation Project
  13. 13. Background • Market of VNF is now emerging • ETSI proposed NFV architecture • Open Source VNF • VyOS (Vyatta) • Open vSwitch …etc • Vender VNF • Cisco CSR1000V • Juniper FireFly, vSRX • Brocade Vyatta Core …etc • But we don‘t have test automation platform of VNF
  14. 14. Background • Problem of VNF testing • There is no standard • Performance and stability is depend on flavors • Many test patterns are required than H/W appliance • Impossible to cover all test patterns by manual • Risk of human error or miss configuration • Configurations are depend on vendors • Need support from venders or skilled engineers in all phase • To solve these problems, we have developed VNF automation testing system
  15. 15. How to test VNF? • Workflow • Deploy • Deploy VNF on VIM with flavor • Initialize VNF • Configuration • Configure and confirm • Check the network connectivity with other devices • Run • Subscribe test patterns and run on traffic generator. • Check VNF status and behavior • Reporting • Writing report based on result
  16. 16. Test workflow Test Deploy VNF Initialize Initialize VNF Config Delete VNF Log Archive Analysis Report Check
  17. 17. Test workflow • Benefit of automation • Engineer can focus on troubleshooting • Engineers who don’t have much skills can operate the tests • High repeatability Repeat Test Deploy VNF Initialize Initialize VNF Config Delete VNF Log Archive Analysis ReportCheck
  18. 18. Architecture PoC User Orchestrator Web-Portal Web-Portal Report generationManagement of Test REST API Interoperability Traffic Test REST API Virtual Appliance Ctr (Deploy of VNF appliance) NW Device Ctr (Insert/delete of configure) Tester Ctr (Control of IXIA)
  19. 19. Architecture PoC User Orchestrator Web-Portal Web-Portal Report generationManagement of Test REST API Interoperability Traffic Test REST API Virtual Appliance Ctr (Deploy of VNF appliance) NW Device Ctr (Insert/delete of configure) Tester Ctr (Control of IXIA)
  20. 20. Web Portal • Set the test parameters • Select the test scenario from drop down lists Orchestrator Web-Portal REST API PoC User Test Parameter
  21. 21. Architecture PoC User Orchestrator Web-Portal Report generationManagement of Test REST API Interoperability Traffic Test REST API Virtual Appliance Ctr (Deploy of VNF appliance) NW Device Ctr (Insert/delete of configure) Tester Ctr (Control of IXIA) Web-Portal
  22. 22. Orchestrator • Controller management • Controls each components based on the configuration of test scenarios. • Creating test scenario • Create test scenarios by GUI • Reporting • Output PDF format Scenario generate tool
  23. 23. Architecture PoC User Orchestrator Web-Portal Report generationManagement of Test REST API Interoperability Traffic Test REST API Virtual Appliance Ctr (Deploy of VNF appliance) NW Device Ctr (Insert/delete of configure) Tester Ctr (Control of IXIA) Web-Portal
  24. 24. Architecture PoC User Orchestrator Web-Portal Report generationManagement of Test REST API Interoperability Traffic Test REST API Virtual Appliance Ctr (Deploy of VNF appliance) NW Device Ctr (Insert/delete of configure) Tester Ctr (Control of IXIA) Web-Portal
  25. 25. Virtual Appliance Controller • Multi virtualized platform control • Deploy virtual appliance on selected virtualized platform. Orchestrator REST-API REST-Message OpenStack Link Virtual Appliance Controller REST-API REST-API
  26. 26. Architecture PoC User Orchestrator Web-Portal Report generationManagement of Test REST API Interoperability Traffic Test REST API Virtual Appliance Ctr (Deploy of VNF appliance) NW Device Ctr (Insert/delete of configure) Tester Ctr (Control of IXIA) Web-Portal
  27. 27. NW Device Controller • Multi vendor control • Provides the ability to control multiple vendor VNF Orchestrator NW Device Controller REST-API Telnet/SSH/Netconf REST-Message Parameter Ex ・Vender Name ・Vender Version ・Test scenario other…
  28. 28. Architecture PoC User Orchestrator Web-Portal Report generationManagement of Test REST API Interoperability Traffic Test REST API Virtual Appliance Ctr (Deploy of VNF appliance) NW Device Ctr (Insert/delete of configure) Tester Ctr (Control of IXIA) Web-Portal
  29. 29. Tester Controller • Generating Tcl script • Based on the parameters set by REST API from Orchestrator • Ixia Driver • Load Tcl Script by IXIAAPI. Orchestrator Tester Controller REST-Message Parameter Ex ・Interface Information ・Packet Size(Short,Long,Imix) ・Frame rate other... Test Traffic IXIA API REST API Tcl Script
  30. 30. Test workflow Repeat Virtual Appliance Controller Tester ControllerNW Device Controller Orchestrator Test Deploy VNF Initialize Initialize VNF Config Delete VNF Log Archive Analysis Report Check
  31. 31. Our environment VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600
  32. 32. Our environment for interoperability VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600
  33. 33. Our environment for interoperability VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600 VNF appliance test environment VNF appliance test environment
  34. 34. Our environment for interoperability VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600 VNF appliance for interoperability testing VNF appliance for interoperability testing
  35. 35. Our environment for interoperability VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600
  36. 36. Our environment for Traffic Test VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600
  37. 37. Our environment for Traffic Test VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600 VNF appliance test environment VNF appliance test environment
  38. 38. Our environment for Traffic Test VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600 Traffic Generator
  39. 39. Our environment for Traffic Test VyOS vSwitch vnic0 vSwitch InteroperabilityL2SW TrafficTestL2SW vnic1 vnic1 OpenStack (node2) VMware Esxi (node1) VMware Esxi (node3) OpenStack (node4) vnic2 vnic3 vnic1 vnic2 vnic3 MGMT-Seg vnic0 vnic0 vSwitch Juniper Firefly Cisco CSR1000v Vyatta 5600 vnic0 vSwitch vnic1 VyOS Juniper Firefly Cisco CSR1000v Vyatta 5600
  40. 40. Demonstration (Movie)
  41. 41. Result of test automation • Enables multiple patterns of test runs which are difficult by manual Automate the combination test of 2736 patterns by 40 hours 【CSR1000v,Firefly,Vyatta,VyOS】VNF appliance 【VMware ESXi, OpenStack】VIM 【CPU,Mem,HDD,VNIC】 【CSR1000v,Firefly,Vyatta,VyOS】Interoperability with reference router 【VMware ESXi, OpenStack】VIM of reference router 【BGP:5 Pattern,OSPF:7 Pattern】 【Test Pattern:3 Pattern ,FrameRate:6 Pattern】 Flavor of VNF Appliance Routing protocol Traffic test pattern
  42. 42. Current Activities • We have just joined OPNFV, and deploy Community Lab. • Refactoring to align NFV Architecture of ETSI • Using TOSCA based orchestrator (we are considering Tacker) • Collaborate with Functest
  43. 43. TesterTester Collaborate with Functest Web Portal Functest Web Portal VNF Tester VNFM VNF VNF Functest
  44. 44. Any Questions?

×