Downloaded 23 times
![Self-Designed Test Tool
更新日期：
2010/2/10
NBL開發工具/系統目前已授權提供多家國內廠商
適合的待測物類型
(DUT)
工具/系統名稱(Tool
Name)
與預估開發人力
規格/特色
(SPEC & Features)
應用說明
(Description)
工具類型測試類型
SOHO Router
DSL Router
IAD Gateway
Security Appliance
In-Lab Live
Test(ILLT)
(6人月)
1. HW: PC
2. SW: Java and C programs
3. Continuously increasing real-world traffic
4. Support Bridge/Router/NAT mode
5. Automatically check the status of DUT
6. Stateless and Stateful Replay
[ 穩定性測試與除錯]
利用錄製、播放流量的技術，完整呈現真
實網路環境。在實驗室內部再造真實網路
流量環境，與Field Test相較之下，在測試
上更具彈性、網路流量涵蓋範圍更大、並
且更容易重製Bug。
流量產生、
結果解析、
自動化
Stability
Security Appliance
UTM
Anti-Virus
IPSP2P/IM
Network Forensics
PCAP Library
(3人月)
1. HW: PC
2. SW: Java and C progrmas
3. A database of PCAP files
4. Continuously increasing real-world traffic
5. Classified netowrk traffic such as P2P,
Attack
[ 網路流量資料庫]
流量資料庫，儲存各種未分類與已分類的
PCAPs。目前分類的類別包括P2P、Attack、
HTTP、SMTP、POP3、FTP等。
[ 協助漏判(FN)與誤判(FP)情況的改善]
可應用在"找出可能造成產品漏判與誤判
的網路流量"，進一步提供這些可疑的網路
流量(i.e. PCAP files)給研發人員參考，方
便找出其特徵值或行為模式。
流量產生、
結果解析、
自動化
Accuracy
QoS
SOHO Router
Security Appliance
UTM
Anti-Virus
IPSP2P/IM
Network Forensics
APP-Test
(5人月)
1. HW: PC
2. SW: Java and C programs
3. 支援多種Applications:
HTTP、FTP、Mail、SIP/RTP、IM、P2P
、Skype、Streaming (ex. Youtube、MLB
、NBA)
4. 自動分析DUT所產生的Log資訊來判斷
測試結果
[自動產生各種類型的應用]
透過使用者端應用軟體，自動產生各類應
用層網路流量，例如測試QoS時，可產生
各類streaming應用，檢查各streaming應用
是否有達到頻寬的保證。
[自動解析測試結果]
從各類待測物的output中(ex. Log)收集相關
資訊，來判斷本次的測試結果為Pass or
Fail
流量產生、
結果解析、
自動化
Functionali
ty](https://image.slidesharecdn.com/introductiontonbl-141122230613-conversion-gate01/85/Introduction-to-NBL-7-320.jpg)
![Demo: OAM Web System
[Demo URL] http://140.113.243.175/login.php
56
Login Webpage &
Dashboard Webpage](https://image.slidesharecdn.com/introductiontonbl-141122230613-conversion-gate01/85/Introduction-to-NBL-56-320.jpg)
Uploaded byFei Ji Siao
Introduction to NBL
This document discusses the development of test tools and RealFlow testing at a third-party testing lab. It describes various self-designed test tools developed to automate testing of devices like routers, switches, and wireless products. It also outlines how RealFlow testing uses real network traffic replay techniques to test products under realistic conditions captured from a beta testing site. Finally, it mentions some experiments conducted with Software Defined Networking and OpenFlow technologies.
More Related Content
Introduction to NBL
- 1.
- 2. Agenda • Introduction • Tool Development • RealFlow • Software Defined Network and OpenFlow 2
- 3. Introduction • 3rd-PartyTesting Lab • Self-designed Tools • RealFlow Test • Research Team • Look forward for the future 3
- 4. 4 Test Coverage Area DUT/FUT Test Coverage Bridging and Routing Bridging and Routing WLAN and VoIP Ethernet L2/L3 Switch Functionality, Conformance, Stability, RFC 2544/2889 NCSec UTM, Anti-Virus, IPS, SSL VPN, IPSec VPN, P2P/IM Management Functionality, Interoperability, Stability, Session Capacity and Rate, Accuracy WLAN and VoIP SOHO Router, DSL Router, IAD Gateway, SIP Phone, SIP Gateway, SIP Proxy, Access Point Voice Quality, Mobility, Functionality, Interoperability, Stability, Session Capacity and Rate
- 5. 5 Public BenchmarkEvent 2001 網路安全閘道器 網路頻寬管理器 網頁交換器 網路服務品質 Wireless LAN SOHO路由器 2002 VoIP產品 電子商務網站 2004 無線區域網路產品 網路安全閘道器 Internet內容遞送 2010 Network Anti-botnet Solution Benchmarking 2005 VoIP互通性插拔大會 網路/內容安全產品 2006 入侵防禦系統 10GbE Ethernet Switch VoWLAN 產品的語音品質 2007 P2P Friendly Properties of NAT Wireless SIP Residential Gateways 2009 SOHO NAT於真實網路流 量下的穩定性表現 Benchmarking, Workshop, Publish 2013 International SDN Symposium Taiwan
- 6.
- 7. Self-Designed Test Tool 更新日期： 2010/2/10 NBL開發工具/系統目前已授權提供多家國內廠商 適合的待測物類型 (DUT) 工具/系統名稱(Tool Name) 與預估開發人力 規格/特色 (SPEC & Features) 應用說明 (Description) 工具類型測試類型 SOHO Router DSL Router IAD Gateway Security Appliance In-Lab Live Test(ILLT) (6人月) 1. HW: PC 2. SW: Java and C programs 3. Continuously increasing real-world traffic 4. Support Bridge/Router/NAT mode 5. Automatically check the status of DUT 6. Stateless and Stateful Replay [ 穩定性測試與除錯] 利用錄製、播放流量的技術，完整呈現真 實網路環境。在實驗室內部再造真實網路 流量環境，與Field Test相較之下，在測試 上更具彈性、網路流量涵蓋範圍更大、並 且更容易重製Bug。 流量產生、 結果解析、 自動化 Stability Security Appliance UTM Anti-Virus IPSP2P/IM Network Forensics PCAP Library (3人月) 1. HW: PC 2. SW: Java and C progrmas 3. A database of PCAP files 4. Continuously increasing real-world traffic 5. Classified netowrk traffic such as P2P, Attack [ 網路流量資料庫] 流量資料庫，儲存各種未分類與已分類的 PCAPs。目前分類的類別包括P2P、Attack、 HTTP、SMTP、POP3、FTP等。 [ 協助漏判(FN)與誤判(FP)情況的改善] 可應用在"找出可能造成產品漏判與誤判 的網路流量"，進一步提供這些可疑的網路 流量(i.e. PCAP files)給研發人員參考，方 便找出其特徵值或行為模式。 流量產生、 結果解析、 自動化 Accuracy QoS SOHO Router Security Appliance UTM Anti-Virus IPSP2P/IM Network Forensics APP-Test (5人月) 1. HW: PC 2. SW: Java and C programs 3. 支援多種Applications: HTTP、FTP、Mail、SIP/RTP、IM、P2P 、Skype、Streaming (ex. Youtube、MLB 、NBA) 4. 自動分析DUT所產生的Log資訊來判斷 測試結果 [自動產生各種類型的應用] 透過使用者端應用軟體，自動產生各類應 用層網路流量，例如測試QoS時，可產生 各類streaming應用，檢查各streaming應用 是否有達到頻寬的保證。 [自動解析測試結果] 從各類待測物的output中(ex. Log)收集相關 資訊，來判斷本次的測試結果為Pass or Fail 流量產生、 結果解析、 自動化 Functionali ty
- 8. Self-Designed Test Tool(cont.) 適合的待測物類型 (DUT) 工具/系統名稱(Tool Name) 與預估開發人力 規格/特色 (SPEC & Features) 應用說明 (Description) 工具類型測試類型 SSL VPN SSL VPN Tunnel Capacity (2人月) 1. HW: PC 2. SW: Java and C programs 3. Generate more than 500 SSL VPN Tunnels in a single PC 4. Support background traffic in the tunnel such as HTTP, CIFS 測試SSL VPN Tunnel數量，使用一台PC即 可建立超過500條以上的Tunnel，並且在每 條建立起來的Tunnel中可傳送應用層流量， 如網頁的瀏覽、網路芳鄰抓檔。 流量產生、 結果解析、 自動化 Performanc e PPTP/L2TP VPN PPTP/L2TP VPN Tunnel Capacity (2人月) 1. HW: PC 2. SW: Java and C programs 3. Generate more than 500 PPTP/L2TP VPN Tunnels in a single PC 4. Support background traffic in the tunnel such as HTTP, CIFS 測試PPTP/L2TP VPN Tunnel數量，使用一 台PC即可建立超過500條以上的Tunnel，並 且在每條建立起來的Tunnel中可傳送應用 層流量，如網頁的瀏覽、網路芳鄰抓檔。 流量產生、 結果解析、 自動化 Performanc e Switch IGMP Snooping (2人月) 1. HW: SmartBits 2. SW: Tcl/Tk programs 3. Support IGMP v1/v2/v3 測試switch在IGMP snooping的功能是否正 確。流量產生、 結果解析、 自動化 Functionali ty General MIB-AutoChecker (3人月) 1. HW: PC 2. SW: Net-SNMP and Perl programs 3. MIB Object Accessibility 4. MIB Definition Customization 5. SNMP v1, v2c and v3 Support 即自動化MIB 物件存取技術。 MIB AutoChecker 可匯入自訂ASN.1的物件 檔案，它以Net-SNMP 動作為基礎，自動 地展開指定的數個群組的所有物件並加以 存取，記錄下所有訊息使得測試者很快就 找出有問題的部份。 以Net-SNMP為基礎，可以在MIB中指定的 結點與其下的sub-tree自動化存取物件。 自動化Functionali ty
- 9. Self-Designed Test Tool(cont.) 適合的待測物類型 (DUT) 工具/系統名稱 (Tool Name) 與預估開發人力 規格/特色 (SPEC & Features) 應用說明 (Description) 工具類型測試類型 Switch/Router ANVL-MultiParser (3人月) 1. HW: PC 2. SW: ANVL and Perl programs 清楚解析出ANVL的測試結果，讓研發人員 可以快速找出產品的問題點。結果解析Conforman ce Switch/Router Performance Test Suites(PTS) (3人月) 1. HW: SmartBits 2. SW: Tcl/Tk and PHP programs 3. 圖型化使用者介面：DUT Configurator、 SmartBits Configurator、Test Editor、Test/ Result Viewer 4. 自動化組態待測物： 5. 自動化收集測試結果及產生測試報告 Layer 2/3 Switch的效能測試自動化。 自動化Performanc e Switch/Router Auto-ANVL (3人月) 1. HW: PC 2. SW: ANVL and Perl programs 3. 圖型化使用者介面：DUT Configurator、 IxANVL Configurator、Test Editor、Test Viewer、Result Viewer、Preference 4. 自動化組態待測物 5. 自動化執行Test Suite及測試案例 6. 自動化剖析測試記錄檔及分析結果 7. 自動化產生測試報告 Layer 3 Switch的符合性測試自動化。 自動化Conforman ce SOHO Router DSL Router WebUIAutoChecker (2人月) 1. HW: PC 2. SW: Java programs 3. Web GUI invalid value auto check 4. Suppoer Firefox and IE Web GUI 自動檢測工具，將Web GUI輸入 各種錯誤值的測試自動化，提高測試效率 與準確性。自動化Functionali ty
- 10. Self-Designed Test Tool(cont.) 適合的待測物類型 (DUT) 工具/系統名稱(Tool Name) 與預估開發人力 規格/特色 (SPEC & Features) 應用說明 (Description) 工具類型測試類型 WiFi Phone WLAN Card Access Point WLAN Mobility Test System(WMTS) (6人月) Mobility Test Scenarios: 1. HW: Azimuth 2. SW: Tcl/Tk 3. Distance Test for Computer Adapter 4. Distance Test for Mobile Phone 5. Motion Adaptation Test for Computer Adapter 6. Motion Adaptation Test for Mobile Phone 7. Motion Roaming Test for Computer Adapter 8. Motion Roaming Test for Adaptation for Mobile Phone 9. Switch Roaming Test for Computer Adapter 10. Switch Roaming Test for Adaptation for Mobile Phone Programmable Channel Model: 11. Free Space and Typical Home/Office/Street Channels 12. More channels can be added by the programmable parameters Traffic Generation/Analysis: 1. HW: Abacus and PC 2. SW: IxChariot and Perl programs 3. Ixia IxChariot® (with the scripts and QoS selective) 4. Abacus® for PESQ/PSQM analog testing 5. NBL traffic analyzers for link adaptation and re- association 自動化WLAN移動測試系統（WMTS）可 進行Roaming的延遲分析。提供具有可參數 化channel模擬與WMM支援的Distance Adaptation和Roaming測試Scenarios For Distance, Adaptation, and Roaming scenarios with a parameterized channel approximation and WMM support The WLAN Mobility Test System (MTS) for Azimuth W-Series Platform® version 1.3 can support 8 novel mobility test scenarios and integrate standard traffic utilities like IxChariot® and VQT®, including QoS enabled configuration. A programmable channel model is introduced for the mobility control to offer the approximation of signal fading for the in-lab wireless testing, which is more stable and repeatable than the outdoor testing. 結量產生、 結果解析、 自動化
- 11. from Lab Testto RealFlow Test 11
- 12. Solutions 12 真實網路 流量測試 Real Network •Users •Beta Site Replay Tech. •Capture •Replay Test System •PCAP Library •In-Lab Live Test (IL2T) RealFlow
- 13. Beta Site Zone 1 End-user software Zone 2 Ethernet L2/L3 Switch Wireless AP Zone 3 Core Router Zone 4 (Inline, one-in-one-out) UTM, IPS, Anti-Virus, QoS Firewall Zone 5 (Sniff) Network Forensic Anti-Malware/Botnet Zone 6 (ILLT) SOHO Router, Home Gateway Broadband Gateway DSL Router, IAD Gateway 13
- 14. RealFlow Certification •Function •Performance Phase 1 Phase Stability 2 14 USG 2000 ZyXEL Security Appliance DIR-300 D-Link SOHO Router DIR-615C1 D-Link SOHO Router Mini guard Lionic Security Appliance DIR-655 D-Link SOHO Router
- 15.
- 16. Service Solution •Solution Prototype • SDN Enabled Wi-Fi Tool • Test Tool develop and Automation • Cloud based conformance test tool based on oftest • SDN Enabled Testbed Service • Test/Benchmark of SDN • Device and Apps • OpenFlow Certification 16
- 19.
- 20. Why SDN -Requirement • Too many different kinds of devices with different operation method for 1 or 2 persons • Devices with high cost • Impossible to fully match the campus network administration policy
- 21. What solution weneed? • An unify control message and method to integrated different devices from different vendors • An easy way to develop the related service/software and deploy the suitable devices with suitable spec. and cost • An flexible method to integrate the other services and devices
- 22. Advantage of OpenFlow(Manage) • Open source with low cost • A generic protocol to control different kinds of devices • The network administrator could choose the suitable HW devices and policy/SLA for each site and account(person) • Don’t need to add more human resource to manage more devices (cost again)
- 23. Advantage of OpenFlow(Technical) • Control the forwarding table of switch/router and wireless devices from different vendors with the same message • Control the forwarding policy by the software development of each site with different rule. • Easy to control the traffic to integrated with other services
- 24.
- 25. Campus Wi-Fi Solution(cont.)
- 26.
- 27. Example: System control Web UI RYU AP REST APP OpenFlow OpenFlow CURL OpenFlow 1.3 Experimenter message RESTAPI POST
- 28. Feature List •Auto-Provisioning and Configuration through experimenter message • Multi-path routing/switching with fail-over and switch-over • Switch and AP control and management • Multicast Routing • Streaming Control • Wi-Fi AP-STA Association Load Balance 28
- 29.
- 30. SDN Test (concept) APP/Controller Topology Generator •Automatically •Physical devices •Simulator Monitor •Flow Checker (App for Testing) Traffic Generator Test bed •Trackable mechanism Test Script •Test Methodology •Test Plan Test Controller Test Portal Test bed Report Generator Monitor Test Script Topology Generator Traffic Generator
- 31. SDN Test (Laband Auto) SDN Network Device 1. Execute Automation Test Program Monitor System information collector CPU Loading, Link utilization, counters, client simulator Packet Generator Scriptable Error packets Attack traffic Trackable packets and mechanism* L4+ level packet generator Topology generator GUI based traffic tracker Lab test automation SDN-enabled Layer 1 switch Flow entry add/remove Controllable VM migration Test plan/case/script/reporter* SDN-enabled Layer 1 switch 2. Topology Generator generate a topology 3. Monitor Start to gather information DUT & SUT 4. Start to generate the required packets 5. If any Link Needs to be Removed During Test Period, the L1 Switch can Do it.
- 32. SDN Test (BetaSite) Traffic Reproducer Real-time time machine Reproduce defect of Apps Reproduce traffic Reproduce topology by simulator and emulator Lab Test Automation Flow entry add/remove Controllable VM migration Test plan/case/script/reporter* 1. Execute Automation Test Program Topology Generator 2. AuGtUoIm baatsiceadl ltyr aGfafitch terra cthkee r Topology Information 3. Monitor Start to gather information DUT & SUT Monitor System information collector CPU Loading, Link utilization, counters, client simulator 4. Traffic Start SDN Network Device 5. 1 Capture traffic 5. 2 Replay traffic to reproduce bugs PKT Packet Genera tor
- 33.
- 34.
- 35.
- 36. 基於SDN和Cloud架構之 無線/寬網技術與服務 SDN-enabledCloud-based Wireless and Broadband Network Technologies & Services
- 37. Project Goal •Establish the first end-to-end test platform for SDN in Asia • Contribute to the open networking (SDN related) standards and set up international reputation • Cultivate human talents with SDN related technology and service knowledge and experience to build an SDN industry in Taiwan • Strengthen SDN IP portfolio for Taiwan industry • Enhance the competitive advantage of Taiwan ICT industry
- 38. Technology Layered Architecture 38 ①CHT (Wireless/BB/Security) ②CHT (Cloud/Billing/IoT) ③D-Link ④ IoT/M2M ② Cloud Technologi es ② ④ ⑧ Northbound API ①③④⑥⑧ Controller Multi-controller ④ Southbound API Switch Testbed and Methods ④ Campus Field Trial ①②③④ Northern TW Field Trial ①② ③④⑥⑧ Service APP OA&M Network APP Security ④⑧ Performance ④ Accounting ① Configuration ①③⑥ Fault EstiNet ⑤MediaTek ⑥ Arcadyan ⑦ Xinguard ⑧ Inventec Enterprise/ Campus ③ ⑥ ⑧ Broadband Technologi es ① ⑦ Network Virtualization V&T SDN Chip ⑤ SDN Switch OS ④ Wireless Technologi es ① Smart Data Pricing ② Common controller ①
- 39. Subproject E –Integration and Field Test 39 小型SDN環境 (小型OpenFlow Switch) Internet OpenFlow WiFi 有線網路 無線網路 OpenFlow CPE 終端SDN環境 雲端資料中心 雲端SDN環境 x4 x2 行動網路 UE TL Core Network 小型SDN環境 (小型OpenFlow Switch) 中型SDN環境Internet (中型OpenFlow Switch ) OpenFlow WiFi 有線網 路 無線網 路 SDN Controllers OpenFlow CPE 終端SDN環境 雲端資料中 心 x4 x2 雲端SDN環境 行動網 路 UE NCTU Core Net. emulator 小型SDN環境 (小型OpenFlow Switch) Internet SDN Core Switch (中型OpenFlow Switch ) OpenFlow WiFi 有線網 路 無線網 路 SDN Controllers OpenFlow CPE 雲端資料中 心 x4 x2 雲端SDN環境 CHT 中型SDN環境 (中型OpenFlow Switch ) NCTU NTHU SDN Master Controllers SDN Controllers 中型SDN環境 (中型OpenFlow Switch ) Internet SDN APP SDN APP SDN APP
- 40. SDN Enabled Wi-FiSolution 40
- 41. Outlines • SystemArchitecture • Topology Example • Environment Description • Features vs. Components • Vendor-Defined Feature Specification • Feature: Auto Provision Function • Feature: Authentication Portal Function • Demo: OAM Web System 41
- 42.
- 43.
- 44. Environment Description (1/3) • SDN Controller Ryu SDN Framework 3.8 Ubuntu 12.04 • SDN Access Point Device: TPLink TL-WR1043ND V2, AP222, … OS: OpenWRT trunk Patch: ofsoftswitch13 • SDN OAM Server / SDN Portal Server PHP 5.5 Apache 2.4.10 • SDN Database Server MySQL 5.5
- 45. Environment Description (2/3) • Ryu SDN Framework Python-based Project Written in Python language Environment OS: Ubuntu 10.04 or higher Network simulation tool: Mininet Features provided Component-based SDN framework Support OpenFlow v1.0, v1.2, v1.3, v1.4 Support various protocols for managing network devices, such as OpenFlow, Netconf, OF-config, etc. Provide REpresentational State Transfer (REST) service 45
- 46. Environment Description (3/3) REpresentational State Transfer (REST) Service 46 Controller Ryu App data in JSON format REST API Operations via HTTP request with URL. • GET • POST • DELETE • PUT PHP HTTP request OAM Web UI curl HTTP request Mininet (Simulated network) OpenFlow Switch (e.g., Access Point)
- 47. Features vs. Components SDN Controller Component SDN Access Point OAM App System Management Generic Wireless Control Auto Provision Function Authentication Portal Function Device Information Management Channel Non-overlapping Quality of Service Roaming 47
- 48. Vendor-Defined Feature Specification(1/2) • Vendor-defined features Implemented with “Experimenter Messages” Experimenter Message An optional field (padding) in OpenFlow protocol Provide SDN vendors to develop self-defined functions and services • Experimenter Message Format 48 Header: EXPERIMENTER_ID (8 bytes) EXP_TYPE (8 bytes) Data: EXP_SUBTYPE (1 byte) PAYLOAD (255 bytes) • The format should be defined in both controller and access points. • Corresponding handler function will be triggered according to the EXP_TYPE and EXP_SUBTYPE.
- 49. Vendor-Defined Feature Specification(2/2) 49 Experimenter Message Type and Subtype List Example
- 50. Feature: Auto ProvisionFunction (1/3) 50 System Scenario
- 51. Feature: Auto ProvisionFunction (2/3) 51 Trigger Proxy Mode of Configured AP Automatically Trigger/Close Proxy Mode of Configured AP Manually by Administrator
- 52. Feature: Auto ProvisionFunction (3/3) 52 Relay the Configuration Request from Unconfigured AP and Response
- 53. Feature: Authentication PortalFunction (1/3) 53 Topology Example
- 54. Feature: Authentication PortalFunction (2/3) 54 Message Flow
- 55. Feature: Authentication PortalFunction (3/3) 55 Demo Portal Webpage
- 56. Demo: OAM WebSystem [Demo URL] http://140.113.243.175/login.php 56 Login Webpage & Dashboard Webpage
Editor's Notes
- #15 在2008年一年間，有四十幾件產品送測NBL，一開始大部分都有不穩定的情況，後來有少數幾台日趨穩定。這些拿到RealFlow Certification的產品都是測試超過半年以上、日趨穩定