1. Standards of Excellence
TestShell Framework
Introduction
Tuesday, December 20, 2011
Richard Hsu
QualiSystems Proprietary & Confidential
2. QualiSystems – Company Overview
• Global Software Company
•Santa Clara, USA
•London, UK
•Munich, Germany
•Tel-Aviv, Israel
•APAC
• Singapore , APAC HQ
• Shanghai, China
• Taipei, Taiwan
• Seoul, Korea
• APAC Channels, Australia, Japan , India
• Founded 2004
9001:2000 90003:2004
• Flag product – TestShellTM Test-Automation Framework
• “Customer First” Culture
QualiSystems Proprietary & Confidential
3. What we do - TestShell Framework
• End-to-End framework for Test Automation
that supports the entire life cycle of a test for components,
devices, or systems
• TestShell automates and simplifies the process of testing,
allows you to focus on pure testing, and thus substantially:
Improve Product Quality
Shorten Time to Market
Reduce Expenditures
QualiSystems Proprietary & Confidential
4. Testing Challenges without Automation
• Lengthy test processes that can take up to
several days
• The wide array of benchmark
applications/Scripts include Hardware and
Software Time Lost
• The variety of vendors and models requiring
Quality Impacted
diverse specifications
• The need to control and manage many Expense Overhead
distributed test stations
Resources
• Time-consuming test result collation and Unexploited
analysis
• Compound aggregation of extensive amounts
of test data
QualiSystems Proprietary & Confidential
5. Proven ROI within the first year of use
• Significantly reduce Time-To-Market
• Full regression cycle 64 hours → 24 hours at USI (環隆電氣)
• Single test cycle 15 days → 1.5 day at Cameo (友勁科技)
• Significantly reduce Operational Expenditures
• Resources involved in test creation down by 50% at Deutsche Telekom
• Single test cycle effort 2 Days → 5 hours at Telstra
• Managing and Improving Quality Control
• Daily Build and regression at Ceragon Networks
• Customer service centers at Crow
• Saving X10 on RMA at Syneron
QualiSystems Proprietary & Confidential
6. Our Customers – across the value chain
QualiSystems Proprietary & Confidential
7. End-to-End Test Automation Framework
Schedule Execute
Run Tests & Regressions
Coordinate Test Execution
Collect Logs & Results
Launch Test Stations
Monitor Real-time
Tests & Regressions
Assets
Create Equipment
Analyze
Test Results
Write Tests & Regressions Aggregate Results
Create reusable functions Central Customized Reports
Control Test Assets
Management Customized Dashboards
QualiSystems Proprietary & Confidential
9. End-to-End Test Automation Framework
Schedule Execute
Create Analyze
Central
Management
QualiSystems Proprietary & Confidential
10. TestShell System
Interfaces to Software and hardware compnents (Any many more..)
Scripts & • Exe, DLLs, CLI, VIs • Mail & SMS, File Engine
Software Utilities
• Script-Engines • GUI Automation, Attributes
• TCL, Perl, Python, VB, Java • Requirements
• MatLab, OCR Management, QTP ,LR , Etc..
Networking
• Telnet, SSH, TCP, UDP Mobile • Symbian
Modules
• SNMP (& Traps), FTP • Windows Mobile
• Traffic Generators, Switches • iPhone, Blackberry
• etc. • Android
T&M • Traffic generators • Messages, Instructions
Modules Dialogs
• Network analyzers • Alerts, Remarks
• Spectrum, scope • Watchers, Graphs
• Generators, etc. • etc.
Logic • Variables Web
Modules • Web GUI
• Cases, Loops, Delays Modules
• Web Services
• Parallelization, events
• etc.
• etc.
11. Testing in the All IP World - Example Scenario
Start • Typical test flow for validating :
• Allocate all testing resources
• Configure the Device Under Test
Setup Topology • Configure Traffic Generator(s)
• Applying traffic to the DUT and
Configure DUT sniffing /getting statistics
• Analyzing
Configure Traffic
Generator (ex. IXIA
, SPIRENT) • Typical tests:
• Traffic blasting, protocol emulation, load & stress,
conformance , acceptance, interoperability, etc
Run Traffic
Analyze Results • Typical validations:
• End-to-End: Path Recovery, Clock Recovery,
Delay, Latency, Throughput, Loss Ratio,
End Information-leakage, routing/switching performance, etc
QualiSystems Proprietary & Confidential
12. Testing in the All IP World - Example Scenario
Start
Multiple Network Elements from
different vendors such as:
Setup Topology
• Ethernet Switch
Configure DUT
• OSS
Configure Traffic • Wireless devices (Access Points)
• WiMAX/4G (STB)
Run Traffic
• Multi Service Router
Analyze Results • High Speed Router
• BBRAR
Store & Report Data
• DSLAM
• Gateway
End • Modem
QualiSystems Proprietary & Confidential
13. Testing in the All IP World - Example Scenario
Start
Various network protocols and
interfaces
Setup Topology
SSH Telnet SMTP
Configure DUT
SOAP TCP SNMP Traps
Configure Traffic
Run Traffic TFTP SNMP VISA
Analyze Results Serial SFTP GPIB
Store & Report Data
UDP HTTP FTP
End
QualiSystems Proprietary & Confidential
14. Testing in the All IP World - Example Scenario
Start Multiple test equipment types
from different vendors
Setup Topology
• Traffic Generators
Configure DUT • Network Emulators
Configure Traffic • Network Sniffers
• Network Analyzers
Run Traffic • Baseband Emulators
Analyze Results • Logic analyzers
• Signal analyzers
Store & Report Data
• Oscilloscopes
• Signal generators
End
QualiSystems Proprietary & Confidential
15. Testing in the All IP World - Example Scenario
DEMO
QualiSystems Proprietary & Confidential
17. DEMO 1: Router Testing via Traffic Generator
- Configure DUT by Telnet
Traffic Data Data Traffic
Sender Receiver
Configure Check alive
Router/Switch Command line
Run Traffic Telnet Analyze Results
Driver Driver
18. DEMO 1: Router Testing via Traffic Generator (cont.)
Code Free Test Creation as easy as Visio
Visio Flow Studio Flow
QualiSystems Proprietary & Confidential
19. DEMO 2: AP Router Testing via Traffic Generator
- Configure DUT by Website(GUI)
Traffic Data Data Traffic
Sender Receiver
Configure Router Check alive
Website(GUI) Command line
Run Traffic Analyze Results
Driver Driver
20. DEMO 2: AP Router Testing via Traffic Generator (cont.)
- Configure DUT by Website(GUI)
• Special features for Wireless Device:
1. Use different configuration files to complete diverse
testing, ex. different mode(802.11n,b,g), different
channel(1~11) or different encryption type(WEP, TKIP).
2. Use excel file to maintain the path of configuration files
and import data of excel file by “File Engine” on
Testshell.
3. TestShell’s GUI Automator can provide “Inspect” and
“Valuate” feature to check states of web screen.
4. TestShell can get snapshoot of testing result and export
to report automatically.
QualiSystems Proprietary & Confidential
21. Tests Creation - Benefits
• Drag & Drop Test Creator – non programmers can join the automation
• Reusable Building Blocks
• Record and Replay Actions
• Integration with both Traffic Generators and T&M
• Supported interfaces:
• Network Protocols (Telnet, SSH, Serial, TCP, UDP, SNMP, etc)
• GUI – Web, Java, Win app
• Logic and Dialog
• Integration with external files (configuration and results)
• Resource Asset Management & Deployment
• Documentation
• Scripts
• Executables
• Dlls
22. End-to-End Test Automation Framework
Schedule Execute
Create Analyze
Central
Management
QualiSystems Proprietary & Confidential
24. Tests Scheduling –
Scheduler
• Automatic scheduling according to time, station and priority
• Test scheduling across single or multiple stations
• Advanced "Outlook-like" interface
• Virtual station locking
• Fully customizable execution rules
• Automatic real-time execution conflict handling
25. End-to-End Test Automation Framework
Schedule Execute
Create Analyze
Central
Management
QualiSystems Proprietary & Confidential
26. Tests Execution –
Runner
• Execution planning
• Select Test or Regression
• Select Unit Under Test and Topology
• Record Admin information
• Serial, Version, Batch etc… (Dynamic)
• Run
• Complete Run
• Step by Step
• Automatic troubleshooting and action execution
• Real-time data tracking
• Standardize data collection
• Data transfer guarantee
27. End-to-End Test Automation Framework
Schedule Execute
Create Analyze
Central
Management
QualiSystems Proprietary & Confidential
28. Test Analysis - InSight
Central data collection
Data is then ready
Test Lab 1 Test Lab 2 21,6 C Test Lab n for comprehensive
128 ms 19 W 101,3 3E-07
102,1
81,9
MHz Melbourne Canbara analysis, using
51 kb/s 3,2 V
MHz 11,9 mA 57 kb/s 69% 12 5,3 TestShell InSight
21 C Sydney 3,3 V mA
www
Data from all
test labs and
stations is
Multi-Site
collected and Repository
stored in a
central server
29. Test Analysis - InSight
Online customizable display
www
The dashboards are
completely customizable
according to user
The analayzed data is
preferences, supporting
presented via online & display
multiple analysis
Dasboards: including OLAP
options,
• Providing real time analysis
• Reviewed any time anywhere
30. End-to-End Test Automation Framework
Schedule Execute
Create Analyze
Central
Management
QualiSystems Proprietary & Confidential
31. Test Management modules - Snapshot
Tests and Regression
management
DUT
management
Central repository across the organization
Tests version
control Fully scalable and openand user control
Permission framework
QualiSystems Proprietary & Confidential