Software Testing:  Gets the Bugs out, and More Dean Cai Motorola Computer Group
Software Testing in the Life Cycle of a Product   <ul><li>Life cycle of a telecommunication platform product </li></ul><ul...
Challenges of Developing a Telco-grade, High Availability Platform   <ul><li>Five-9’s and six-9’s </li></ul><ul><li>High A...
The Software Design Goals Unified implementation running on all platforms and  multiple operating systems. Scaleable softw...
The Software System   <ul><li>An Architectural view of the software system in a complex telecommunication platform </li></...
A Sample System Application Network Manager Packet   Network ATM IP CPX Network Elements
CPX System HA Features <ul><li>Hot Swappable CPU and I/O boards  </li></ul><ul><li>Dual CompactPCI   backplanes </li></ul...
Advanced High Availability SW Customer Application Programs Customer Developed Scripts & Utilities Event Management APIs M...
Application Layer <ul><li>SNMP v3 Agent </li></ul><ul><li>Monolithic or Master-Subagent models  </li></ul><ul><li>Standard...
Application Interface Layer <ul><li>APIs (libraries) for event and Alarm Management </li></ul><ul><li>API to general messa...
Event Manager Layer <ul><li>Clustering agent synchronizes Databases, Rules and actions to  multiple systems. </li></ul><ul...
Operating System Layer <ul><li>PCI services for dynamic insertion/ deletion of devices. </li></ul><ul><li>HSC driver for c...
Strategies of Testing a Complex Software System (1) <ul><li>Unit test </li></ul><ul><li>Integration test </li></ul><ul><li...
Strategies of Testing a Complex Software System (2) <ul><li>static int </li></ul><ul><li>pxe_open(struct device *dev) </li...
Strategies of Testing a Complex Software System (3) <ul><li>System test </li></ul><ul><li>System level functionalities </l...
Test Planning (1)   <ul><li>The process of generating a test plan </li></ul><ul><li>MRD </li></ul><ul><li>SRD </li></ul><u...
Test Planning (2) <ul><li>An example of a test plan </li></ul>
Test Planning (3) <ul><li>Test scenario design </li></ul><ul><li>Scenario for function coverage </li></ul><ul><li>Scenario...
Test Planning (4) <ul><li>Test tool developing </li></ul><ul><li>API test </li></ul><ul><li>Utilities and demo application...
Test Execution   <ul><li>DVT (hardware, firmware) </li></ul><ul><li>Software installation and configuration </li></ul><ul>...
Problem Reporting, Resolving, and Verifying <ul><li>Quality control from software testing perspective </li></ul><ul><li>Th...
Test Management and Automation   <ul><li>Ways to improve the quality of testing </li></ul><ul><ul><li>Process control </li...
Project Goals <ul><li>Increase test coverage </li></ul><ul><li>Increase test efficiency </li></ul><ul><li>Reduce System Te...
Description of the Project (1) <ul><li>Identify requirements for test coverage </li></ul><ul><li>Analyze current test cove...
Description of the Project (2) <ul><li>Improve the existing tools </li></ul><ul><li>Develop new tools </li></ul><ul><li>Se...
Description of the Project (3) <ul><li>Build a test database for collecting and archiving test data </li></ul><ul><li>deve...
Analyze Current Test Coverage <ul><li>Overall system coverage </li></ul><ul><li>DVT coverage (CPU, backplane, IO slots, al...
Overall System Coverage HCT AIX Suite ports GoAhead Test Suite Florian 100% coverage HA tests Holes Overlaps No-needs Manu...
Software Functionality Coverage OS Appl Driver API Unit Test Bd Level DVT OS Integ Test System Test Config. 1 Config. 2 Co...
Test Framework Architecture User Login HW Configurator SW Configurator Test Monitor Tool Download/ Rm mount Data Collector...
Sample Test Flow (1) Platform CPX8221 CPX1205 Gemini 2.0 Platform Configurator Suite #1 Suite #2 Suite #X Test Suite Confi...
Sample Test Flow (2) Target hostname TH-1 TH-2 TH-X 6.  Enter target host name 7.  Test suite download from Server to Targ...
Test Framework Selection <ul><li>TETware - Unix Script type, no GUI </li></ul><ul><li>LabView  </li></ul><ul><li>HP-VUE </...
Upcoming SlideShare
Loading in …5
×

Guest lecture from Motorola

2,366 views

Published on

  • Be the first to comment

  • Be the first to like this

Guest lecture from Motorola

  1. 1. Software Testing: Gets the Bugs out, and More Dean Cai Motorola Computer Group
  2. 2. Software Testing in the Life Cycle of a Product <ul><li>Life cycle of a telecommunication platform product </li></ul><ul><li>Software testing in different stages of a product life cycle </li></ul>
  3. 3. Challenges of Developing a Telco-grade, High Availability Platform <ul><li>Five-9’s and six-9’s </li></ul><ul><li>High Availability </li></ul><ul><li>Fault Tolerant </li></ul><ul><li>Open architecture issues </li></ul><ul><li>                        Linux source level compatibility </li></ul><ul><li>                       User support in an open source environment </li></ul><ul><li>Out-of-box experience </li></ul><ul><li>auto detect and auto configure in an user defined environment </li></ul>
  4. 4. The Software Design Goals Unified implementation running on all platforms and multiple operating systems. Scaleable software architecture for the continuous advancement of system availability. Ease of integration of telecom applications. Ease of integration of telecom specific subsystems to expand the platform offering.
  5. 5. The Software System <ul><li>An Architectural view of the software system in a complex telecommunication platform </li></ul><ul><li>Example: Motorola Computer Group’s CPX system </li></ul>
  6. 6. A Sample System Application Network Manager Packet Network ATM IP CPX Network Elements
  7. 7. CPX System HA Features <ul><li>Hot Swappable CPU and I/O boards </li></ul><ul><li>Dual CompactPCI  backplanes </li></ul><ul><li>Hot Swappable drives, fans, power </li></ul><ul><li>supplies, bridges and alarm panel </li></ul><ul><li>Chassis status & control via HSC </li></ul><ul><li>Domain control via HSC </li></ul><ul><li>Telco alarms </li></ul><ul><li>2N, N+1 and N+m configurations </li></ul>Compact PCI Bus Segment A Compact PCI Bus Segment B I/O Slots CPU-A CPU-B Bridge-B Bridge-A I/O Slots
  8. 8. Advanced High Availability SW Customer Application Programs Customer Developed Scripts & Utilities Event Management APIs Messaging & Checkpointing Alarm Management Distributed Messaging Inter-Host Comm. & Checkpointing Heartbeat & Switchover Protocols System Configuration & Event Manager System Alarm Manager Event Management Checkpointing Agent Policies, Rules, Actions, Methods HA Utility Programs Web Server/ Graphical User & Interface PCI Services HSC Driver ENUM Handler Event Mgr. Driver HA Aware Drivers Backplane Messaging Driver MIB SNMP MIB MIB I/O Bus Driver Cluster Agent
  9. 9. Application Layer <ul><li>SNMP v3 Agent </li></ul><ul><li>Monolithic or Master-Subagent models </li></ul><ul><li>Standard MIB2 Support </li></ul><ul><li>UCD extensions for some sysadmin functions </li></ul><ul><li>Motorola MIB for Event, Domain, Alarm and Chassis Management </li></ul><ul><li>Graphical User Interface to Event, Domain, Alarm and Chassis Management </li></ul><ul><li>Customer provided applications, utilities and scripts </li></ul>Customer Application Programs Customer Developed Scripts & Utilities Web Server/ Graphical User & Interface MIB SNMP MIB MIB
  10. 10. Application Interface Layer <ul><li>APIs (libraries) for event and Alarm Management </li></ul><ul><li>API to general messaging Application interface </li></ul><ul><li>Simplified checkpointing interface </li></ul><ul><li>Database of system component/object definitions </li></ul><ul><li>Policies and Rules for managing system state transitions </li></ul><ul><li>Actions and methods/programs for handling state transitions </li></ul><ul><li>Utility programs for Event, Domain, Alarm and Chassis Management </li></ul>Event Management APIs Messaging & Checkpointing Alarm Management Policies, Rules, Actions, Methods HA Utility Programs
  11. 11. Event Manager Layer <ul><li>Clustering agent synchronizes Databases, Rules and actions to multiple systems. </li></ul><ul><li>Checkpointing agent provides synchronization services for applications and Event Manager. </li></ul><ul><li>System Configuration & Event Manager provides configurable system </li></ul><ul><li>event and availability management. </li></ul><ul><li>System Alarm manager controls and maintains chassis alarms, LEDs, </li></ul><ul><li>Fans and power supplies. </li></ul>Distributed Messaging System Configuration & Event Manager System Alarm Manager Event Management Checkpointing Agent Cluster Agent
  12. 12. Operating System Layer <ul><li>PCI services for dynamic insertion/ deletion of devices. </li></ul><ul><li>HSC driver for chassis status and control. </li></ul><ul><li>ENUM handler for fielding slot events. </li></ul><ul><li>HA Aware drivers specification and examples to include fault detection, </li></ul><ul><li>fault diagnosis, fault recovery and active - standby configuration. </li></ul><ul><li>Backplane messaging driver and utilities to provide low bandwidth </li></ul><ul><li>board to board communication utilizing the PCI bus. </li></ul><ul><li>Event Manager driver funnels event reporting to management layer. </li></ul><ul><li>Inter-Host communication for synchronization of state and checkpoint data between redundant drivers and stacks. </li></ul>Inter-Host Comm. & Checkpointing Heartbeat & Switchover Protocols PCI Services HSC Driver ENUM Handler Event Mgr. Driver HA Aware Drivers Backplane Messaging Driver I/O Bus Driver
  13. 13. Strategies of Testing a Complex Software System (1) <ul><li>Unit test </li></ul><ul><li>Integration test </li></ul><ul><li>Mix of vendor FW, in-house FW, vendor middleware, in-house middleware, utility commands, and open source OS </li></ul><ul><li>Example: pxe driver and hsc driver confliction </li></ul>
  14. 14. Strategies of Testing a Complex Software System (2) <ul><li>static int </li></ul><ul><li>pxe_open(struct device *dev) </li></ul><ul><li>{ </li></ul><ul><li>// __asm__ __volatile__ (&quot;.align 2 &quot; &quot;.byte 241&quot;); </li></ul><ul><li>if (net_debug) </li></ul><ul><li>printk(&quot;Entered pxe_open() &quot;); </li></ul><ul><li>  </li></ul><ul><li>if (request_irq(dev->irq, &pxe_interrupt, SA_SHIRQ, &quot;Universal Network Driver&quot;, dev)) { </li></ul><ul><li>  </li></ul><ul><li>/* </li></ul><ul><li>if (request_irq(dev->irq, &pxe_interrupt, 0, &quot;Universal Network Driver&quot;, dev)) { </li></ul><ul><li>*/ </li></ul><ul><li>return -EAGAIN; </li></ul><ul><li>} </li></ul><ul><li>  </li></ul>
  15. 15. Strategies of Testing a Complex Software System (3) <ul><li>System test </li></ul><ul><li>System level functionalities </li></ul><ul><li>System stress environment </li></ul><ul><li>User friendly evaluation </li></ul>
  16. 16. Test Planning (1) <ul><li>The process of generating a test plan </li></ul><ul><li>MRD </li></ul><ul><li>SRD </li></ul><ul><li>Design specification </li></ul><ul><li>Test requirement </li></ul><ul><li>Test plan generation </li></ul><ul><li>Review and approval </li></ul>
  17. 17. Test Planning (2) <ul><li>An example of a test plan </li></ul>
  18. 18. Test Planning (3) <ul><li>Test scenario design </li></ul><ul><li>Scenario for function coverage </li></ul><ul><li>Scenario for system stress </li></ul><ul><li>Scenario for fault injection, fail-over, and fail recovery </li></ul>
  19. 19. Test Planning (4) <ul><li>Test tool developing </li></ul><ul><li>API test </li></ul><ul><li>Utilities and demo applications using user accessible APIs </li></ul><ul><li>Hooks in application and device driver for fault injection </li></ul><ul><li>Remote test tools, such as SNMP </li></ul>
  20. 20. Test Execution <ul><li>DVT (hardware, firmware) </li></ul><ul><li>Software installation and configuration </li></ul><ul><li>Utility command based testing (feature verification, code validation) </li></ul><ul><li>Stress test </li></ul><ul><li>Expose potential timing issues (HW and SW) </li></ul><ul><li>Expose potential system resource allocation problems </li></ul><ul><li>Memory leak </li></ul><ul><li>User application simulation (user scenario test, API validation) </li></ul>
  21. 21. Problem Reporting, Resolving, and Verifying <ul><li>Quality control from software testing perspective </li></ul><ul><li>The process flow of our in-house problem reporting system (CPR) </li></ul>
  22. 22. Test Management and Automation <ul><li>Ways to improve the quality of testing </li></ul><ul><ul><li>Process control </li></ul></ul><ul><ul><li>Automation </li></ul></ul><ul><li>An sample approach: </li></ul><ul><li>System Test Management Framework </li></ul>
  23. 23. Project Goals <ul><li>Increase test coverage </li></ul><ul><li>Increase test efficiency </li></ul><ul><li>Reduce System Test Cycle Time </li></ul><ul><li>Reduce preparation time and learning curve </li></ul><ul><li>Increase test traceability </li></ul><ul><li>Built-in process control </li></ul>
  24. 24. Description of the Project (1) <ul><li>Identify requirements for test coverage </li></ul><ul><li>Analyze current test coverage </li></ul><ul><li>Examine current test tools </li></ul><ul><li>Identify the needs for improving existing tools </li></ul><ul><li>Identify the needs for new tools </li></ul>
  25. 25. Description of the Project (2) <ul><li>Improve the existing tools </li></ul><ul><li>Develop new tools </li></ul><ul><li>Select a test framework package (TETware, HP-VUE, LabView, etc.) </li></ul><ul><li>Consolidate tools into an user-friendly framework </li></ul><ul><li>Develop an user front-end of the framework for test configuration and deployment </li></ul>
  26. 26. Description of the Project (3) <ul><li>Build a test database for collecting and archiving test data </li></ul><ul><li>develop a test plan generator </li></ul><ul><li>develop a test report generator </li></ul><ul><li>Develop a CM package for tool development and maintenance </li></ul><ul><li>Staff training </li></ul>
  27. 27. Analyze Current Test Coverage <ul><li>Overall system coverage </li></ul><ul><li>DVT coverage (CPU, backplane, IO slots, alarm panels, power supply, etc.) </li></ul><ul><li>Software functionality coverage </li></ul><ul><li>Software integration coverage </li></ul><ul><li>System functionality coverage </li></ul><ul><li>System performance coverage </li></ul>
  28. 28. Overall System Coverage HCT AIX Suite ports GoAhead Test Suite Florian 100% coverage HA tests Holes Overlaps No-needs Manual test
  29. 29. Software Functionality Coverage OS Appl Driver API Unit Test Bd Level DVT OS Integ Test System Test Config. 1 Config. 2 Config. X
  30. 30. Test Framework Architecture User Login HW Configurator SW Configurator Test Monitor Tool Download/ Rm mount Data Collector Report Generator Test Event Log HW Config Database SW Config Database Test Doc Database Test Data Database Go Test Tool Lib Test Tool Studio Test Plan Generator User Interface
  31. 31. Sample Test Flow (1) Platform CPX8221 CPX1205 Gemini 2.0 Platform Configurator Suite #1 Suite #2 Suite #X Test Suite Configurator 1. Login on Test Server 2. Select Platform 3. Enter platform config data 4. Select test suite(s) to run 5. Enter test suite config data
  32. 32. Sample Test Flow (2) Target hostname TH-1 TH-2 TH-X 6. Enter target host name 7. Test suite download from Server to Target 8. Target console opens on the server login 9. Test suite starts on Target 10. Test data transfer from Target to Server Server Target Test suite Go Server Target Test data
  33. 33. Test Framework Selection <ul><li>TETware - Unix Script type, no GUI </li></ul><ul><li>LabView </li></ul><ul><li>HP-VUE </li></ul><ul><li>VisualBASIC </li></ul><ul><li>SNMP Manager </li></ul>

×