Successfully reported this slideshow.

The art of system and solution testing


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The art of system and solution testing

  1. 1. The Art Of System and Solution Testing
  2. 2. Example • A software developer/tester convention was being held. On the train to the convention, there were a bunch of developer majors and a bunch of tester majors. Each of the developer majors had his/her train ticket. The group of testers had only ONE ticket for all of them. The developer majors started laughing and snickering. Then, one of the software testers said, "here comes the conductor" and then all of the testers went into the bathroom. The developer majors were puzzled. The conductor came aboard and said "tickets please" and got tickets from all the developer majors. He then went to the bathroom and knocked on the door and said "ticket please" and the testers stuck the ticket under the door. The conductor took it and then the testers came out of the bathroom a few minutes later. The developer majors felt really stupid. So, on the way back from the convention, the group of developer majors had one ticket for the group. They started snickering at the testers, for the whole group had no tickets amongst them. Then, the tester lookout said "Conductor coming!" All the testers went to one bathroom. All the developer majors went to another bathroom. Then, before the conductor came on board, one of the testers left the bathroom, knocked on the other bathroom, and said "ticket please." Lesson learned: Any test that passed in unit testing can fail in system testing.
  3. 3. Goal • Find bugs that can not be fund by feature level testing but will most likely be found on customer’s Usage. • If no time, can even only do system level and solution testing. • Development is a science, testing is an art.
  4. 4. What Is a System Level Testing • Company N has a testbed called “Mother of All Testbed (MOAT). It is the superset of almost all Company N’s product features. • To fully execute a cycle, it takes 2 weeks. • It has just one test case. Compare to feature level testing where you have thousands of test cases
  5. 5. What Is a Solution Level Testing • Company C has a dedicated solution testing team • Each solution is an end-to-end application and has a code name – VoIP end-to-end solution – Security end-to-end solution – MPLS solution – Cable/DSL solution.
  6. 6. Difference? • Solution is customer oriented. Not just one company’s product. • System testing is product oriented. • Feature testing is code oriented. (need a functionality Specification as input) • System/Solution: 30% strategy, 70% experience • Feature:60% strategy, 40% experience
  7. 7. What makes a System/Solution tester • System/Solution testing is to test the “Customer”.Input Tester Output User Manual Customer Secenrios. Experience Test Plan/Test Cases System Level Defects
  8. 8. System Tester’s 360 Degree Advantage • They interface with Customer (understand customer) • They interface with SE, TME, Product manager (understand marketing) • They interface with developer (understand development) • They interface with feature tester. (understand testing)
  9. 9. Trademark of System and Solution Testing • Race condition • Memory corruption • Algorithm performance (routing protocol convergence time for both uni-cast and multicast) • How system perform in low memory and high CPU (timers will be stretched out)
  10. 10. Typical Bugs Found on System Testing • Crash: Infrastructure design issue • Crash: Race condition: Malloc memory twice, crash only happens when 1st one success and 2nd one fail. • Solution : Interoperability with 3rd party products
  11. 11. System Testing is a Time Capsule • 99.9999% up time = 10 seconds down time in 3 years • What your customers will experience in 3 years? • Emulate 3 years in 2 weeks • That is why lots of stress testing
  12. 12. System Testing is a Time Capsule 正常 异常 1 异常 2 异常 3 异常 4 异常 5 异常 6 1-2 年 2 周
  13. 13. Ideas on Building Your System Cases • Defined the baseline – Background traffic. – # of clients – Complexity of the configuration • Identify scalability (performance) index. • Get the typical customer scenarios and topologies. • Design test cases.
  14. 14. Build Your Background Traffic • Background traffic need to drive CPU and memory to 30%-40%. • Need to be diverse among typical applications • Need to be measurable on – Latency – Transit loss. • IMIX traffic may be considered
  15. 15. No Performance Index, No Testing • Performance index means scalability • Size of the routing tables • Number of VPN tunnels • Voice call per second • Routing protocol convergence time • Traffic throughput. • Peer count (P2P) • Connection Rate (P2P) • Concurrent connections
  16. 16. QC Meets Customer • The product is a solution. If the problem isn't solved, the productdoesn't work. • The best test case can ever be designed on your product is in the customer’s real network. • The complexity of your test cases should be equal to your most valuable customers. • Customers like QC visit from vendor company very much. You will be surprised
  17. 17. Define Your Scenarios • This is the way customer use it. • Different location, different area (signal coverage) • Barrtery • Different combination way of using our product – GPRS browsing with MP3 in the background while have an unidentified call • Different speed for mobile devices
  18. 18. 20/80 Rule in the consumer market • Satisfied customers seldom speak up • Unhappy customers will complain • 1% chance of reproducible rate is ok in the network equipment market, but not ok in the consumer market • Higher testing standard in the consumer market
  19. 19. Simulation Is Your Friends • Customer level’s scalability can be achieved in our lab by simulator • Traffic simulator (commercial testing equipment are expensive). • Client simulator • Protocol Simulator. • Device Simulator.
  20. 20. Simulation Is Your Friends • A 32 timer reverse Bug, happen once in every 2 years, how you test that?
  21. 21. Steps to Design a System Level Testing • One liner for your testing goals. • List the features you want to cover. • Define your features performance index (metrics) • Design your topology – Customer feedback – Performance index requirement • Pick your test simulator (budget conscious)
  22. 22. Give a Detail Example on DHCP server testing • Definition: Carrier grade high availability DHCP server System level testing. • Goals: testing system instability especially during the failover. • Performance index: – IP address release per second. – Size of the database. – Size of the log. – Command and Control traffic (admin traffic) – DNS traffic. • Customer’s topology (simple) • Simulator: DHCP client simulator
  23. 23. Give a Detail Example on DHCP server testing • Scenarios: 1 Primary down, then come back after secondary switch over 2 Primary down, then come back right away before the switch over. 3 Primary down, then come back after switch over, the down again. 4 Primary keep flapping – up, down, up, down for a while (both before and after the switch over). 5 Secondary down, then come back. 6 Secondary keep flapping – up, down, up, down for a while (both before and after the switch over). 7 Both primary and secondary keep flapping for a while. 8 Both shut down then come back the same time. 9 Both shut down, the secondary come back first. 10 Both shut down, the primary come back first.
  24. 24. Get the Pass/Fail Criteria. • DHCP servers can response DHCP request. • No server performance down grade, the DHCP can still response to a high rate of DHCP request during the failover and recovery • The failover relationship between primary and secondary, and the failover state is working as expected. • Database sync correctly. And the data integrity is fine.
  25. 25. Overlap your System Test Cases • Unlike feature testing, the impact among features are welcomed • Overlap your test cases one on top of another increase the feature interaction. • Carefully record your execution steps. (reduce non-reproducible bugs)
  26. 26. Example: Firewall Instability Testing• Definition: Testing a firewall system’s instability • Goals: Discover critical issues when firewall is under severe network conditions • Performance index: – # of current sessions – # of VPN tunnels – # of content filtering rules. – # of policies – # of the NAT • Customer’s topology (simple) • Simulator: DHCP client simulator
  27. 27. Example: Firewall Instability Testing • Customer’s topology
  28. 28. Example: Firewall Instability Testing • Baseline traffic (pre-conditions) – Valid traffic • L3 traffic • L7 traffic (HTTP/FTP, voice, rich application) – Invalid traffic (noise) • To the box • Through the box • Trust and un-trust zone • Malicious (hacker traffic) and just invalid
  29. 29. Example: Firewall Instability Testing • Pass and fail criteria – No crash – No severe performance downgrade on • Packet loss • Long latency – Major functionality working on each of the performance index.
  30. 30. Example: Firewall Instability Testing • Customer Secenrios – HA – VPN – Routing – QoS – DPI
  31. 31. 14 Steps Firewall TestingPhase1: start valid traffics Phase2: layer3 and layer4 attacks Phase3: Inject firewall attack traffic Phase4: Inject invalid frame Phase5: Drop In Phase6: management attacks Phase7: standard security testing Phase8: Internet applications Phase9: IP fragmentation, ICMP Phase10: HA function Phase11: VPN function Phase12: dynamic routing function Phase13: QoS Phase14: contents filter
  32. 32. System and Solution Automation • System/Solution automation is to make your life easy • End-to-end full automation is hard, try to do semi-automation first.