Your SlideShare is downloading. ×
0
Focused Iterative Testing: A Test Automation Case Study Mechelle Gittens, Pramod Gupta, David Godwin,  Hebert Pereyra, Jef...
Problem Definition: Find Timing Related Defects <ul><li>Server software (e.g. DB2) is a multithreaded, multiprocess system...
Different Approaches to Address trigger TRDs <ul><li>Vary CPU load by external programs </li></ul><ul><li>Instrument code ...
What is Focused Iterative Testing? <ul><li>Find a focus area which has timing related defects e.g. crash recovery </li></u...
Why do we use FIT ? <ul><li>Improve reliability of mission critical components in DB2 e.g. crash recovery, monitor, FCM et...
How do we write a FIT tool ? <ul><li>Input from APAR (Customer Defects) Causal Analysis  </li></ul><ul><li>Input from Cust...
Positive Feedback Loop <ul><li>Early implementation of Agile methodologies: </li></ul><ul><li>Weekly meeting with develope...
Case Study: Crash recovery for a database server <ul><li>Stringent Requirement: A database server must keep the database i...
FIT tool for crash recovery <ul><li>Timing Defects due to logging and replaying of logs </li></ul><ul><li>FIT crash recove...
FIT tool for Fast Communication Manager Testing and for Monitor Testing <ul><li>Timing defects due to buffers and channels...
FIT tool for Monitor Testing <ul><li>Timing defects since  code has lot of locking of resources </li></ul><ul><li>Monitor ...
To FIT or not to FIT <ul><li>FIT is approach for System Testing  </li></ul><ul><li>It can be used to detect major defects ...
Investment in FIT <ul><li>At beginning of test cycle: </li></ul><ul><li>Extra tooling effort </li></ul><ul><li>During test...
Return on Investment in FIT <ul><li>Larger number of timing related defects found during test cycle. So customer gets more...
Conclusion <ul><li>Focused Iterative Testing is a general technique  </li></ul><ul><li>Very successful in testing multithr...
Questions?
Upcoming SlideShare
Loading in...5
×

IBM Software Group IBM Toronto Lab | ACM SIGMOD DBTEST2008 ...

261

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
261
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "IBM Software Group IBM Toronto Lab | ACM SIGMOD DBTEST2008 ..."

  1. 1. Focused Iterative Testing: A Test Automation Case Study Mechelle Gittens, Pramod Gupta, David Godwin, Hebert Pereyra, Jeff Riihimaki
  2. 2. Problem Definition: Find Timing Related Defects <ul><li>Server software (e.g. DB2) is a multithreaded, multiprocess system with a large number of concurrent users </li></ul><ul><li>Server software uses logging, locking/latching </li></ul><ul><li>This causes Timing Related Defects (TRD) </li></ul><ul><li>Difficult to reproduce customer problems </li></ul>
  3. 3. Different Approaches to Address trigger TRDs <ul><li>Vary CPU load by external programs </li></ul><ul><li>Instrument code to put in random sleeps </li></ul><ul><li>Change priorities of processes </li></ul><ul><li>Iteratively execute commands with background database workload (Most Successful approach) </li></ul>
  4. 4. What is Focused Iterative Testing? <ul><li>Find a focus area which has timing related defects e.g. crash recovery </li></ul><ul><li>Identify what finite number of commands are to be tested e.g. restart database command after a crash </li></ul><ul><li>Write a tool to automate the whole test case and control the background workload </li></ul><ul><li>The tool is run repeatedly till the target number of successful iterations is reached frequently. Track the number of successful iterations as a function of time. </li></ul>
  5. 5. Why do we use FIT ? <ul><li>Improve reliability of mission critical components in DB2 e.g. crash recovery, monitor, FCM etc. </li></ul><ul><li>FIT gives us a objective metric of quality which is very useful to management </li></ul><ul><li>Easier to find and easier to debug Timing Related Defects </li></ul><ul><li>Each FIT setup has different environment e.g. database configuration parameters </li></ul>
  6. 6. How do we write a FIT tool ? <ul><li>Input from APAR (Customer Defects) Causal Analysis </li></ul><ul><li>Input from Customer Critical Situations </li></ul><ul><li>Input from developers (weekly meeting with developers) </li></ul><ul><li>Implement FIT Tools </li></ul><ul><li>Ensure tool is easy to use so more testers use it </li></ul>
  7. 7. Positive Feedback Loop <ul><li>Early implementation of Agile methodologies: </li></ul><ul><li>Weekly meeting with developers </li></ul><ul><li>1. Meet developers to discuss previous week's defects and get suggestions for enhancing FIT tool </li></ul><ul><li>2. Enhance FIT tool </li></ul><ul><li>3. Run tool and find more defects </li></ul><ul><li>4. Goto step1 </li></ul>
  8. 8. Case Study: Crash recovery for a database server <ul><li>Stringent Requirement: A database server must keep the database in consistent state regardless of external events like power failures etc. </li></ul><ul><li>Crash Recovery: When database is restarted after a outage then it must correctly replay the transaction logs and ensure that the database is in a consistent state. </li></ul><ul><li>Crash recovery failure seriously affects customer's business and is a customer critical situation. </li></ul><ul><li>Hence, it is very important to test crash recovery. </li></ul>
  9. 9. FIT tool for crash recovery <ul><li>Timing Defects due to logging and replaying of logs </li></ul><ul><li>FIT crash recovery tool: </li></ul><ul><li>Run clients issuing SQL and DB2 commands for a database </li></ul><ul><li>Deliberately crash the system by killing DB2 processes </li></ul><ul><li>Restart the database so server plays transaction logs. </li></ul><ul><li>Repeat </li></ul><ul><li>If the FIT tool finds a problem, it sends e-mail to tester. </li></ul><ul><li>FIT tools for crash recovery are run on AIX and Windows </li></ul>
  10. 10. FIT tool for Fast Communication Manager Testing and for Monitor Testing <ul><li>Timing defects due to buffers and channels between nodes. </li></ul><ul><li>Fast Communication Manager FIT tool: </li></ul><ul><li>Similar to tool for crash recovery testing but we only kill selected nodes in a multi-node database system. </li></ul><ul><li>Ensure communication between still running nodes is OK. </li></ul><ul><li>Failure of subset of nodes should not bring other nodes down. </li></ul>
  11. 11. FIT tool for Monitor Testing <ul><li>Timing defects since code has lot of locking of resources </li></ul><ul><li>Monitor FIT tool: </li></ul><ul><li>run finite set of monitor commands with background workload </li></ul><ul><li>Snapshot family of commands give current picture </li></ul><ul><li>Event monitor commands give historical picture </li></ul><ul><li>We do not kill any nodes </li></ul>
  12. 12. To FIT or not to FIT <ul><li>FIT is approach for System Testing </li></ul><ul><li>It can be used to detect major defects like traps or hangs in code with timing related defects </li></ul><ul><li>It is not applicable to: </li></ul><ul><li>Code which does not have timing related defects </li></ul><ul><li>Function testing </li></ul><ul><li>Unit testing </li></ul>
  13. 13. Investment in FIT <ul><li>At beginning of test cycle: </li></ul><ul><li>Extra tooling effort </li></ul><ul><li>During test cycle: </li></ul><ul><li>Weekly meetings with development </li></ul><ul><li>Enhancing FIT tool </li></ul>
  14. 14. Return on Investment in FIT <ul><li>Larger number of timing related defects found during test cycle. So customer gets more reliable product. </li></ul><ul><li>Problems found earlier in cycle </li></ul><ul><li>Mean number of successful iterations gives objective measure of component quality </li></ul><ul><li>To ship or not to ship decisions can be taken more objectively </li></ul>
  15. 15. Conclusion <ul><li>Focused Iterative Testing is a general technique </li></ul><ul><li>Very successful in testing multithreaded, multiprocess, multiuser enterprise server software. </li></ul><ul><li>Beyond databases it could be applied to Messaging Systems like WebSphere MQ, journaled file systems like JFS in AIX, Enterprise Service Bus like WebSphere Message Broker etc. </li></ul>
  16. 16. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×