Designing Software For Testability: A forgotten design pattern


Published on

In the semiconductor industry, Design For Testability (DFT) is an essential part of the architecture and design of components. Software designers on the other hand do not pay much (if any) attention to the testing needs of their code.

In this talk we review some core DFT principles like Built-In Self Test, Test Point Insertion, Fault Modeling and Fault Simulation and map them to software testing. Examples of using DFT to create testable software will be given. DFT fits in especially well with the increasing use of Test Automation and Agile Methodologies.

We hope this talk will empower test leads and engineers with knowledge they can use to get their developer counterparts to modify the application-under-test to significantly increase automation, enhance test coverage, run tests faster and reduce the costs of testing.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Designing Software For Testability: A forgotten design pattern

  1. 1. Designing Software For Testability A forgotten design pattern Rohit Nayak Talentica
  2. 2. Agenda <ul><li>Background and Motivation </li></ul><ul><li>DFT in VLSI and hardware design </li></ul><ul><li>Issues in Software Testing </li></ul><ul><li>DFT for Software Development </li></ul><ul><li>Automation and Testability </li></ul><ul><li>Some Examples </li></ul><ul><li>Cons, Review </li></ul>
  3. 3. <ul><li>The Anti-Pattern </li></ul><ul><li>If it works, </li></ul><ul><li>its the developer, </li></ul><ul><li>if not blame QA! </li></ul>
  4. 4. VLSI/PCB Testing Issues <ul><li>Mass produced </li></ul><ul><li>Each piece needs to be validated </li></ul><ul><li>Complexity </li></ul><ul><ul><li>Number of subsystems </li></ul></ul><ul><ul><li>Amount of logic </li></ul></ul><ul><li>Access to internal logic </li></ul><ul><li>Testing costs </li></ul><ul><li>Cost of recall </li></ul>
  5. 6. <ul><li>Testability: </li></ul><ul><li>“ Effort required to test a product to ensure that it performs its intended function” </li></ul>
  6. 7. DFT Principles <ul><li>Controllability </li></ul><ul><li>Observability </li></ul><ul><ul><li>Test Point Insertion </li></ul></ul><ul><li>Built-In Self Test (BIST) </li></ul><ul><li>Fault Modelling </li></ul><ul><li>Fault Simulation </li></ul><ul><li>Test Pattern Generation </li></ul>
  7. 10. Software Testing <ul><li>“ Testability is a design issue and needs to be addressed with the design of the rest of the system. Projects that defer testing to the later stages of a project will find their programmers unwilling to consider testability changes by the time testers actually roll onto the project and start making suggestions. Testability takes cooperation, appreciation and a team commitment to reliability.” </li></ul><ul><li>- Bret Pettichord </li></ul>
  8. 11. Cost Of Bugs 10-25x 10x 1x Coding 25-100x 15x 10x 1x Architecture 10x-100x 10x 5-10x 3x 1x Requirements Production System Test Coding Architecture Requirements Where Found Where Introduced
  9. 13. Software Testability <ul><li>Suitability: clarity of specifications QA has </li></ul><ul><li>Observability: we can only test what is visible </li></ul><ul><li>Simplicity: easier design/UI makes testing easier </li></ul><ul><li>Controllability: better the control, better the coverage and automation </li></ul><ul><li>Stability: how often it changes </li></ul><ul><li>Performance: how fast it works </li></ul><ul><li>Diagnosability: writing effective bug reports! </li></ul>
  10. 14. Software DFT Patterns <ul><li>BIST </li></ul><ul><ul><li>Automation suite w/ oracle </li></ul></ul><ul><ul><li>Unit Tests </li></ul></ul><ul><ul><li>Assertions </li></ul></ul><ul><li>Fault Simulation </li></ul><ul><ul><li>Mock/Stub </li></ul></ul><ul><ul><li>Invalid params </li></ul></ul><ul><li>Controllability </li></ul><ul><ul><li>Decoupling </li></ul></ul><ul><ul><li>Bypassing </li></ul></ul><ul><ul><li>Mock/Stub </li></ul></ul><ul><ul><li>Set/Reset </li></ul></ul><ul><li>Observability </li></ul><ul><ul><li>Logging </li></ul></ul><ul><ul><li>Reporting </li></ul></ul><ul><ul><li>Test Interfaces </li></ul></ul>
  11. 15. Good Automated Tests are <ul><li>Repeatable </li></ul><ul><li>Easy to Write </li></ul><ul><li>Easy to Understand </li></ul><ul><li>Fast </li></ul><ul><li>Way Easier with a Testable Software </li></ul>
  12. 16. When/How to Automate? <ul><li>Manager & Team are commited </li></ul><ul><li>Testers/Dev either experienced or interested in learning to script </li></ul><ul><li>Product release cycles managed well </li></ul><ul><li>Functionality / UI changes under control </li></ul><ul><li>Early/Incremental approaches work best </li></ul><ul><li>Build integration, reporting </li></ul><ul><li>Each bug results in a automated test </li></ul>
  13. 17. Automation As Coding <ul><li>Automation scripts and frameworks are CODE! </li></ul><ul><li>Use tools and scripting languages </li></ul><ul><li>Evaluate tools on real problems </li></ul><ul><li>Follow development processes </li></ul><ul><ul><li>Spiral/Agile </li></ul></ul><ul><li>Use Source Control same as rest of code </li></ul><ul><li>These will have bugs as well! </li></ul>
  14. 18. Test “oracle” <ul><li>Expected Result </li></ul><ul><ul><li>Generated once from previous run or </li></ul></ul><ul><ul><li>Manually specified or </li></ul></ul><ul><ul><li>Legacy system </li></ul></ul><ul><li>Time adjustment </li></ul><ul><li>Results </li></ul><ul><ul><li>Database </li></ul></ul><ul><ul><li>File </li></ul></ul><ul><ul><li>Inline with code </li></ul></ul>
  15. 19. Test Environment <ul><li>Multiple VMs </li></ul><ul><li>Automated installs/images </li></ul><ul><li>Automation tools </li></ul><ul><li>Bug reporting tools </li></ul>
  16. 20. Unit Tests <ul><li>Each module has independent set </li></ul><ul><li>Developer written/maintained </li></ul><ul><li>setup, teardown </li></ul><ul><li>xUnit, TestNG </li></ul><ul><li>Runs on build </li></ul><ul><li>Can run sub-set on install </li></ul><ul><li>TDD </li></ul>
  17. 21. Diagnosability <ul><li>All environments: Test / Staging / Production </li></ul><ul><li>Errors result in visual messages </li></ul><ul><li>Errors are raised where they occur </li></ul><ul><li>Errors can be localized </li></ul><ul><li>Details are sufficient to fix issue </li></ul><ul><li>Ability to send diag data to dev </li></ul>
  18. 22. Logging Module <ul><li>Log levels ( VERBOSE, INFO, WARNING, ERROR, CRITICAL, TIME ) </li></ul><ul><li>Define CRITICAL/ERROR levels well and use them! </li></ul><ul><li>These should result in urgent notifications </li></ul><ul><li>Lower levels in Production </li></ul><ul><li>Time, App, Component, ThreadId, Message </li></ul><ul><li>8/2/2010 14:22pm: Notifier: Pop3: 8242: Sending welcome email to userid 334 </li></ul><ul><li>Delimited columns for import </li></ul><ul><li>Rotation based on Time, Size </li></ul>
  19. 23. Built-in Self Test <ul><li>Inserting test code/interfaces </li></ul><ul><li>Set/Reset to bring state to known value </li></ul><ul><li>Reporting to get current state </li></ul><ul><li>Assertions about values/state </li></ul><ul><ul><li>assert(order.billed==true) </li></ul></ul><ul><li>“ oracle” based regression </li></ul>
  20. 24. Some Examples <ul><li>Web Application </li></ul><ul><ul><li>Software As A Service </li></ul></ul><ul><ul><li>Browser Only UI </li></ul></ul><ul><ul><li>Ajax / Dynamic HTML </li></ul></ul><ul><li>Banking (Client-Server) </li></ul><ul><ul><li>Installed Application </li></ul></ul><ul><ul><li>Desktop Client </li></ul></ul><ul><ul><li>Server API </li></ul></ul>
  21. 25. Web Application <ul><li>HTML Page Title </li></ul><ul><li>Id values for important divs/controls </li></ul><ul><li>Hidden values (non-textual, graphs, tables) </li></ul><ul><li>Measurable Ajax responses </li></ul><ul><li>Tools: Selenium, Sahi, Watir, WebTest, curl </li></ul><ul><li>Logging incl. browser/ip/session cookie </li></ul><ul><li>Ability to simulate time zone, language </li></ul>
  22. 26. Web Application - 2 <ul><li>Bypassing Captcha </li></ul><ul><li>Mock TP APIs eg. Facebook, Google, OpenId </li></ul><ul><li>SSL bypassing </li></ul><ul><li>Transactions to be voided </li></ul><ul><li>Multiple runs </li></ul><ul><li>Always initialise controls </li></ul><ul><ul><li>Browser autofill </li></ul></ul>
  23. 27. Desktop <ul><li>Unique identifiers to GUI controls </li></ul><ul><li>Key Replay tools, SendKeys </li></ul><ul><li>API test suites bypassing UI </li></ul><ul><li>Client logs, ability to email </li></ul><ul><li>Ability to run db queries in scripts </li></ul>
  24. 28. Desktop - 2 <ul><li>Automation Friendly Third Party Controls </li></ul><ul><ul><li>Ability to select cell </li></ul></ul><ul><ul><li>Copy-enable text fields </li></ul></ul><ul><ul><li>Key shortcuts to Forms UI for control focus, clicking, navigation </li></ul></ul><ul><ul><li>Access by value (tree/list controls) </li></ul></ul><ul><li>OCR for images/text consoles </li></ul>
  25. 29. Bypassing <ul><li>Credit card payment </li></ul><ul><ul><li>Test cards </li></ul></ul><ul><ul><li>Mock object </li></ul></ul><ul><ul><li>Dev auto-approval bypass code </li></ul></ul><ul><li>Order placement </li></ul><ul><ul><li>Dummy users, auto-fulfill </li></ul></ul><ul><ul><li>Dummy vendors (email order) </li></ul></ul><ul><ul><li>Admin screens move </li></ul></ul>
  26. 30. Dates <ul><li>Never use system date directly </li></ul><ul><li>Config override </li></ul><ul><ul><li><?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; ?> </li></ul></ul><ul><ul><li><configuration> </li></ul></ul><ul><ul><li><appSettings> </li></ul></ul><ul><ul><li><add key=“SystemDate&quot; value=“$now&quot;> </li></ul></ul><ul><ul><li></appSettings> </li></ul></ul><ul><ul><li></configuration> </li></ul></ul><ul><li>Nant: XMLPoke </li></ul><ul><ul><li><xmlpoke file=&quot;App.config&quot; </li></ul></ul><ul><ul><li>xpath=&quot;/configuration/appSettings/add[@key = 'SystemDate']/@value&quot; </li></ul></ul><ul><ul><li>value=&quot;1-1-2009&quot; /> </li></ul></ul>
  27. 31. Sending Emails <ul><li>Email subsystem should be a stub in dev </li></ul><ul><li>Bypass SMTP: log to file … </li></ul><ul><li>Email should be sent only to a whitelist </li></ul><ul><li>Use Gmail Id & Atom API </li></ul><ul><li>Use Unique Id/Timestamp to distinguish </li></ul>
  28. 32. Mocking / Stubbing <ul><li>Replace module during testing </li></ul><ul><li>Different kinds: </li></ul><ul><ul><li>Sub-system (Email, Print) </li></ul></ul><ul><ul><li>Webservice (Credit Card, Flight Booking) </li></ul></ul><ul><ul><li>Hardware device (Biometric, CNC Machine) </li></ul></ul><ul><li>Return default values, implement simple/fixed logic </li></ul>
  29. 33. Mocks - 2 <ul><li>Simulate Errors and Exceptions </li></ul><ul><li>Provide more logging </li></ul><ul><li>Jmock, Dependency Injection </li></ul><ul><li>Con: </li></ul><ul><ul><li>Code level </li></ul></ul><ul><ul><li>Needs to change with API </li></ul></ul>
  30. 34. Anti-Testability Viewpoints <ul><li>Security compromise by testability interfaces, logging </li></ul><ul><ul><li>Remove or lock down in prod </li></ul></ul><ul><li>Extra coding time </li></ul><ul><ul><li>Much lower testing costs, better quality </li></ul></ul><ul><li>Privacy issues in logging </li></ul><ul><ul><li>Show only partial data </li></ul></ul><ul><li>Performance </li></ul><ul><ul><li>Almost never an issue in practice! </li></ul></ul>
  31. 35. Review <ul><li>BIST </li></ul><ul><ul><li>Automation suite w/ oracle </li></ul></ul><ul><ul><li>Unit Tests </li></ul></ul><ul><ul><li>Assertions </li></ul></ul><ul><li>Fault Simulation </li></ul><ul><ul><li>Mock/Stub </li></ul></ul><ul><ul><li>Invalid params </li></ul></ul><ul><li>Controllability </li></ul><ul><ul><li>Decoupling </li></ul></ul><ul><ul><li>Bypassing </li></ul></ul><ul><ul><li>Mock/Stub </li></ul></ul><ul><ul><li>Set/Reset </li></ul></ul><ul><li>Observability </li></ul><ul><ul><li>Logging </li></ul></ul><ul><ul><li>Reporting </li></ul></ul>