In the semiconductor industry, Design For Testability (DFT) is an essential part of the architecture and design of components. Software designers on the other hand do not pay much (if any) attention to the testing needs of their code. In this talk we review some core DFT principles like Built-In Self Test, Test Point Insertion, Fault Modeling and Fault Simulation and map them to software testing. Examples of using DFT to create testable software will be given. DFT fits in especially well with the increasing use of Test Automation and Agile Methodologies. We hope this talk will empower test leads and engineers with knowledge they can use to get their developer counterparts to modify the application-under-test to significantly increase automation, enhance test coverage, run tests faster and reduce the costs of testing.

1. Designing Software For Testability
A forgotten design pattern
Rohit Nayak
Talentica

2. Agenda

Background and Motivation

DFT in VLSI and hardware design

Issues in Software Testing

DFT for Software Development

Automation and Testability

Some Examples

Cons, Review

3. The Anti-Pattern
If it works,
its the developer,
if not blame QA!

4. VLSI/PCB Testing Issues

Mass produced

Each piece needs to be validated

Complexity
− Number of subsystems
− Amount of logic

Access to internal logic

Testing costs

Cost of recall

6. Testability:
“Effort required to
test a product to
ensure that it
performs its intended
function”

7. DFT Principles

Controllability

Observability
− Test Point Insertion

Built-In Self Test (BIST)

Fault Modelling

Fault Simulation

Test Pattern Generation

10. Software Testing
“Testability is a design issue and needs to be
addressed with the design of the rest of
the system. Projects that defer testing to
the later stages of a project will find their
programmers unwilling to consider
testability changes by the time testers
actually roll onto the project and start
making suggestions. Testability takes
cooperation, appreciation and a team
commitment to reliability.”
- Bret Pettichord

11. Cost Of Bugs
Where Found Requirements Architecture Coding System Test Production
Where Introduced
Requirements 1x 3x 5-10x 10x 10x-100x
Architecture 1x 10x 15x 25-100x
Coding 1x 10x 10-25x

13. Software Testability

Suitability: clarity of specifications QA has

Observability: we can only test what is visible

Simplicity: easier design/UI makes testing easier

Controllability: better the control, better the coverage
and automation

Stability: how often it changes

Performance: how fast it works

Diagnosability: writing effective bug reports!

14. Software DFT Patterns

BIST 
Controllability
− Automation suite − Decoupling
w/ oracle − Bypassing
− Unit Tests − Mock/Stub
− Assertions − Set/Reset

Fault Simulation 
Observability
− Mock/Stub − Logging
− Invalid params − Reporting
− Test Interfaces

15. Good Automated Tests are

Repeatable

Easy to Write

Easy to Understand

Fast

Way Easier with a Testable Software

16. When/How to Automate?

Manager & Team are commited

Testers/Dev either experienced or interested
in learning to script

Product release cycles managed well

Functionality / UI changes under control

Early/Incremental approaches work best

Build integration, reporting

Each bug results in a automated test

17. Automation As Coding

Automation scripts and frameworks are CODE!

Use tools and scripting languages

Evaluate tools on real problems

Follow development processes
− Spiral/Agile

Use Source Control same as rest of code

These will have bugs as well!

18. Test “oracle”

Expected Result
− Generated once from previous run or
− Manually specified or
− Legacy system

Time adjustment

Results
− Database
− File
− Inline with code

19. Test Environment

Multiple VMs

Automated installs/images

Automation tools

Bug reporting tools

20. Unit Tests

Each module has independent set

Developer written/maintained

setup, teardown

xUnit, TestNG

Runs on build

Can run sub-set on install

TDD

21. Diagnosability

All environments: Test / Staging / Production

Errors result in visual messages

Errors are raised where they occur

Errors can be localized

Details are sufficient to fix issue

Ability to send diag data to dev

22. Logging Module

Log levels (VERBOSE, INFO, WARNING, ERROR, CRITICAL, TIME)

Define CRITICAL/ERROR levels well and use them!

These should result in urgent notifications

Lower levels in Production

Time, App, Component, ThreadId, Message
8/2/2010 14:22pm: Notifier: Pop3: 8242: Sending welcome email to userid 334

Delimited columns for import

Rotation based on Time, Size

23. Built-in Self Test

Inserting test code/interfaces

Set/Reset to bring state to known value

Reporting to get current state

Assertions about values/state
assert(order.billed==true)

“oracle” based regression

24. Some Examples

Web Application
− Software As A Service
− Browser Only UI
− Ajax / Dynamic HTML

Banking (Client-Server)
− Installed Application
− Desktop Client
− Server API

25. Web Application

HTML Page Title

Id values for important divs/controls

Hidden values (non-textual, graphs, tables)

Measurable Ajax responses

Tools: Selenium, Sahi, Watir, WebTest, curl

Logging incl. browser/ip/session cookie

Ability to simulate time zone, language

26. Web Application - 2

Bypassing Captcha

Mock TP APIs eg. Facebook, Google, OpenId

SSL bypassing

Transactions to be voided

Multiple runs

Always initialise controls
− Browser autofill

27. Desktop

Unique identifiers to GUI controls

Key Replay tools, SendKeys

API test suites bypassing UI

Client logs, ability to email

Ability to run db queries in scripts

28. Desktop - 2

Automation Friendly Third Party Controls
− Ability to select cell
− Copy-enable text fields
− Key shortcuts to Forms UI for control focus,
clicking, navigation
− Access by value (tree/list controls)

OCR for images/text consoles

29. Bypassing

Credit card payment
− Test cards
− Mock object
− Dev auto-approval bypass code

Order placement
− Dummy users, auto-fulfill
− Dummy vendors (email order)
− Admin screens move

30. Dates

Never use system date directly

Config override
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key=“SystemDate" value=“$now">
</appSettings>
</configuration>

Nant: XMLPoke
<xmlpoke file="App.config"
xpath="/configuration/appSettings/add[@key = 'SystemDate']/@value"
value="1-1-2009" />

31. Sending Emails

Email subsystem should be a stub in dev

Bypass SMTP: log to file …

Email should be sent only to a whitelist

Use Gmail Id & Atom API

Use Unique Id/Timestamp to distinguish

32. Mocking / Stubbing

Replace module during testing

Different kinds:
− Sub-system (Email, Print)
− Webservice (Credit Card, Flight Booking)
− Hardware device (Biometric, CNC Machine)

Return default values, implement simple/fixed
logic

33. Mocks - 2

Simulate Errors and Exceptions

Provide more logging

Jmock, Dependency Injection

Con:
− Code level
− Needs to change with API

34. Anti-Testability Viewpoints

Security compromise by testability interfaces,
logging
Remove or lock down in prod

Extra coding time
Much lower testing costs, better quality

Privacy issues in logging
Show only partial data

Performance
Almost never an issue in practice!

35. Review

BIST 
Controllability
− Automation suite − Decoupling
w/ oracle − Bypassing
− Unit Tests − Mock/Stub
− Assertions − Set/Reset

Fault Simulation 
Observability
− Mock/Stub − Logging
− Invalid params − Reporting