October 2017 ● Society for Technical Communication - New England, InterChange Conference, UMass Lowell The importance and long-term impact of testing examples in documentation. Impact on product, documentation, and customers. Relationship of technical writing to quality testing, marketing, and training.

1. Documentation Example Testing:
Value & Impact
Dallas C. Kennedy
Ab Initio Software LLC
Society for Technical Communication – New England
InterChange Conference 2017
UMass Lowell

2. What Is Doc Example Testing All About?
• In the context of software … although you might do similar things
elsewhere ….
• Examples are essential to users in any documentation and structured
learning beyond the most superficial.
Explain it to me … Show me … Involve me …
… in ascending order of effectiveness
• Testing your doc examples …
… keeps examples working,
… keeps examples relevant,
… tests new and updated features as they appear
… and much more!

3. More about the speaker
• Technical writer in the Boston area since 2000
• Currently at Ab Initio Software in Lexington* …
… Previously at MathWorks in Natick
• Wide experience in documenting complex software systems for
advanced technical and data transformation applications
• Co-author of multiple documentation sets …
… sole author of several significant product documentation suites and books
• About 15 years experience in testing documentation examples
• Previous experience in physics and astrophysics research, including
creating, testing, and using numerical programs for analysis and prediction
* Talk not sponsored or endorsed by Ab Initio

4. Testing Your Documentation
and Products

5. What are the value and impact of
example testing?
As you test and validate your doc examples, you are doing something
vital. You are:
• Keeping your examples working and relevant
• Pre-emptively removing a major source of user frustration and
support requests, saving your company money and time
• Testing new features, and upgrades to existing features, as they
emerge from development
• Developing insight into products, features, tasks, and customers

6. Deeper cumulative impact
Pursued consistently and long term, doc example testing
accomplishes less obvious and more profound things:
• It shapes how you choose and construct examples.
• Can your example be tested properly? If not, why? If not, maybe it
shouldn’t be a doc example, or maybe it needs to be rewritten.
• It keeps you from tolerating bad examples whose weaknesses are
obscured by fancy pictures and words.
• It checks your products for regressions – if your examples are
supposed to continue to work as new versions of software appear.
• Developers can be surprised by this, but they shouldn’t be:
Realistic examples and edge cases often test high-level, complex functioning
missed by simple tests (like unit tests).

7. How Doc Example Testing Works in Practice

8. Where do documentation examples come from?
Examples have many sources:
• Development
• Quality engineering and its test cases …
… especially more complex cases and work flows
• Technical marketing
• Training
• Customers  Lots of “street cred” when you publish examples and
applications openly co-authored with customers

9. Two broad approaches to testing …
• Automated – Convenient, scalable, fast
• Examples can be automatically co-tested with demos and the product itself.
• Automated testing must often meet important restrictions.
• Manual – Sometimes more appropriate for complex examples: complex
workflows often most valuable tests, most difficult to implement.
… and related in a deep way to doc usability testing – testing that checks the usability, not of
your product, but of your documentation and specifically of your examples.
• Doc usability testing is time-consuming and requires willing, live human testers
in a usability lab. Done right, it is almost always worthwhile. Tests user’s reaction
to examples and features.
Several things happening at once worth disentangling:
• The product and various discrete features
• The example
• How well the doc presents the product and feature
• How well the doc presents the example

10. Automated testing
• Most easily implemented with code or code-like examples.
• Most sophisticated versions of testing automatically extract and
reformat examples from the documentation and pass the example
code to a product test harness for execution and evaluation.
• Requires coordination of processes: product and feature
development and testing, and documentation authoring and build.
Doc
source
Extracted and
reformatted
examples
Product
test
harness
Example
test
results

11. Automated testing (continued)
• Graphical interface (GUI) testing more difficult to implement,
fewer tools to automate, although not impossible
• GUI-based examples: various approaches available
• If scriptable, entire example can be automated with quality testing
on the development side in the engineering test suite.
Some GUI-based products have scripting or similar languages behind the GUI
application. Others can provide handles to graphical objects.
• In other cases, can use automated GUI tools like AutoIt to develop a
macro- or script-like automated test, often with real system scripts.
• Can always be tested manually, at cost of consuming more time.
• Co-test with training and marketing videos and other materials.

12. Different levels of testing rigor
• Basic and least rigorous – Verification of procedures: Are
prerequisites met? Does example finish? Does it freeze or crash?
• Deeper rigor – Parsing of syntax, compilation of code, verification of
structure and elements of example
• Deepest rigor – Evaluation of specific results – numeric,
alphanumeric, graphical, etc. Sets of baselines.
• Baselines = Numeric and other specific test results
Baseline testing is tricky in an environment where results can change
from run to run; for example:
• Dynamic engineering and scientific simulations
• Generation of random numbers or sequences
Dependence on platform and machine state

13. Building Relationships

14. The other side of example testing
• Example testing checks products and examples.
• It also builds relationships in your organization.
• Technology work is usually divided into functional areas:
• Development = primary engineering of products and features
• Quality or Testing = verification and validation of product functionality
• Documentation = explanation to users of product
• Marketing* = showing current and potential customers what your products can
do and how they might solve customers’ problems
• Training* = educating new and current users about products and features
• Example testing touches each of these areas and requires cooperation
among all of them, especially between Documentation and Quality.
* Customer-facing

15. Creates and enhances flow of information
about products, features, and examples
Products
Features
Examples
Development
Quality
Documentation
Marketing
Training

16. Mutually reinforcing supports of each:
• Development generates products, features, in response to projected
or actual customer needs  examples
• Quality builds product test harness, generates test cases 
examples   test cases from Development, Doc, Marketing
• Documentation documents examples, tests examples and test cases
with Quality  more test cases
• Marketing studies and forecasts customers’ problems  test cases
 feedback from customers
• Training uses and possibly generates examples  examples and test
cases for Doc and Quality  feedback from experience with users

17. Building most important relationship …
… with customers!
• Important to concentrate documenting and testing examples on more
complete, more realistic examples …
… not unit tests and other basic functional tests.
Basic testing important for Development and Quality, not so much for Doc,
Marketing, or Training.
• Such examples are what customers need and want.
Documenting and validating such examples – keeping them
fresh, relevant, and functional – together form an essential
source of value to your product’s users.

18. Some References
• Software testing:
• B. Laboon, A Friendly Introduction to Software Testing
• M. Feathers, Working Effectively with Legacy Code
• No good book sources on testing documentation examples –
Limited material available on doc usability testing
• See latest and greatest online courses from Coursera, EdX, Udacity,
MS Virtual Academy, and so on, concerning software testing and
especially writing test cases.
Email: dalet@stanfordalumni.org
Email: dkennedy@abinitio.com
LinkedIn: linkedin.com/in/dckennedy/