This document summarizes a presentation on improving test quality for automotive products. It discusses challenges in ensuring test program quality such as verifying test coverage and limits. A concept for a technical solution is proposed with modeling of test requirements and specifications, automated test program generation from specifications, and traceability between requirements, tests and results. A pilot project demonstrated the feasibility of the approach for microcontroller testing, with automated verification reducing errors versus manual methods.

1. Teradyne Users Group Conference | April 28 – 30, 2014 | Anaheim, California
TEST QUALITY IMPROVEMENT FOR
AUTOMOTIVE PRODUCTS
Davide Appello – STMicroelectronics
davide.appello@st.com
Tamas Kerekes – NplusT
tamas.kerekes@n-plus-t.com
1

2. MOTIVATIONS
• Quality is a key performance for products targeting automotive
applications
• Test quality has a strong impact on product quality
• Test complexity is growing in parallel with product complexity
• Thousands of tests are applied
• Tests are split in several test insertions
• Recent data shows that quality issues are dominated by “errors” in test
coverage implementation rather than by intrinsic test methods and
approaches
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 2

3. GOALS OF OUR WORK
• Automation of the annoying test development and industrialization tasks,
like:
• Verification of the expected test coverage
• Coherency with the expected test limits
• Guard-banding and Cpk analysis,
• Creation of the basis of a new toolset to provide additional advantages:
• Scalability of test program structure
• Reuse of test IPs
• Support for concurrent engineering
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 3

4. CONTENT / AGENDA
• Challenges in ensuring test program quality
• Concepts of a technical solution
• Implementation and deployment aspects
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 4

5. CHALLENGES IN ENSURING
TEST PROGRAMS QUALITY
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 5

6. CHALLENGES
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 6
Requirements (what
and how to test)
Design/Test Engineer DFT/Test Program
Does the test program
- include all the tests defined in the requirements?
- implement correctly these tests?

7. DREAM (IDEAL WORLD)
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 7
Requirements (what to test)
Test Results
Test Program
Tool

8. DREAM (“LESS IDEAL” WORLD)
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 8
Requirements (what to test)
Test Results
Test Program
Tool
List of not covered
requirements
List of not correctly
implemented tests
List of correctly
implemented tests

9. CURRENT “METHODOLOGY”
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 9
Not structured
documentation describing
test requirements
Ad-hoc information pick-
up Manual coding
Manual verification

10. Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 10
THIS PROCESS IS NOT SUITABLE TO
ENSURE TEST COVERAGE AND AS A
CONSEQUENCE, THE PRODUCT
QUALITY, ESSENTIAL IN AUTOMOTIVE
APPLICATIONS

11. “THE IDEAL WORLD”
CONCEPTS OF A TECHNICAL SOLUTION
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 11

12. TARGET PROCESS
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 12
Modeling and Formal Description of the Requirements
• Test requirements described based on a predefined model and implemented in
specific format files and/or databases.
• Support for importation of external databases and file formats.
• Possibility of manual insertion and editing.
Built-in Traceability
• Test specifications described following a predefined model and stored in a database.
• Automated connection of requirements/specification/test program/datalog.
• Manual editing.
• Possibility of import and export in files.
Automated Generation of Test Program Structure
• Test program skeleton (not the flow) is generated automatically from the test
specification (which is ATE independent) with an ATE-specific code generator.
Automated Validation of the Results
• Reference datalog is created by running the test program on a golden setup and
golden device lot.

13. TOOL ARCHITECTURE
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 13
Databases
MINT Data Model
Business Logic
GUI
• requirements
• test specifications
• test results
CustomImporters
(requirements,testprograms)
CustomGenerators
(testprogramskeletons)
(documentation)
(verification/validationreports)

14. REQUIREMENT MODELING
• Formal requirement description is fundamental
• Sources
• User (datasheet)
• Device functionality and parameters
• Ex.: “ADC linearity error < 1%”
• Design
• Verifications linked to implementation
• Ex.: ATPG test
• Process
• Verifications needed for correct execution
• Ex.: continuity test
• Quality
• For ensuring reliability and quality
• Ex.: need of burn-in
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 14

15. REQUIREMENT MODEL
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 15
Type
• Parametric or
functional test
• Single or multiple
measures
Limits
Only for parametric
• Min limit
• Max limit
CPK Target
Only for parametric
Tag
Unique identifier
Ex.: U1234
Imported or generated
automatically
Description
Free text description
Ex.: “ADC linearity
test”
Mnemonic
Short text reference
Ex.: ADC_LIN
Can be the name of
the covering test

16. TEST CONDITIONS
• A device feature might need to be tested in different conditions
• ADC linearity test at -40°C and at +125°C
• ADC linearity test at Vccmin and at Vccmax
• Combination of temperature and Vcc
• The requirement model has been extended by
• Definition of test conditions
• Manual assignment of test conditions to requirements
• Possibility to verify if matching test conditions using formulas
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 16

17. TEST SPECIFICATION MODELING
• We model the test specification in a hierarchic
structure of test blocks
• Test strategy (“production test”)
• Test insertion (“EWS1”)
• Module test (“analog”)
• Operation (“ADC linearity”)
• Support of reuse
• Parametrizable subflows
• Test IP libraries
• Dealing with project variants
• Different package, memory size, …
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 17

18. TEST SPECIFICATION EXAMPLE (1)
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 18

19. TEST SPECIFICATION EXAMPLE (2)
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 19

20. TEST SPECIFICATION EXAMPLE (3)
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 20

21. BACKBONE FOR TRACEABILITY
CONCEPTS OF A TECHNICAL SOLUTION
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 21

22. REQUIREMENT TRACEABILITY THROUGHOUT THE FLOW
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 22
Requirements
&
Test Conditions
Test Specification
Test Blocks
Test Program(s)
Datalog(s)
Requirement Tagging
Test Block Tagging
Test Names
Test Numbers/Names

23. COVERAGE ANALYSIS SUMMARY
• Requirements vs. Test Specification
• Extraction of the tags of the test blocks
• Verification of the presence of each requirement tag
• Requirements vs. Test Program
• Analysis on multiple test programs, per test insertion
• Extraction of the test names and retrieval of the tags
• Verification of the presence of each requirement tag
• Requirements vs. Datalog
• Analysis on multiple datalogs, per test insertion, by choosing a golden
device
• Extraction of test names and retrieval of the tags
• Verification of the presence of each requirement tag
• Datalog Quality
• Analysis on multiple datalogs, per test insertion, on all good devices
• Link to the requirements via tagging
• Check of the limits
• Cpk calculus and check
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 23

24. COVERAGE REPORT EXAMPLE
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 24

25. TEST SKELETON GENERATION
CONCEPTS OF A TECHNICAL SOLUTION
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 25

26. TEST PROGRAM GENERATION
• “Flattening”: puts in sequence of the test blocks, as they will be executed
• Resolves the parameter expressions
• Automatically generates unique test names from the name of the test
blocks
• Will be used for coverage analysis of the test program and of the datalog
• Creation of an output file
• Customizable format
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 26

27. TEST PROGRAM GENERATION EXAMPLE
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 27
void main()
{
Insertion_EWS1();
ModuleTest_Digital_EWS();
ModuleTest_Analog_EWS();
Operation_ADC_Linearity();
Operation_Power_On();
SubFlow_ADC_LIN();
Operation_Power_Off();
Operation_ADC_Accuracy();
}
void Insertion_EWS1()
{
}
void ModuleTest_Digital_EWS()
{
}
void ModuleTest_Analog_EWS()
{
}
void Operation_ADC_Linearity()
{
}
void Operation_Power_On()
{
}
…

28. “THE REAL WORLD”
IMPLEMENTATION AND DEPLOYMENT
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 28

29. IN THEORY…
• The “ideal process” might be a too big step in an already consolidated
workflow
• Experience of the engineering team
• Code reuse in new test programs
• Interfacing with the environment
• Requirement input/import
• Difficulty of automated import due to the not (not fully) codified requirement
database
• Legacy cases
• Existing test program pool
• Our proposed process should allow a gradual introduction
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 29
“In theory, theory and practice are the same. In practice, they
are not.” (Albert Einstein)

30. REQUIREMENT INPUT
• Challenges
• Manual input is a too big headache
• Several sources and formats
• PDF, database, …
• Solution
• Importing procedure using customizable plug-ins for the various formats
• Manual adjustment and/or input if needed
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 30

31. TEST PROGRAM GENERATION
• Option 1: fully automated
• Test engineer inputs all information in the tool
• Tool generates a final output format using an ATE-
dependent plug-in module
• Option 2: skeleton generation
• Tool generates test program frame with test names (ATE-
dependent plug-in)
• Test engineer extends it manually
• Option 3: manual
• Test engineer uses the test names suggested by the tool
(specific output file for instructions)
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 31

32. LEGACY TEST PROGRAMS
• Legacy test programs have
not been prepared to enable
fully automated analysis
• A manually supported
procedure has been
implemented
• The same procedure can be
used for datalog analysis
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 32
Parsing of the Test Program
(ATE specific plug-in)
or STDF
Generation of the Test Specification
(flat rather than hierarchic)
Manual assignment of the test
blocks to the requirements
Automated coverage analysis and
reporting

33. EXPERIMENTS AND PILOT PROJECT
IMPLEMENTATION AND DEPLOYMENT
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 34

34. PILOT PROJECT BACKGROUND
• Background
• Reuse of the experience of custom test program generation and analysis
tools
• Developed as joint ST – NplusT projects in the past years
• Skeleton generation for embedded non-volatile memory testing
• Test vector generation from high level description
• Test program change workflow management
• Test program difference detection and analysis for J750
• Built on NplusT’s new MINT technology
• Target
• Automotive microcontrollers
• J750 environment
• Limited to the test of the flash modules
• Requirement input
• User requirements imported from tagged PDF
• Manual adjustment for test conditions
• Other requirements input manually
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 35

35. PILOT PROJECT TEST PROGRAM MANAGEMENT
• Step 1:
• Existing test program
• Automated load and creation of test specification
• Manual connection to the requirements
• Step 2:
• New test program
• Test specification input via tool
• Automated test program frame generation
• Manual final test program implementation
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 36

36. CONCLUSIONS
• Automated verification process reduces the possibility of human error
in test program implementation – quantification in progress
• Automated generation tools support test engineer in the development
increasing efficiency and quality
• The concept is applicable for legacy cases as well – with more manual
operations thus with more human error possibility
Teradyne Users Group Conference | April 28 – 30 | Anaheim, California 37
Requirements
Test Specification
Test Program
Datalog
Check
&
Certification
Supported
Generation
Supported
Generation