on to conformance and interoperability testing Beyond functional testing By: Derk-Jan de Grood Date: Oct 2009 Location: QA & Test Bilbao

Objectives for this presentation The relevance of conformance en interoperability testing Why is striving for interoperability a challenge Conformance testing, how to…? The difference between conform and certified . www.collis.nl

The relevance of conformance en interoperability testing

Why is striving for interoperability a challenge

Conformance testing, how to…?

The difference between conform and certified .

Announcement www.collis.nl an 'All True' presentation Certificate Signed, 20 Sept 2008, Leiden, the Netherlands Neighbor of Derk-Jan

Experience Collis has track a record in conformance testing: Design, realization & maintenance of conformance test environments, e.g.: Gezundheidskarte (Germany), VISA/Mastercard validated test suites for testing payment applications in Conclusion ®. Execution of conformance testing, e.g.: ATM test service on behalf of Equence. Smart Card testing (e.g. ISO-7816,ISO-14443) Interoperability testing for Public Transport Card Auditing/ 2 nd opinion or coordination, e.g.: IVERA: Traffic light protocol audit . EM radiation and immunity test of industrial PC. www.collis.nl

Collis has track a record in conformance testing:

Design, realization & maintenance of conformance test environments, e.g.:

Gezundheidskarte (Germany),

VISA/Mastercard validated test suites for testing payment applications in Conclusion ®.

Execution of conformance testing, e.g.:

ATM test service on behalf of Equence.

Smart Card testing (e.g. ISO-7816,ISO-14443)

Interoperability testing for Public Transport Card

Auditing/ 2 nd opinion or coordination, e.g.:

IVERA: Traffic light protocol audit .

EM radiation and immunity test of industrial PC.

Conformance and Interoperability

Bug report Change the spec not the code . Implementation works better than designed www.collis.nl

Change the spec not the code . Implementation works better than designed

But not always Certain branches conformance to the norm and guarantied interoperability is crucial . www.collis.nl

Certain branches conformance to the norm and guarantied interoperability is crucial .

Definition ? www.collis.nl Interoperability = the capability of different programs to exchange data via a common set of business procedures, use the same file formats and protocols. Conformance = The extend in which a process or system satisfies a standard.

Example 1: size and composition www.collis.nl

Example 2: COTS www.collis.nl [Susan Waucomont – Bits & Chips testing dag 2007]

Example 3: SOA www.collis.nl [Source:www.hinttech.com]

Interoperability

Systems working together Vertical Inter-operability

Horizontal interoperabilty www.collis.nl

Cross-over testing … www.collis.nl Card type works with ATM type ATM 1 ATM 2 ATM 3 ATM 4 Card I.a     Card I.b     Card II     Card III    

… has its disadvantages It can be quite extensive. How to test future systems? Problems are found late in the development cycle. Retesting can be quite extensive www.collis.nl

It can be quite extensive.

How to test future systems?

Problems are found late in the development cycle.

Retesting can be quite extensive

Test before you integrate… www.collis.nl

Interoperability by setting the standard www.collis.nl

… Conformance testing www.collis.nl If you decide to use a standard, you should also test the conformance to it.

Conformance adds up to… www.collis.nl   + ?

Chain test www.collis.nl

Check to connect www.collis.nl

Beyond functional testing

Conformance Testing www.collis.nl Coverage: Risk based testing or ALL testing ? I know functional testing, what’s different in conformance testing? Involvement with s/w development ? Test automation ? Quality of test base ? (un) Ambiguity of the test set ?

Completeness of the specification www.collis.nl

Completeness of the specification www.collis.nl

Stability of the standard Unlike many specifications, standards have a high stability. www.collis.nl

Unlike many specifications, standards have a high stability.

Coverage with standard www.collis.nl In standard ? Risk: The system contains bugs that will not be found. Specified functions are not tested Risk: The system works different than expected. The tests suggest that the system behaves as expected, but the expectation is not specified. Risk: The system might contain unknown functions of unknown quality and impact with other functions. In testdesign ? Y N Y √  N  ?

(un) Ambiguity of test design www.collis.nl

Certification

Conformance vs. Certification = (legal) Liability www.collis.nl +

= (legal) Liability

Functional testing @ s/w factory www.collis.nl Dev. Team Test Team Involvement test team Discuss strategy Review specifications Perform Risk analysis Join triage meetings Give release advice

Involvement test team

Discuss strategy

Review specifications

Perform Risk analysis

Join triage meetings

Give release advice

Certification process www.collis.nl Manufacturer Test centre Certifying Party 1. Accreditation 2. Request for conformance test 3. Conformance test passed 4. Request certificate 5. Certification

Organization of the tests Certifying party has major role Setting up the standard Being the oracle Pre-describing the tests Making sure tests execution is objective Granting the certificate www.collis.nl

Certifying party has major role

Setting up the standard

Being the oracle

Pre-describing the tests

Making sure tests execution is objective

Granting the certificate

Bug fix and solution www.collis.nl Deviation report states the differences with the standard. Finding the source of the deviation and determining the solution is responsibility of the manufacturer. Dev. Team Test Team

Deviation report states the differences with the standard.

Finding the source of the deviation and determining the solution is responsibility of the manufacturer.

TAAS www.collis.nl

Test automation Business case Objectivity Technical necessity www.collis.nl

Business case

Objectivity

Technical necessity

Risk based testing Test every aspect of the standard. www.collis.nl No risk ? No test ! The implementation conformance statement (ICS) tells where to focus on. Test functions that are likely to deviate first.

Test every aspect of the standard.

The implementation conformance statement (ICS) tells where to focus on.

Test functions that are likely to deviate first.

Getting a certificate

Any one can give a certificate And some have an added value: www.collis.nl an 'All True' presentation Certificate Signed, 20 September 2007, Leiden, the Netherlands Neighbor of Derk-Jan

And some have an added value:

Certification process www.collis.nl Manufacturer Test centre Certifying Party 1. Accreditation 2. Request for conformance test 3. Conformance test passed 4. Request certificate 5. Certification

Wrap up

Wrap up Interoperability, Conformance operations Purpose shifts from ‘Fit for purpose’ to ‘conform the standard’. Different organization, TAAS Challenge of having critical test designs Mistakes can have big impact www.collis.nl

Interoperability, Conformance operations

Purpose shifts from ‘Fit for purpose’ to ‘conform the standard’.

Different organization, TAAS

Challenge of having critical test designs

Mistakes can have big impact

Differences at a glance www.collis.nl Functional testing Conformance Testing Scope Fit for purpose Conform the standard Test basis Changes often Stable Test environment 1-to-1 relation 1-to-n relation Test design Made by test team Often prescribed Reuse of test design Limited Many times Test automation Can be useful Often very useful Risk Based Testing Much added value Limited added value Bug fix and solution Involvement Up to manufacturer Organization Often in projects Often as a service

Further reading www.collis.nl Contains a section on Conformance testing ISBN 9783540788287

Questions? www.collis.nl Derk-Jan de Grood [email_address] Collis De Heijderweg 1 2314 XZ Leiden The Netherlands www.testgoal.com Leiden Singapore Minneapolis Edinburgh