Conformance testing and standards


                            How do you know it works
                            if yo...
Standards make the world go round




http://jcp.org          2
Language




http://jcp.org   3
Writing




http://jcp.org   4
Number systems




http://jcp.org   5
Currency




http://jcp.org   6
Time...




http://jcp.org   7
And space




http://jcp.org   8
Weights and measures




http://jcp.org         9
Machine tools




http://jcp.org   10
Screws and threads




http://jcp.org       11
Interchangeable parts




http://jcp.org          12
Railways




http://jcp.org   13
Mass production




http://jcp.org    14
Shipping




http://jcp.org   15
Traffic




http://jcp.org   16
Beer




http://jcp.org   17
Chocolate




            WHO/FAO: Codex Alimentarius Official Standard for Chocolate
http://jcp.org                      ...
Music




                 ISO 16:1975 Acoustics -- Standard tuning frequency
                             (Standard music...
Color




                 http://www.color.org
http://jcp.org            20
Sport




             http://www.lords.org/laws-and-spirit/laws-of-cricket/laws/
http://jcp.org                         21
Medicine

                      Chronic rheumatic heart diseases
                      I05: Rheumatic mitral valve disease...
Shopping




http://jcp.org   23
Home entertainment




http://jcp.org       24
The Holy Grail


                      • Components,
                       systems, and
                       processes ...
Then




http://jcp.org   26
Now

• We are no longer willing to buy all of our hardware and
  software from a single supplier.
• We want the freedom to...
Interfaces




http://jcp.org   28
Protocols




http://jcp.org   29
No vendor lock-in




http://jcp.org      30
Specifications...




http://jcp.org      31
Are not enough...

• Interoperability and interchangeability are harmed by:
• Poor-quality specs.
    – Imprecise or ambig...
We also need testing




http://jcp.org         33
Conformance testing

• The process of verifying that implementations of a
  technology conform to the specification.
• Tes...
What makes a good spec?

• Specify.
    – Unspecified or implementation-specific behaviour can't
      be tested.
• Requir...
Java: what?



                      Open, standards-based
                      technologies enabling
                   ...
Java: how?




http://jcp.org   37
Deliverables




http://jcp.org   38
Deliverables




http://jcp.org   39
Java conformance testing




http://jcp.org             40
Why conformance-test?

• To promote the compatibility and interoperability of Java
  technology implementations.
• To ensu...
Compatibility

• Compatibility is a contractual obligation.
    – Shipping incompatible products is prohibited.
• Compatib...
Planning and building a high-quality TCK




http://jcp.org          43
A TCK is not just a collection of tests

• It should also include:
    – A test harness to automate execution.
    – Docum...
A good test is...

• Mappable to the specification.
    – You must know what portion of the specification it tests.
• Atom...
Specification markup

• Identify normative requirements (test assertions)
  within the spec.
• Provide feedback to the aut...
Test assertions

• A specific statement of functionality or behavior
  derived from a specification.
    – java.lang.Integ...
How many tests are enough?

 • There is no simple answer to this question.
      – It depends on your goals and on the ava...
Calculating coverage




http://jcp.org         49
Coverage analysis

 • Partition the spec.
      – By feature, APIs, language elements, testable assertions,
        logica...
Test development strategy

 • Define coverage goals.
      – Where should resources be focused?
      – How extensively sh...
What to test and what not to test?

• “Full coverage” for the majority of real-world specs is
  impossible (impractical.)
...
Spec + RI + TCK =




http://jcp.org      53
Upcoming SlideShare
Loading in...5
×

Patrick jcp

458

Published on

Published in: Technology, Health & Medicine
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
458
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Patrick jcp

  1. 1. Conformance testing and standards How do you know it works if you don't know what it's supposed to do? Patrick Curran JCP Chair patrick@jcp.org http://jcp.org 1
  2. 2. Standards make the world go round http://jcp.org 2
  3. 3. Language http://jcp.org 3
  4. 4. Writing http://jcp.org 4
  5. 5. Number systems http://jcp.org 5
  6. 6. Currency http://jcp.org 6
  7. 7. Time... http://jcp.org 7
  8. 8. And space http://jcp.org 8
  9. 9. Weights and measures http://jcp.org 9
  10. 10. Machine tools http://jcp.org 10
  11. 11. Screws and threads http://jcp.org 11
  12. 12. Interchangeable parts http://jcp.org 12
  13. 13. Railways http://jcp.org 13
  14. 14. Mass production http://jcp.org 14
  15. 15. Shipping http://jcp.org 15
  16. 16. Traffic http://jcp.org 16
  17. 17. Beer http://jcp.org 17
  18. 18. Chocolate WHO/FAO: Codex Alimentarius Official Standard for Chocolate http://jcp.org 18
  19. 19. Music ISO 16:1975 Acoustics -- Standard tuning frequency (Standard musical pitch) http://jcp.org 19
  20. 20. Color http://www.color.org http://jcp.org 20
  21. 21. Sport http://www.lords.org/laws-and-spirit/laws-of-cricket/laws/ http://jcp.org 21
  22. 22. Medicine Chronic rheumatic heart diseases I05: Rheumatic mitral valve diseases Includes: conditions classifiable to 105.0 and 105.2-105.9, whether specified as rheumatic or not Excludes: when specified as nonrheumatic I05.0: Mitral stenosis Mitral (valve) obstruction (rheumatic) I05.1: Rheumatic mitral insufficiency Rheumatic mitral ● Incompetence ●Regurgitation I05.2: Mitral stenosis with insufficiency Mitral stenosis with incompetence or regurgitation I05.8:Other mitral valve diseases Mitral (valve) failure I05.9: Mitral valve disease, unspecified Mitral (valve) disorder (chronic) NOS From the World Health Organization International Classification of Diseases http://jcp.org 22
  23. 23. Shopping http://jcp.org 23
  24. 24. Home entertainment http://jcp.org 24
  25. 25. The Holy Grail • Components, systems, and processes that are: – Reproducible – Predictable – Reusable – Interoperable – Interchangeable http://jcp.org 25
  26. 26. Then http://jcp.org 26
  27. 27. Now • We are no longer willing to buy all of our hardware and software from a single supplier. • We want the freedom to chose and the option to switch. – All systems are heterogeneous. • This requires standards. – For interfaces, so we can mix and match components. – For protocols, so systems can talk to each other. http://jcp.org 27
  28. 28. Interfaces http://jcp.org 28
  29. 29. Protocols http://jcp.org 29
  30. 30. No vendor lock-in http://jcp.org 30
  31. 31. Specifications... http://jcp.org 31
  32. 32. Are not enough... • Interoperability and interchangeability are harmed by: • Poor-quality specs. – Imprecise or ambiguous, language. • Poor-quality implementations. – Specified requirements are not met. – Specified requirements are implemented incorrectly. http://jcp.org 32
  33. 33. We also need testing http://jcp.org 33
  34. 34. Conformance testing • The process of verifying that implementations of a technology conform to the specification. • Tests only what is normatively required in the specification. – Quality, robustness, performance, usability, and other desirable attributes of software must not be tested (unless explicitly specified.) • Can make no assumptions about the internals of the implementation (black-box testing.) • Improves the quality of specifications: – by identifying ambiguities, contradictions, omissions, • And of implementations: – by identifying failures to conform to the spec. http://jcp.org 34
  35. 35. What makes a good spec? • Specify. – Unspecified or implementation-specific behaviour can't be tested. • Require. – In clear, unambiguous language (see RFC 2119) – We like “must,” “shall,” “must not”... – We don't like “may,” “it's obvious,” “it's up to you”... • Beware optional functionality. – Can be tested, but doesn't promote interoperability or application portability. • Developers won't know what they can depend on. – If you must, clearly define optionality with Profiles. http://jcp.org 35
  36. 36. Java: what? Open, standards-based technologies enabling the development and deployment of secure, portable, reliable, and scalable applications on hardware platforms from cellphones to high-end servers. http://jcp.org 36
  37. 37. Java: how? http://jcp.org 37
  38. 38. Deliverables http://jcp.org 38
  39. 39. Deliverables http://jcp.org 39
  40. 40. Java conformance testing http://jcp.org 40
  41. 41. Why conformance-test? • To promote the compatibility and interoperability of Java technology implementations. • To ensure that the technologies are well specified and that implementations conform to the specifications. • Results: – Multiple compatible implementations are available. – Developers know how implementations will behave. http://jcp.org 41
  42. 42. Compatibility • Compatibility is a contractual obligation. – Shipping incompatible products is prohibited. • Compatible products can use the Java name and display the Java Compatible logo. • Compatibility is binary. – You can't be “almost compatible” or “a little bit incompatible.” – You must pass all the tests and meet all of the compatibility requirements. http://jcp.org 42
  43. 43. Planning and building a high-quality TCK http://jcp.org 43
  44. 44. A TCK is not just a collection of tests • It should also include: – A test harness to automate execution. – Documentation explaining • How to run the test suite. • How to interpret test results. • Compatibility Requirements (The Rules.) • The test appeals process. • It must be portable. – Unlike most other software, a TCK must be capable of running on systems that don't yet exist. • You can't test it on the platforms where it will be run! • The Spec Lead must commit to ongoing maintenance. – Fix bugs, expand coverage. http://jcp.org 44
  45. 45. A good test is... • Mappable to the specification. – You must know what portion of the specification it tests. • Atomic. – Tests a single feature rather than multiple features. • Self-documenting. – Explains what it is testing and what output it expects. • Focused on the technology under test rather than on ancillary technologies. • Useful. – Likely to catch real-world problems. • Correct, efficient, portable, and maintainable. http://jcp.org 45
  46. 46. Specification markup • Identify normative requirements (test assertions) within the spec. • Provide feedback to the authors where the spec is ambiguous, contradictory, incomplete, or untestable. • Publish an assertion list. – Ask the spec authors to review and approve it. • This process significantly improves spec quality. http://jcp.org 46
  47. 47. Test assertions • A specific statement of functionality or behavior derived from a specification. – java.lang.Integer.toString(int i, int radix) • "If the radix is smaller than Character.MIN_RADIX or larger than Character.MAX_RADIX, then the radix 10 is used instead." – “During preparation of a class or interface C, the Java virtual machine also imposes loading constraints (§5.3.4). Let L1 be the defining loader of C. For each method m declared in C that overrides a method declared in a superclass or superinterface, the Java virtual machine imposes the following loading constraints: Let T0 be the name of the type returned by m, and let T1, ..., Tn be the names of the argument types of m. Then TiL1=TiL2 for i = 0 to n (§5.3.4).” http://jcp.org 47
  48. 48. How many tests are enough? • There is no simple answer to this question. – It depends on your goals and on the available resources. • Aim to get the best possible coverage with the resources you have available. • You cannot do this unless you set explicit goals, and measure or estimate test coverage. http://jcp.org 48
  49. 49. Calculating coverage http://jcp.org 49
  50. 50. Coverage analysis • Partition the spec. – By feature, APIs, language elements, testable assertions, logical sections, or even pages or paragraphs. • Estimate or measure the extent of coverage in each area • Breadth coverage (relatively simple) – What percentage of spec elements are covered by at least one test? • Depth coverage (more subjective) – On average, what percentage of the tests that would be required to completely test each element have actually been written? • (How thoroughly is each element tested?) http://jcp.org 50
  51. 51. Test development strategy • Define coverage goals. – Where should resources be focused? – How extensively should each area be tested? • Start with breadth (test everything minimally.) • Drill down (increase depth coverage) in selected areas. • Publish a test-coverage report. – Minimally, map tests to areas of the spec. – Ideally, provide counts and averages of the number of tests in each area. • This helps users to understand the strengths and weaknesses of the test suite. • It will also help you to improve the next version. http://jcp.org 51
  52. 52. What to test and what not to test? • “Full coverage” for the majority of real-world specs is impossible (impractical.) • Don't just test what's easiest. • Focus on areas where: – The consequences of non-conformance are greatest. • Eg, breaking interoperability or jeopardizing security. – Implementations are more likely to be non-conformant because: • Implementation presents technical difficulties. • The specification is ambiguous. • Implementers are less likely to discover problems. • Implementers have an incentive to cheat (eg, to increase performance.) http://jcp.org 52
  53. 53. Spec + RI + TCK = http://jcp.org 53
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×