Open testing to verify JDK binaries in an effort to "make quality certain to happen".

1. AdoptOpenJDK Quality Assurance (AQA)
Shelley Lambert
July 17, 2019

2. What is AQA?
 Set of tests run against JDK binaries to “make
quality certain to happen”
 Addresses enterprise customer requirements
that JDK binaries must be:
– Functionally correct
– Performant
– Secure
– Scalable
– Robust
 Tests selected in the spirit of the AQA manifesto

3. AQK manifesto
AQA should be:
• open & transparent
• diverse & robust set of test suites
• evolving alongside implementations
– continual investment
– process to modify (TSC / quarterly review)
– codecov & other metrics
– comparative analysis
• portable
• tagged/tracked & published

4. AdoptOpenJDK Quality Assurance
• “Make quality certain to happen”
• Testing a wide criteria representing actual enterprise requirements to
identify binaries ready for production usage
Today AQA Roadmap
Functional correctness Security
OpenJDK regression (open) Passes known vulnerability tests
Oracle JCK (closed) Functional correctness
OpenJDK regression
Builder-specific testing (unknown) Eclipse functional
Application & framework tests
Performance
Published metrics
Achieves minimum throughput
scores
Scalability & durability
Load & stress testing

5. AQA in practice
• See https://github.com/AdoptOpenJDK/openjdk-tests/issues/1166
• AQA v1.0 definition
• Mark JDK binaries if they pass defined set of
AQA tests
• BoM (Bill of Materials) for all tested releases
– SHAs of all test material
– summary of test results & exclusions available for
all binaries

6. AQK 1.0: Definition
openjdk (8) system (62) external (4) perf (4) functional (45)
sanity.openjdk = 8
targets
sanity.system = 38
targets (+ 16)
sanity.external = 4
targets
sanity.perf = 4
target
sanity.functional
extended.openjdk extended.system =
4 targets (+ 4)
extended.external extended.perf extended.function
al
special.openjdk special.system special.external special.perf special.functional=
45
• diverse & robust
 123 test targets span regression, performance, app and stress categories
 Applicable to all implementations, platforms and versions
 Passing on all current build variations on current set of test hardware*
*Exclude implementation specific tests (example: DAA load tests not included)
(numbers in brackets have some disabled tests on some version/platforms)
Test Groups

7. AQK 1.0: Definition (groups x specs x versions)
Platform/spec = os + arch + variant
• Linux: Linux: x86_64, x86_64_XL, s390x, ppc64le, aarch64, aarch32
• Windows: x86_64, x86_64_XL, x86_32
• Mac: x86_64, x86_64_XL
• AIX: ppc64
• Solaris: sparcv9
Versions
• 8
• 11
• 12
• 13
Currently testing:
Build_Impl
• adopt_hotspot
• adopt_openj9
• upstream

8. jdk8
J9 HS Upstream
jdk12
J9 HS
jdk13
J9 HS
jdk11
J9 HS Upstream
x64linux
x64linux_XL
s390xLinux
ppc64leLinux
aarch64Linux
arm32Linux
x64Win
x64Win_XL
x32Win

9. jdk8
J9 HS Upstream
jdk12
J9 HS
jdk13
J9 HS
jdk11
J9 HS Upstream
x64Mac
x64Mac_XL
ppc64Aix
solarisSparcv9

10. • group=openjdk
• Targets (8 + 15):
– sanity.openjdk = jdk_math jdk_math_jre jdk_lang jdk_io
jdk_nio jdk_util jdk_rmi jdk_net jdk_security1
– extended.openjdk = jdk_beans jdk_other jdk_security2
jdk_security3 jdk_security4 jdk_text jdk_time
jdk_management jdk_jmx jdk_tools jdk_jdi jdk_jfr, jdk_awt
jdk_swing jdk_sound
openjdk
sanity.openjdk
jdk_math
Java/math/BigDecimal/NegateTests.java

11. • group=system
• Targets (62 + 8) :
– sanity.system = ClassLoadingTest MauveSingleThreadLoadTest MauveSingleInvocationLoadTest
MauveMultiThreadLoadTest NioLoadTest LambdaLoadTest TestJlmRemoteClassNoAuth
MathLoadTest_autosimd MauveSingleThreadLoadTest MauveSingleInvocationLoadTest
MauveMultiThreadLoadTest CpMpTest_CpMp CpMpTest_MP CpMpTest2 CpMpTest3 CpMpModularJarTest
CpMpModularJarTest2 CpMpModularJarTest3 JDKInternalAPIsTest AutomaticModulesTest1
AutomaticModulesTest_ImpliedReadabilityTest1 AutomaticModulesTest_ImpliedReadabilityTest2
AutomaticModulesTest_ImpliedReadabilityTest3 JDKInternalAPIsTest ServiceLoadersTest
PatchModuleTest_AppModPatchModule PatchModuleTest_AdvancedPatchModule
PatchModuleImageTest_AppModPatchModule PatchModuleImageTest_UnexportedTypePatchModule
PatchModuleImageTest_AdvancedPatchModule UpgradeModPathTest_ExpDirModUpgrade
UpgradeModPathTest_ExpDirModUpgradeCRImage UpgradeModPathTest_JarredModUpgrade
UpgradeModPathTest_JarredModUpgradeCRImage CpMpJlinkTest CLTest CLTestImage CLLoadTest,
– TestJlmLocal TestJlmRemoteClassAuth TestJlmRemoteMemoryAuth TestJlmRemoteMemoryNoAuth
TestJlmRemoteNotifierProxyAuth TestJlmRemoteThreadNoAuth MathLoadTest_bigdecimal ExplicitModulesTest
PatchModuleTest_PlatformModPatchModule PatchModuleImageTest_PlatformModPatchModule
JlinkTest_RequiredMod JlinkTest_AddModLimitMod JlinkPluginOpt_GenOptTest CLStressWithLayers
CLStressWithLayersCRI NioLoadTest
– extended.system = LangLoadTest LockingLoadTest UtilLoadTest HCRLateAttachWorkload
JdiTest HCRLateAttachWorkload ConcurrentLoadTest DirectByteBufferLoadTest
system
sanity.system
ClassLoadingTest
net/adoptopenjdk/test/classloading/ClassMapHog.java

12. • group=external
• Targets (4 + 9) :
– sanity.external = openliberty-mp-tck payara-mp-tck
thorntail-mp-tck tomee-mp-tck
– extended.external = derby elasticsearch jenkins kafka
lucene-solr scala tomcat wildfly wycheproof
external
sanity.external
openliberty-mp-tck

13. • group=perf
• Targets (4 + 25) :
– sanity.perf = dacapo-eclipse dacapo-h2 dacapo-lusearch-fix idlemicrobenchmark
– extended.perf = OpenLibertyStartupDT7 akka-uct als chi-square db-shootout
dec-tree finagle-chirper finagle-http fj-kmeans future-genetic gauss-mix log-
regression mnemonics movie-lens naïve-bayes par-mnemonics philosophers
scala-kmeans dacapo-avrora dacapo-fop dacapo-jython dacapo-luindex dacapo-
pmd dacapo-sunflow dacapo-tomcat dacapo-xalan
perf
sanity.perf
dacapo-eclipse

14. • group=functional
• Targets (137 + 146 + 27) :
– sanity.functional = 137 targets
– extended.functional = 146 targets
– special.functional = 27 targets (MBCS_Tests_regex_ja_JP,
MBCS_Tests_regex_ko_KR, MBCS_Tests_urlclassloader_ja_JP, etc)
functional
special.functional
MBCS_Tests_regex_ja_JP

15. Summary
• AQA v1.0 is MVP to ensure mechanisms work
• Next steps:
– expanded set of tests
– additional metrics to support choices
– greater trss integration (not restricted to a single
Jenkins instance)
– website & releases repo updates