This document discusses strategies for dealing with increasing amounts of verification data from testing Java implementations. It describes the scope of testing across multiple Jenkins servers for projects like AdoptOpenJDK. A Test Result Summary Service (TRSS) was created to monitor and filter results. Initial deep learning experiments were able to classify test outputs with 98.8% accuracy. Plans going forward include change-based testing, bug prediction services, enhancing TRSS with analytics, and using AI techniques like fuzz testing and test generation.

1. SHELLEY LAMBERT
LAN XIA
IBM RUNTIME TECHNOLOGIES
NOV 2019
@SHELLEYMLAMBERT
@BONJOURLAN
DEALING WITH
VERIFICATION DATA OVERLOAD
Dealing with Verification Data Overload

2. • THE SCOPE
• TEST FRAMEWORK (TESTKITGEN)
• JENKINS BUILDS
• TEST RESULT SUMMARY SERVICE (TRSS)
• DATA REFINERY EXPERIMENTS
• PLANS FORWARD
AGENDA
Dealing with Verification Data Overload

3. THE SCOPE
AdoptOpenJDK
• ENSURING FREE AND VERIFIED JAVA™ FOR THE COMMUNITY
• PROJECTS: ECLIPSE OMR, ECLIPSE OPENJ9, ADOPTOPENJDK
• 6+ JENKINS SERVERS
Dealing with Verification Data Overload

4. Dealing with Verification Data Overload

5. DEGREES OF FREEDOM
OpenJ9 Hotspot SAP Corretto
8 9 10 11 12 +
openjdk functional perf system external
RI
13
JDK Implementations
Platforms
JDK Versions
Test Categories
osx
osx aix win xlinux plinux
6 versions
aarch
58 impl_platform
250000 tests
87,000,000 Tests Impl_platform x testLevels x testGroups x versions
58 x 2 x 3 x 6 x 10M = 20G+ test output per nightly build
Plus PR builds,
promotion builds and
personal builds
Dealing with Verification Data Overload

6. GATHER GREAT TESTS
6
functional openjdk perfexternalsystem
Dealing with Verification Data Overload

7. GATHER GREAT TESTS
7
functional openjdk perfexternalsystem
testNG,
cmdlinetester
STF junit & others
Assorted
benchmarks
Jtreg, testNG
Dealing with Verification Data Overload

8. ADOPTOPENJDK QUALITY ASSURANCE
(AQA)
8Dealing with Verification Data Overload
• “Make quality certain to happen”
• Testing a wide criteria representing actual business requirements to identify
binaries ready for production usage
Today Roadmap
Functional correctness Security
OpenJDK regression (open) Passes known vulnerability tests
Oracle JCK (closed) Functional correctness
OpenJDK regression
Builder-specific testing (unknown) Eclipse functional
Application & framework tests
Performance
Published metrics
Achieves minimum throughput scores
Scalability & durability
Load & stress testing

9. AQA MANIFESTO
• open & transparent
• diverse & robust set of test suites
• evolution alongside implementations
• continual investment
• process to modify
• codecov & other metrics
• comparative analysis
• portable
• tag & publish
9Dealing with Verification Data Overload

10. INNOVATE AND COLLABORATE
• Reactive systems
• Latitude
• Flexible
• Common
• Standardized
• Simple
10Dealing with Verification Data Overload

11. GRANULARITY
• Specific Testcases/groups
• Different levels
• Different versions
• Different implementations
• Different iterations
• Different features
• With/Without native test image
• …
11
sanity/extended/special
8/11/13/14…
openj9/hotspot/ibm/corretto/sap
1/2/n…
AOT/JITAAS
functional/system/openjdk/perf/external
Dealing with Verification Data Overload

12. CONSOLIDATE AND CURATE
12
functional openjdk perfexternalsystem
TestKitGen
testNG,
cmdlinetester
STF junit & others
Assorted
benchmarks
Jtreg, testNG
Dealing with Verification Data Overload

13. TEST FRAMEWORK (TESTKITGEN)
13Dealing with Verification Data Overload

14. GROUPING & GRANULARITY
• group=openjdk
• levels=sanity.openjdk, extended.openjdk, special.openjdk
• targets=tests in playlist file
• jdk_awt, jdk_math, jdk_lang, etc.
• jdk_custom=CUSTOM_TARGET env var
• set to individual directories or classes
14
openjdk
sanity.openjdk
jdk_math
Java/math/BigDecimal/NegateTests.java
Dealing with Verification Data Overload

15. ADOPTOPENJDK CI PIPELINE
Run in parallel
Build Deploy
openjdk
functional
system
perf
external
Test
sanity.system
lambdaLoadTest
extended.system
special.system
mathLoadTest
…
daaLoadTest
Dealing with Verification Data Overload

16. JENKINS SCRIPTS FOR TESTING
16
• Repo:
https://github.com/AdoptOpenJDK/openjdk-tests
• One script (JenkinsfileBase) for all
test builds:
• Nightly/release
• Pull Request
• Promotion
• Personal/Grinder
Dealing with Verification Data Overload

17. SURVEY OF TESTS: CI.ADOPTOPENJDK.NET
17
Categorize test builds
based on JDK Version,
JDK Impl, test category
and platform
Dealing with Verification Data Overload

18. TAP & JUNIT PLUGIN
18
Standardize output
Dealing with Verification Data Overload

19. ARCHIVE DATA
• Archive test data from failed tests onto Jenkins master or Artifactory:
• Test logs/output files
• Diagnostic files (core/trace/javacore files)
• TAP file, JUnit *.jtr, *.xml files
• Test repo SHA
19
Minimize stored
artifacts
Dealing with Verification Data Overload

20. ADOPTOPENJDK TESTING
vm
docker
physical
machine
Jenkins
Server
artifactory How to monitor
the result???
Dealing with Verification Data Overload

21. TEST RESULT SUMMARY SERVICE (TRSS)
• Monitors multiple Jenkins servers
• Personalized Dashboard
• Provide filtering, sorting, comparing and searching feature
• Provide history for triaging and performance trends
• Available at https://trss.adoptopenjdk.net/
• Git repo: https://github.com/AdoptOpenJDK/openjdk-test-tools
21Dealing with Verification Data Overload

22. TRSS OVERVIEW
22
frontend
backend
backend
backend
TRSS Server
Client
Multiple
Jenkins
Servers
MongoDB
Dealing with Verification Data Overload

23. TRSS: PERSONALIZED DASHBOARD
23
Personalize views, only
what you need
Dealing with Verification Data Overload

24. TRSS: PERSONALIZED DASHBOARD
24Dealing with Verification Data Overload
Grid view for test
builds summary

25. TRSS: MONITOR JENKINS PIPELINE BUILDS
25
Aggregate results from
top-level pipelines
Dealing with Verification Data Overload

26. TRSS: TEST BUILDS RESULT
26
Test Result Summary
Dealing with Verification Data Overload

27. TRSS: TESTS RESULT
27
Filter and sort on test name, duration, and
test results
Dealing with Verification Data Overload

28. TRSS: TEST RESULT ACROSS ALL BUILDS
28
Compare test results for all JDK
impls, JDK versions and platforms
Dealing with Verification Data Overload

29. TRSS: SEARCH TEST
29Dealing with Verification Data Overload
Search test output among
different test builds

30. TRSS: TEST COMPARE
30
Diff test results among two
different builds
Dealing with Verification Data Overload

31. TRSS: PERF DASHBOARD
31
Visualize data
graphically
Dealing with Verification Data Overload

32. LET US COUNT THE WAYS
• categorize
• standardize
• minimize
• personalize
• aggregate
• summarize
• filter
• sort
• compare
• search
• diff
• visualize
• model
Dealing with Verification Data Overload

33. WHAT IS DEEP LEARNING?
Deep learning is a subset of ML algorithms distinguished by:
• Loosely based on structure and function of the brain, use
artificial neural networks (ANN)
• Multiple layers of processing units, “neurons”, output of a
layer is input to another layer
• Modes of learning, supervised (regression, classification)
or unsupervised (pattern analysis)
33Dealing with Verification Data Overload

34. INITIAL DL EXPERIMENTS
• Preprocess testOutput based on own listed feature key words index.
34
Padding preprocessed testOutput for Deep Learning
model:
“[4, 7, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…]”
Label: “1”
Original testOutput:
“Running test TestIBMJlmLocal_0: ERROR code 3, FAILED test 1,
running second test, exception ValueError, exit code 1”
Label: “FAILED”
Preprocessed testOutput:
“[4,
Label:
7, 10]”
“1”
Dealing with Verification Data Overload

35. INITIAL DL EXPERIMENTS
- Training Accuracy and Validation
Accuracy, our evaluation result is
98.897% with 7800 training data
(3900 failed and 3900 passed) and
7800 test data
- Training Loss and Validation Loss
35Dealing with Verification Data Overload

36. MODEL BUILDING
36
JVM version
Variants used
Things we know
(input layer)
Failure expression
Platform
JVM Impl
Machine ‘age’
Failure age
PR list
Defect category
Things we want to know
(output layer)
Bug prediction scores
Best next action
Rate value of test
SHAs
Dealing with Verification Data Overload

37. Deep Learning for Fuzzing Java Compilers
38Dealing with Verification Data Overload

38. Deep Learning for Fuzzing Java Compilers
39Dealing with Verification Data Overload
• Integrated DeepSmith with TKG and TRSS
• Easily run thousands of DeepSmith tests in Jenkins with different
JDK versions / impls / JVM options
• Compare and monitor test outputs using TRSS

39. PLANS FORWARD
• Test smarter (as test volume increases)
• Change-based testing
• Bug prediction service
• Enhance TRSS with analytics services
• Build skills and continue model/deploy, observe & measure
• Innovate/Collaborate
• AI-driven fuzz testing (with Professor Hugh Leather)
• Test generation service (application of CTD)
• Leverage & deploy useful models in open projects
40Dealing with Verification Data Overload

40. @ShelleyMLambert
@bonjourlan
AdoptOpenJDK: https://adoptopenjdk.net/
Git repo:
https://github.com/AdoptOpenJDK/openjdk-tests
https://github.com/AdoptOpenJDK/openjdk-test-tools
https://github.com/AdoptOpenJDK/TKG
https://github.com/eclipse/openj9
THANK YOU!
Dealing with Verification Data Overload

41. QUESTIONS ?