The document discusses progressive refinement in quality assurance (QA), from initial broad testing to more targeted testing. It provides examples of assessing existing QA capabilities, prioritizing what to test, designing a QA approach, prototyping tests, and refining tests. Specifically, it summarizes the AdoptOpenJDK Quality Assurance (AQA) project, which aims to test OpenJDK builds across platforms and versions using an automated three-layer testing framework that is continuously refined.

1. From
Sledgehammer
to Fine Brush for
QA
@ShelleyMLambert
shelley.lambert@adoptopenjdk.net
DevNexus February 2021

2. Progressive Refinement
Start with brute force
End with finesse
Both needed in QA
• Sledgehammer – ruthless, insensitive, unnecessary force
• Fine brush – detail work

3. The Steps
ASSESS WHERE
YOU ARE
DETERMINE
WHAT
MATTERS
DESIGN PROTOTYPE
APPROACH
REFINE

4. Assess Where You Are
• What resources you have
• Team/people
• Existing tests
• Dove-tailing into existing solution? Machines / CI
• Measure
• Code coverage (sledgehammer blow)
• Functional coverage (brush stroke)
• Other metrics (PR list, change-based testing)
• Issues list
• What your users tell you

5. Determine What Matters
• Requirements
• Stakeholders
• Prioritize
• functionality, security, perf, scalability
• Risk map, meh? (related to Issues list)

6. Design
• Design your QA approach like it is a software solution (because it is)
• Understand where your tool choices will constrain you
• Select tools for prototype, but design to avoid lock-in… (portability)
• Online tutorials focus on ‘test with IDE xxxx’ or ‘use framework yyyy’

7. Prototype Approach
• Bare minimum needed to ‘plug in’ testing

8. Refine
• Keep the good, toss the bad
• Friction is bad
• Testing evolves alongside your code base
• Always ask, how can we improve?

9. Specific Example: AQA
• AdoptOpenJDK Quality Assurance (AQA)
• Community driven, open-source project
• Broadest set of platforms of any OpenJDK
distributor
• Testing a wide criteria representing actual business
requirements to identify binaries ready for
production usage
As of January 24, 2021

10. AQA: Assess
Before AQA AQA Roadmap
Functional correctness Security
OpenJDK regression (open) Passes known vulnerability tests
Functional correctness
Compliance OpenJDK regression
Oracle JCK (closed) Eclipse functional
Usability
Builder-specific testing (unknown) Application & framework tests
Performance
Published metrics
Achieves minimum throughput scores
Scalability & durability
Load & stress testing

11. AQA: Determine What Matters
• “Make quality certain to happen”
• AQA Manifesto
• Open and transparent and community driven
• Diverse and robust enough to cover enterprise requirements
• Evolving alongside JDK
• Portable (run on laptop to support development, run in CI servers)
• Tagged/tracked and published to rerun tests and reproduce results

12. AQA: Design

13. AQA: Scope
OpenJ9 Hotspot Dragonwell Corretto
8 11 15 16 17 +
openjdk functional perf system
external
(3rd party app)
Upstream
Branch X
JDK Implementations
Platforms
JDK Versions
Test Categories
osx aix win xlinux plinux
6 versions
aarch
58 impl_platform
250000 tests
87,000,000 Tests Impl_platform x testLevels x testGroups x versions
58 x 2 x 3 x 6 x 10M = 20G+ test output per nightly build
Plus PR builds,
promotion builds and
personal builds &
Docker image testing
Built in-house Built upstream
scala
scala
scala
scala
scala
camel
scala
scala
scala
derby
scala
scala
scala
scala
elasticsearch
scala
scala
scala
scala
openliberty
scala
scala
scala
scala
jenkins
scala
scala
scala
scala
kafka
scala
scala
scala
scala
quarkus

14. AQA: Design
AQA AUTOMATION: 3 LAYER CAKE
testkitgen (TKG)
Test Results Summary Service (TRSS)
CI System (Jenkins / Tekton / AzDO / Github Actions, etc.)
Responsibilities:
- Categorize logically, Generates test targets based on playlist (level/platform/version/impl specific)
- Execute tests via common command line / make target patterns
- Test addition via auto-discovered directories, Standardize exclusion
- Parameters: BUILD_LIST, EXTRA_OPTIONS, JVM_OPTIONS, TEST_FLAG, ITERATION
- Generate dynamic test playlists based on input (smart parallelization)
Responsibilities:
- Schedule regular builds
- Multiplex tests across multiple nodes of all platforms
- Basic GUI view of test results
- Basic forms of parallelization
Responsibilities:
- Monitor multiple CI servers
- Graphical, Aggregate summary, Deep history
- Search/sort/filter
- Pluggable parsers
- Basis for deeper analytics and deep learning services
Layer 1: Standalone
- Execution
- Exclusion
- Test target report
summary (TAP)
- Reproducibility
Layer 2: Requires L1
- More nodes
- Scheduling
- GUI
- enhanced reporting for
tests that support Junit
output
Layer 3: Requires L1 & L2
- Database queries
- Basis for extras, search /
sort /filter /analyze across
DB entries
- Monitor anything

15. AQA: Prototype Approach
1st layer of the cake

16. AQA: Refine
• Past 3 years have been a shocking amount of refinement
• Refer to tests repo history (openjdk-tests, TKG, openjdk-test-tools,etc)
• Building blocks for further refinement

17. AQA 2021 Plans – Keep on Refinin’
• Performance testing enhancements
• EXTERNAL Test group expansion/clean-up
• Smarter/Better/Faster
• Change-based testing
• Automatically test and re-include excluded tests
• TRSS enhancements
• Collaboration with research

18. The Long List of Brush Strokes
In no particular order and definitely not the full list:
• Bug prediction
• Combinatorial Test Design / Modelling
• Improve your Triage tools
• Change-based testing (combine codecov with PR list)
• Verify your Tests (defect injection)
• Apply deep learning to help with certain QA tasks

19. Functional vs Code Coverage
static Integer mathyJunkFunction(int num1, Integer num2, String num3) {
return new Integer(num1) / (num2 + new Integer(num3));
}
Integer result = mathyJunkFunction(1, new Integer(1), “1”);
TESTED, 100% code coverage
hazaa!
• Defect 1: What if num3 has a value of null (throws NumberFormatException)?
• Defect 2: What if num3 is “1” and num2 is -1, or “0” and 0, or any other combination
that sum to 0 (throws ArithmeticException)?
• Defect 3: What if num2 is null (throws NullPointerException)?

20. Bug Prediction
• SCORES PER FILE BASED ON ‘RECENT’ CHANGES DUE TO DEFECTS (GITHUB PRS/ISSUES),
PREDICT BASED ON CHANGE & DEFECT HISTORY, OTHER FEATURES?

21. Extra Info
• adoptopenjdk.net
• trss.adoptopenjdk.net
• Bi-weekly AQAvit meetups (open, transparent evolution)
• https://github.com/AdoptOpenJDK/openjdk-tests/issues?q=+label%3A%22AQAvit+Meeting%22+

22. Demo TRSS