Talking about Eclipse AQAvit and some of the goals and activities happening in that project...

2. AQAvit: Vitality through Testing
May 2021
Shelley Lambert
Sophia Guo

3. AdoptOpenJDK -> Eclipse Adoptium
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• 2nd largest distributor of OpenJDK binaries (270,000,000+ downloads)
• Differentiated by:
• diverse, open community
• range of platforms
• quality assurance program
AQAvit

4. Eclipse Adoptium Project Structure
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• Adoptium (WG & Project) projects.eclipse.org/projects/adoptium
• Temurin (Build & Distribute) projects.eclipse.org/projects/adoptium.temurin
• AQAvit (Test for Quality) projects.eclipse.org/projects/adoptium.aqavit
• Incubator (Innovate) projects.eclipse.org/projects/adoptium.incubator
• Temurin-compliance (Test for Compliance)

5. Adoptium Quality Assurance (AQA)
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• Openly testing a wide criteria of enterprise business requirements to
identify binaries ready for production usage
Before AQA
Functional correctness
OpenJDK regression (open)
Builder-specific testing (unknown)
AQA Roadmap
Security
Passes known vulnerability tests
Functional correctness
OpenJDK regression (open)
Eclipse functional (open)
Application & framework tests (open)
Performance
Published metrics (open)
Achieves minimum throughput scores (open)
Scalability & durability
Load & stress testing (open)
?

6. AQA Overview
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• Set of tests selected from key open-source stacks
• Ensure enterprise level quality requirements
• Community-driven process
• Designed for developer devops
• Reproducible test environment
• Graduated levels of testing
• Release dashboard with live results

7. About the Tests
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• groups: functional, openjdk, system, external, perf
• levels: sanity, extended, special
• impls: hotspot, openj9
• versions: 8, 11, 16, 17…
• types: regular, native
https://github.com/adoptium/aqa-tests
functional openjdk perf
external
system
testNG, junit
cmdlinetester
STF
junit, mvn &
others
Assorted
benchmarks
Jtreg, testNG
$
(
J
A
V
A
_
B
I
N
)
/
j
a
v
a
-
X
m
x
5
1
2
m
-
j
a
r
j
t
r
e
g
.
j
a
r
-
v
m
o
p
t
i
o
n
s
:
"
-
X
m
x
5
1
2
m
"
-
w
w
o
r
k
-
r
r
e
p
o
r
t
-
j
d
k
:
$
(
J
D
K
_
H
O
M
E
)
-
e
x
c
l
u
d
e
:
P
r
o
b
l
e
m
L
i
s
t
.
t
x
t
j
d
k
_
m
a
t
h
$
(
J
A
V
A
_
B
I
N
)
/
j
a
v
a
-
c
p
G
e
n
e
r
a
l
T
e
s
t
.
j
a
r
o
r
g
.
t
e
s
t
n
g
.
T
e
s
t
N
G
-
d
$
(
R
E
P
O
R
T
D
I
R
)
t
e
s
t
n
g
.
x
m
l
-
t
e
s
t
n
a
m
e
s
f
l
o
a
t
S
a
n
i
t
y
T
e
s
t
s
-
g
r
o
u
p
s
s
a
n
i
t
y
p
e
r
l
s
t
f
.
p
l
-
t
e
s
t
-
r
o
o
t
=
o
p
e
n
j
d
k
-
s
y
s
t
e
m
t
e
s
t
_
r
o
o
t
-
s
y
s
t
e
m
t
e
s
t
-
p
r
e
r
e
q
s
=
s
y
s
t
e
m
t
e
s
t
_
p
r
e
r
e
q
s
-
j
a
v
a
-
a
r
g
s
=
-
X
j
i
t
-
r
e
s
u
l
t
s
-
r
o
o
t
=
r
e
p
o
r
t
d
i
r
-
t
e
s
t
=
C
o
n
c
u
r
r
e
n
t
L
o
a
d
T
e
s
t
s
b
t
"
p
a
r
t
e
s
t
$
(
T
E
S
T
_
S
U
I
T
E
)
”
$
{
A
N
T
_
H
O
M
E
}
/
b
i
n
/
a
n
t
-
D
i
v
y
_
i
n
s
t
a
l
l
_
p
a
t
h
=
$
{
A
N
T
_
H
O
M
E
}
/
l
i
b
-
l
i
b
$
{
A
N
T
_
H
O
M
E
}
/
l
i
b
-
f
$
{
L
U
C
E
N
E
_
S
O
L
R
_
H
O
M
E
}
/
l
u
c
e
n
e
-
s
o
l
r
/
b
u
i
l
d
.
x
m
l
-
D
u
s
e
r
.
h
o
m
e
=
$
{
L
U
C
E
N
E
_
S
O
L
R
_
H
O
M
E
}
-
D
c
o
m
m
o
n
.
d
i
r
=
$
{
L
U
C
E
N
E
_
S
O
L
R
_
H
O
M
E
}
/
l
u
c
e
n
e
-
s
o
l
r
/
l
u
c
e
n
e
t
e
s
t
$
(
J
A
V
A
_
B
I
N
)
/
j
a
v
a
-
D
B
u
m
b
l
e
B
e
n
c
h
.
l
i
s
t
o
p
t
i
o
n
s
-
j
a
r
B
u
m
b
l
e
B
e
n
c
h
.
j
a
r
[
B
e
n
c
h
m
a
r
k
n
a
m
e
]
functional openjdk perf
external
system
testNG,
cmdlinetester
STF junit & others Assorted
benchmarks
Jtreg, testNG
testkitgen
m
a
k
e
_
f
u
n
c
t
i
o
n
a
l
m
a
k
e
_
o
p
e
n
j
d
k
m
a
k
e
_
s
y
s
t
e
m
m
a
k
e
_
e
x
t
e
r
n
a
l
m
a
k
e
_
p
e
r
f
Slice & Dice

8. Automation and the “3 layer cake”
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
TestKitGen (TKG)
Test Results Summary Service (TRSS)
CI System (Jenkins / Tekton / AzDO / Github Actions, etc.)
Responsibilities:
- Categorize logically, Generates test targets based on playlist (level/platform/version/impl specific)
- Execute tests via common command line / make target patterns
- Test addition via auto-discovered directories, Standardize exclusion
- Parameters: BUILD_LIST, EXTRA_OPTIONS, JVM_OPTIONS, TEST_FLAG, ITERATION
- Generate dynamic test playlists based on input (smart parallelization)
Responsibilities:
- Schedule regular builds
- Multiplex tests across multiple nodes of all platforms
- Basic GUI view of test results
- Basic forms of parallelization
Responsibilities:
- Monitor multiple CI servers
- Graphical, Aggregate summary, Deep history
- Search/sort/filter
- Pluggable parsers
- Basis for deeper analytics and deep learning services
Layer 1: Standalone
- Standardized test execution, exclusion, reporting (TAP)
- Overlay additional tests
- Single code base for multi-version
- Reproducibility
- Parallel modes
Layer 2: Requires L1
- Multiplex across more nodes/platforms
- Scheduling
- Basic GUI
- Enhanced granularity for test results in JUnit format
Layer 3: Requires L1 & L2
- Database queries
- Search/sort/filter/analyze across DB entries
- Monitor anything
- Basis for extras

9. Layer 1: TKG
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• Demo - Basic local execution • Fetch test definitions & materials
• git clone aqa-tests
• get.sh (to fetch materials)
• Tell TKG what to compile & test
• export BUILD_LIST=openjdk,system
• export TEST_JDK_HOME=$SUT
• Compile test material
• make compile
• Run tests
• make _test_target

10. Layer 2: CI with Jenkins
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• Grinder
• Grinder PARALLEL=Dynamic

11. Layer 2: CI with Github Actions
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• Demo
• setup-java
• install-jdk
• build-jdk
• build-openj9
• run-aqa

12. GitHub Action Background
@User-Handle | Company
• Automate your workflow
• Yaml file under .github/worksflows
• Run a workflow on any GitHub event
• Push, issues, comment, label, release, etc
• GitHub-hosted runners (win, ubuntu, mac)

13. build-openj9
@User-Handle | Company
• https://github.com/eclipse-openj9/build-openj9
• Supported: ubuntu, mac, win
• Related repos:
• openj9, openj9-omr, openj9-openjdk-jdk${version}
• usePersonalRepo
• Auto use developers’ branch

14. run-aqa
@User-Handle | Company
• https://github.com/adoptium/run-aqa
• Supported: ubuntu, win, mac
• Test Category:
• openjdk, functional, external, system, etc
• Test JDK:
• JDK built by GitHub action: build-jdk, build-openj9
• Installed JDK: Adoptium EA, GA, zulu Openjdk, https

15. Layer 3: TRSS
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• Demo - API, Grid view
• Demo - Triage helpers, “the Big Four”
• Across platforms
• Deep history
• Possible issues
• Raise an issue
• Release Triage report
• Performance views

16. 2021 AQAvit Program Plan
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• AQAvit quality mark & Adoptium marketplace
• Enhancements
• External tests (Tweetest)
• Performance benchmarks
• ML on results (Deep AQAtik)

17. Connect!
•Twitter: @adoptium
•Github: https://github.com/adoptium
•Slack: https://adoptium.net/slack.html

18. Quality versus Conformance
@ShelleyMLambert | Red Hat
@Sophia_Wfguo | Red Hat
• https://github.com/AdoptOpenJDK/TSC/issues/222#issuecomment-818020349
TCK AQA
Purpose
Compliance
(conforms to the Java
specification)
Test for Quality
(functional correctness,
performance, scalability,
security, robustness)
Availability Closed under OCTLA
(license agreement with Oracle)
Open-source /
available to all