This document discusses using microservices for testing and provides examples of potential test-related microservices. It describes decomposing test activities like planning, implementation, automation, execution, triage, and reporting into discrete microservices. Examples of microservices are provided for various test activities like the Core Analytics Service, Test Generation Service, BenchEngine, Results Analytics Service, and Results Comparison Service. The document argues that a microservices approach can help share functionality across products and simplify testing processes.

1. Cloud-based Test
Microservices
Shelley Lambert
IBM Runtime Technologies Test Lead

2. Some Legal Statements
• THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
• WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED.
• ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
• ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
• IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
• IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT
OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
• NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
– CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS
OR THEIR SUPPLIERS AND/OR LICENSORS

3. Development Transformation
• Drawn from work in IBM Runtimes and on Eclipse
OMR, Eclipse OpenJ9, and AdoptOpenJDK projects
• Hundreds of thousands of tests of differing
formats, on thousands of test machines
representing 18+ platform variants, multiple
versions of Java (v5 – 10)
• Goal to simplify, redesign and decouple from
legacy/proprietary tools and processes and create
open, agile, flexible tests and test solutions to
empower developers
AdoptOpenJDK

4. Questions to Address
• What aspects of your testing can be shared across
multiple products?
• How to identify, abstract, and convert functionality
used by many test systems into shared services
• Why bother with microservices? In a cloud?
• Examples of test microservices at various points in
the flow of activities related to Java testing
The small print… Not covering…
• Testing cloud-based microservices (is the title of this talk confusing?)
• Where to deploy your services? …Not trying to sell you a cloud solution

5. What is a microservice?
• A succinct API to accomplish a particular task
(happiness knows no bounds, but microservices do…)
• A set of microservices assembled together can
create a whole system (replacing a monolithic
approach)
• While often used in concert, each service can
be used independently
• Benefits: Modular, Simpler, Portable,
Customizable Apps, Easier Development &
Deployment, Co-operative

6. Microservices Architecture
ResultAnalytics
Data
Services
UI Layer
Cores
raw refined
custom dashboard
other clients
TestGeneration BenchEngine
CoreAnalytics
TestSelection
BugPrediction
InputOptions
ResultSummary
ResultCompare
github
repos
Jenkins
servers

7. Activities (and Questions) Related to Test
Plan Implement Automate Execute
Triage Exclude
Report
What? How?
How
often?
How
easy?
How
few?
How
fast?
What failed?
Why? What
next?
Decompose into a set of services by test activity, services to help answer the questi

8. Planning Phase
What tests do you need?
How are people using your product?
What are their user stories?
What additional info do you have (code
coverage, code churn information)?
Plan Implement Automate Execute
Triage Exclude
Report
What?

9. ‘Planning Phase’ Microservice Examples
• Core Analytics Service – inspect java core
files from service reports, gather
information on usage, particularly
patterns of input parameters
• Bug Prediction Service – based on
research* an inexpensive algorithm to find
hot spots (most changed files) in code
base, applied to any given repo
*BugCache for Inspections: Hit or Miss?

10. Core Analytics Service
Demo

11. Bug Prediction Service
Demo

12. Implementation Phase
How best to design and write your tests?
What tools/frameworks to use?
How to minimize number of tests while
maximizing functional coverage?
Plan Automate Execute
Triage Exclude
Report
Implement
How?

13. ‘Implementation Phase’ Microservice
Example
• Test Generation Service – assist developers in
the onerous task of writing thorough
functional tests, standardized test naming and
structure, CTD for minimal tests with good
functional coverage
– Upload header files (function/method signatures)
– Inspect input types (primitive or complex)
– Generate combinations of inputs (pairwise/CTD)
– Generate partial test code… (with standardized
naming, parameterized inputs)

14. Test Generation Service
Link to header file in github
Command line options to us
Complex input parameters
-> equivalence classes

15. Automation Phase
• How do I add my tests into a CI workflow?
• How do I replace old practices with new
improved ones?
Plan Execute
Triage Exclude
Report
Implement Automate
How often?
How easy?

16. ‘Automation Phase’ Microservice
Example
• BenchEngine Service – make it easy for
anyone to run performance benchmarks
– generate scripts to run benchmarks from
definition files
– allow for variations and specific platform
settings
– send scripts to Jenkins nodes to run
automatically run benchmarks

17. BenchEngine
Demo

18. BenchEngine
Developer:
• fires up browser
• connects to BenchEngine service
• select inputs
Jenkins
input selections:
• benchmark definition file
• action type
• which ci server
generateScript:
• local run
• insert into
automated
builds
submitJob
BenchEngine
Service
github
repo
addToBuild
new feature:
• “add to build” button
• Insert into automated
build(s)
Jenkins

19. Execution Phase
• What is the minimal set of tests I need to
run, given a particular change?
• What input options do I run with?
• What is the execution time? Has it
changed over time?
Plan
Triage Exclude
Report
Implement Automate Execute
How few?
How fast?

20. ‘Execution Phase’ Microservice Example
• Test Selection Service – change-based
testing, uses code coverage information
to generate a smaller/smarter set of tests
to run on a pull request
• Input Options Service – grabs input
options defined in tests at start of a build,
names and stores them (unique sentences
of options), for sharing with other builds/
tests

21. TestSelection
Developer:
• fires up browser
• connects to Test Selection service
• select inputs
Jenkins
input selections:
• pull request in github
• platform(s)
• Code coverage info
submitJob
Test Selection
Service

22. Input Options Service
• (nearly) headless service to feed information to builds
• not all services have (or need) a pretty GUI
InputOptions
Jenkins mongodb

23. Triage Phase
• Has this failure happened before? Is this failure
intermittent? If so, are there recognizable patterns?
• Is there other information besides test output that I
can use to help find patterns?
• Do I need to exclude this test? Can I re-include it
automatically when the issue is fixed?
Plan Execute Report
Implement Automate
Triage Exclude
What failed?
Why?

24. ‘Triage Phase’ Microservice Example
• WIP: Results Analytics Service - automate
the “googling of error messages”, look for
patterns
– search across builds at one server?
– search across builds at multiple servers?
– search the internet?
– Use Watson pattern matching / analytics engine...
– If you didn’t want all of the above, then Jenkins
plugin?

25. Results Analytics
Service
Developer:
• Runs test build registered with
service
• Failures added to database
• Service queries test history for
similar failures
Jenkins
Inputs:
• ci server / build id
• DB
• query type (test
history,
stackoverflow)
Results Analytics
Service
Jenkins
Returns:
• Matches
(with
scores)
DB
Why not use Jenkins plug in?
• Multiple Jenkins servers / DBs
• Tie into other cloud services
(analytics engine…)
• Instrument test output with
concise machine-readable info
for triage assistance (info
from core files)

26. Reporting Phase
• Can I summarize a large/diverse set of
tests?
• Where can I see the test report?
• Can I compare test results easily? From
many data sources?
Plan Execute
Triage Exclude
Implement Automate Report
What
next?

27. ‘Report Phase’ Microservice Example
• Results Summary Service – summarize the
results of multiple builds running on multiple
servers, overlay customizable dashboard
• Results Comparison Service – given more than
one result of same test, compare them.
Examples of results to compare:
– FV tests from Build 1 compared with those of Build 12
– Scala (or other 3rd party app) community tests
running on OpenJDK vs OpenJDK with OpenJ9
(Java8 / Java9)
– Last 4 BenchmarkXYZ runs against OpenJ9 (trend)
– Last several months of GC perf stats from OMR

28. Results Summary Service
• Model the common elements (status, execution time,
failure age, etc) for uniform representation
• Leverage Jenkins API
• Progression of enhancements
– Raw data
– Simple diff
– Graphical representations
– Overlay customizable dashboard, simple elements, continuous
feedback

29. Results Summary
Service
Developer:
• Fires up browser
• Selects / deselects builds to
display in customizable
dashboard
• Takes action on failing builds
by clicking through to more
details from summary view
Jenkins
Inputs:
• ci servers / build ids
• DB
Results Summary
Service
Jenkins DB
Jenkins

30. Results Comparison
Service
Demo
• Gather perf metrics from
Eclipse OMR builds
• Store historical data in
mongodb
• GC perf dashboard allows
many different views
onto the data

31. Review: Examples of microservices
Plan Implement Automate Execute
Triage Exclude
Report
Core analytics service
Bug prediction service
Test Generation Service Test Selection Service
Input-Options Service
Result Analytics Service Result Summary Service
Result Comparison Service
BenchEngine

32. Microservices Examples Review
ResultAnalytics
Data
Services
UI Layer
Cores
raw refined
custom dashboard
other clients
TestGeneration BenchEngine
CoreAnalytics
TestSelection
BugPrediction
InputOptions
ResultSummary
ResultCompare
github
repos
Jenkins
servers
Plan Execute Triage Report
Implement Automate

33. Questions Addressed
• What aspects of your testing can be shared across
multiple products? Decompose by common activities…
• How to identify, abstract, and convert functionality
used by many test systems into shared services?
Common patterns (apps take inputs, tests produce
results)...
• Why bother with microservices? In a cloud? Modular
(super-fast dev), portable, assemble for a ‘system’ or
singular use, leverage other services…
• Gotchas... Network reliance, fallbacks needed…

34. Get Involved /
Innovate with us at:
• Eclipse OMR
• Eclipse OpenJ9
• AdoptOpenJDK
AdoptOpenJDK
Details:
• https://github.com/eclipse/omr
• https://github.com/eclipse/openj9
• https://github.com/AdoptOpenJDK
• @ShelleyMLambert
Stay in touch