Testing Assembly 2022

1. Anomaly Detection
for Stability Testing
www.profilence.com
Martín Varela - Testing Assembly 2022

2. Profilence
platform
See the status of your
SW and solve the issues.
PSI
Telemetry client
running directly in
Android based
devices.
TAU/ZETA
Full scale test automation
solution for testing
embedded systems for
long periods of time.
POWERFUL WEB DASHBOARD
• Project management can follow progress and
trends from version to another
• Developers can solve issues by drilling down from
high level alerts all the way to low level data
PROFILENCE
CLOUD PLATFORM
or self-hosted by customers
• Receives data over secure connections from the
Profilence data providers.
• Analyses all data to automatically alert you on any
stability or non-functional issues
CENTER FOR PRODUCT DEVELOPMENT

3. Stability Analysis
▪ Evaluation of SW stability from end user perspective
○ Tests are run continuously for hours, days or even weeks
○ Long term testing shows what kind of problems end users would see over time
▪ Repeating an analysis is always an expensive operation
○ It’s also difficult to predict what data will be needed to solve any particular bug
○ Therefore we store full history of testing – not just test results but the data needed to
solve the issues
○ Our analytics also automatically analyse data for potential hard to find bugs – such
as memory leaks
▪ We aim at shortening the development / test feedback loop, and enable developers to
pinpoint and fix issues as quickly as possible

4. Logs
Full or selected logcat and kernel log
streams to debug issues
Types of data collected
Events
All crash and reset data with stack
traces, analysed to find unique issues
System profiler
Time series of system and application
memory usage to detect memory leaks
Performance
Long UI freezes and application
launch times
CPU load and stack sampling
available per use case
Energy
Energy consumption distribution
per use case, per each iteration
Test results
Test results with detailed steps,
screenshots and performance data
Video
Video of the whole test run

5. Logs
Full or selected logcat and kernel log
streams to debug issues
Types of data collected
Events
All crash and reset data with stack
traces, analysed to find unique issues
Today we’ll focus on these

6. Why Anomaly Detection?
▪ We want to shorten the development / test feedback loop, and enable
developers to pinpoint and fix issues as quickly as possible
○ Detecting anomalies in log files can help pinpoint what went wrong.
○ But
■ Logs are huge
■ Problems can happen well before the actual crash
▪ By finding likely problems in the logs, developers can identify the root cause
of problems more easily

7. How?
A crash or other stability event happens
x
We fetch the related logs
We train the N-gram
model, and check for
anomalies
We identify the
anomalous log lines

8. Limitations and Next Steps
▪ Still at the prototype stage
○ Further tuning and validation for the anomaly detection itself is necessary
○ Actual usage with developers also needed for validation
▪ UI/UX work is still needed
○ Slow and expensive, ideally would be on-demand
■ but, slow!
○ Could be done as stability issues are detected, but might be wasteful if
the results are not used
▪ Will be implemented as a standalone service that interacts asynchronously
with our analytics backend
▪ UI prototype

9. Thank you!
Contact Us
Martín Varela
martin.varela@profilence.com
+358 40 023 6420
www.profilence.com