SEAA/SEaTeD 2018, Prague (Czech Republic): Talk by Marcus Ciolkowski (Principal IT Consultant at QAware), Liliana Guzmán, Adam Trendowicz, Anna Maria Vollmer Abstract: Existing definitions and metrics of technical debt (TD) tend to focus on static properties of software artifacts, in particular on code measurement. Our experience from software renovation projects is that dynamic aspects --- runtime indicators of TD --- often play a major role. This talk summarizes a position paper for SEAA 2018, in which we present insights and solution ideas gained from numerous software renovation projects at QAware and from a series of interviews held as part of the ProDebt research project. We interviewed ten practitioners from two German software companies in order to understand current requirements and potential solutions to current problems regarding TD. Based on the interview results, we motivate the need for measuring dynamic indicators of TD from the practitioners' perspective, including current practical challenges. We found that the main challenges include a lack of production-ready measurement tools for runtime indicators, the definition of proper metrics and their thresholds, as well as the interpretation of these metrics in order to understand the actual debts and derive countermeasures. Measuring and interpreting dynamic indicators of TD is especially difficult to implement for companies because the related metrics are highly dependent on runtime context and thus difficult to generalize. We also sketch initial solution ideas by presenting examples of dynamic indicators for TD and outline directions for future work.

1. Marcus Ciolkowski (QAware), Liliana Guzmán, AdamTrendowicz, Anna Maria Vollmer
marcus.ciolkowski@qaware.de
Challenges in Assessing
Technical Debt based on
Dynamic Runtime Data
Prague, August 31 2018
SEAA 2018
Software Engineering and Technical Debt

2. Key Take-Aways
QAware 2
Research should focus more on dynamic indicators of technical debt
(instead of mining for static aspects)
There are many opportunities for measurement
Highly relevant to industry

3. QAware 3
QAware: We specialize in quality projects.
solve
problems
shape
the future
Consulting
Analysis & Diagnosis
Implementation
Renovate & Migrate
Consulting
Invent & Explore
Implementation
Development & Integration
Focus:
IT-Troubleshooting
Focus:
Cloud Native Computing
Big Data / Machine Learning
IoT
Software Engineering

4. Background 2: Research Project ProDebt
Proactive Management of Technical Debt
QAware 4
Context: ProDebt (management of technical debt)
assessing and managing technical debt
in the context of ASD
Structured interviews in two companies
10 developers/architects (including 2 CTOs and 3 chief architects)
In this presentation: Focus on dynamic aspects of technical debt
challenges
opportunities and examples for dynamic aspects

5. Measuring Quality Debt is not novel.
But: Existing Approaches ignore dynamic behavior.
5
Current software (quality) measurement is based on static metrics
plenty of tools exist (e.g. SonarQube)
rather simple to measure
much experience in measurement and presentation
Dynamic aspects / KPIs are underrepresented
test coverage is most common exception
dynamic indicators influence, e.g.
customer satisfaction
infrastructure and operation costs
tricky cases in maintenance hide often here (e.g., leaks and locks)

6. Gap: Measure and evaluate dynamic aspects
QAware 6
One example of dynamic aspects: Runtime structure
runtime structure: actual call relationships at runtime
runtime structure is important
to gain understanding of system at runtime
to identify performance problems
Challenge: often difficult to detect statically
abstraction and inheritance (many false positives)
code injection
reflection
soft links
EventMgr
Queue
add remove
Object event = queue.remove();
Method act = event.getMethod("act");
act.invoke(event);
Which class is called?
EventMgr
Queue

7. Challenges: Definition
QAware 7
Defining and interpreting appropriate metrics
relevant (dynamic) software aspects highly context-dependent
metrics (measurement opportunities) highly dependent on context
needed: support in defining metrics
High requirement on measurement:
production-ready tools needed
low runtime impact / non-invasive measurement
high degree of automation
long-term storage of metrics
integration of metrics from different sources

8. Challenges: Tools
QAware 8
Measurement implementation / instrumentation
lack of tools for automatic measurement of software behavior at runtime
Highly invasive measurement approaches, such as using profiling tools or measurement through bytecode
injection are typically not feasible for production systems but may be applicable in testing scenarios
lack of simple, integrated access to data

9. Challenges: Interpretation
Identifying underlying fault is more challenging:
still, important to identify problems early
Interpretation also means deciding when to react. Three basic approaches:
Thresholds: An issue is raised when values of interest approach, reach, exceed or fall below a defined limit.
In such cases, not only an instant value but also the frequency or duration with which a certain limit is reached
should raise an issue.
Trends/patterns: indicator (e.g., consumption of resources) shows a continuous trend /tendency or pattern
Correlations: An issue is raised when the values correlate with values of an already known anomaly or other
metric, that related to TD in the past. A typical example of such a pattern is CPU workload vs. garbage
collection activity.
We expect trends/patterns to have a high potential: Many hard-to-detect runtime problems follow similar patterns

10. Opportunities for Measuring Dynamic Indicators
Build (>1 daily) Unit-Tests
Commit metrics
Nightly
Build
daily Integration/performance
automated UI tests
Statistics weekly usage statistics
Performance test Sprint
Release
Performance-Tests
Production data
frequent
Rare/
irregular
Source Frequency Data
System in use
Procotols, logs
continuous runtime
examples!

11. Opportunities for Dynamic Indicators
QAware 11
Many opportunities exist (testing, software in operation)
Many (isolated) tools exist
tools that come with the operational system (e.g., Unix and Windows);
external monitoring tools (e.g., Nagios)
embedding measurement mechanisms (e.g., writing custom code that uses JMX/Jolokia) a
analyzing log files produced during runtime (e.g., using the ELK/Elastic stack).
cloud-native infrastructure (e.g., Zipkin for distributed tracing; Prometheus for metrics; Fluentd for logs)
Challenges:
Support defining relevant metrics
Improve automation
Integrate (collect/combine) data

12. Test Execution Time: Growing Trend
12
2 of 3 longest running test methods showed a
growing trend over months.
Cause: Unknown
A similar behavior in runtime/memory
consumption went undetected and caused a
runtime incident when existing memory was
exceeded.

13. Test Execution Time: Leap
13
Cause: Refactoring of test case.

14. Service Cascades
QAware 14
Service Cascades:
one incoming connection causes many outgoing connections
for example, one incoming web request call resulting in many database calls
Example: Java Server where incoming web request caused up to
2400 JavaScript calls,
30 service requests and
1600 database calls

15. Connections
QAware 15
Runtime input/output operations
Database calls
Connections
File handles
Typical patterns: Rising trend over
Runtime
Versions
Example: Windows system (48 h)
Cause: Missing call of Release()
for Mutex objects (here: file handles)

16. Parallelism
QAware 16
Metric: number of threads
E.g., via JMX/Jolokia
Pattern: rising number of threads over
Runtime
Versions/releases
Example: Java Server (8 days)
Two restarts during observation
Cause: 25 threads added every 5 hours
This works until maximum is reached (400)

17. Summary
QAware 17
Existing approaches for measuring TD
focus mainly on static aspects of code or near-code artifacts,
measured at development or build time
Gap: address dynamic indicators
properties emerging at system runtime
often an important cause of software bankruptcy or cost-intensive failure search
Presented examples of dynamic indicators of TD
frequently occur in practice
were difficult, or even impossible, to detect with static analysis techniques
were observable at system runtime with simple patterns

18. Challenges and Opportunities
QAware 18
Main challenges:
define: support in identifying relevant characteristics & metrics (highly context-specific; risk-based selection?)
measure meaningful things
measure: ready-to use tools for measuring, integrating and storing runtime data (production-ready)
integrate different tools
interpret: rule-based warnings, support follow-up (identifying relevant code vs. false positives; SonarQube-like)
make data accessible for analysis and interpretation
provide unified (simple) access
Future (today?): DevOps and Cloud-native applications create even more opportunities
plenty of measurement opportunities in such environments
every build or nightly build can include, for example, automated load tests;
plenty of measurement tool support (Prometheus, Fluentd, Zipkin,

19. Marcus Ciolkowski
marcus.ciolkowski@qaware.de
@M_Ciolkowski
github.com/qaware
linkedin.com/qaware slideshare.net/qaware
twitter.com/qaware xing.com/qaware
youtube.com/QAwareGmbH
Liliana Guzmán Anna Maria VollmerMarcus Ciolkowski AdamTrendowicz