The document discusses the interactions of variability layers in the evolution of MongoDB, presented at a data challenge in Beijing. It highlights experiments assessing the impact of runtime environments and hardware platforms on software performance evolution, utilizing dynamic time warping to identify similar performance trends across different hardware. Key takeaways emphasize the significance of runtime environments in quantifying software evolution and the necessity for feedback from domain experts.

1. Beware of the Interactions of Variability Layers
when Reasoning about the Evolution of MongoDB
Luc Lesoil, Mathieu Acher, Arnaud Blouin & jean-marc Jézéquel
2022/04/12
Beijing, China
Data Challenge

2. ≠ Thread Levels
≠ Perf Evolutions
Joint evolution of mongoDB change points (top) and performance values (bottom)
Code
User #1 User #2
Thread Level = 512
Perf ↘ Perf ↗
Dataset: Expanded Metrics, Project: sys-perf, Task: industry_benchmark_wmajority,
Hardware: linux-3-node-replSet, Test: csb_50_read_50_update_w_majority
Thread Level = 1
Dev
?
Impact of runtime environments on software evolution
2/5
Interactions between
the runtime environment &
the evolution of the software
[1] The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System, Daly et al., ICPE 2020, https://dl.acm.org/doi/abs/10.1145/3358960.3375791
[1]

3. Experiment - Compute the DTW
for all combinations of hardware platforms
Impact of hardware platforms on software evolution
Heatmap of DTW between times series
related to different variants of hardware
ⓑ DTW = 0.38
ⓓ DTW = 5.39
What is the Dynamic Time Warping?
Similar
Different
Result - Identify hardware platforms
having similar evolutions
to reduce the cost of benchmarking 3/5

4. ⓑ DRPC = 1.61%
ⓒ DRPC = 25.07%
Impact of workloads on software evolution
Experiment - Compute the DRPC
distribution for each workload
Result - Identify stable workloads
to use in benchmarks
Daily Relative Percentage Change
● p(t) the performance value at the time t
● d(t, t+1) the number of days between t and t+1 4/5

5. Takeaway Message
Runtime environments matter (when quantifying software evolution)!
@David and MongoDB performance team
Need feedback & domain knowledge to draw actionable conclusions
Thanks for this Data Challenge !
5/5

6. Back-Up Slides

7. Pre-processing of Time Series
Only consider the period of definition
common to the two Time Series
Linear interpolation if a point is
present only in one TS
1
Time
Performance
TS #1
TS #2

8. These high values can be due to:
- the standardisation if the standard
deviation of the distribution is too
low
- outliers in the TS
We have to standardise because TS
have different scales
High DTW values for couple of hardware platforms
2