SourceWarp is a scalable approach to support data-driven decision making for CI/CD tools and DevSecOps platforms. It allows running experiments on these systems by replaying commit histories from source code repositories without deploying features. This evaluates features through testing and benchmarking. An industrial case study at GitLab evaluated a new vulnerability tracking approach using SourceWarp. It replayed commits from GitLab's source code repository on systems with and without the feature. SourceWarp completed the experiment faster than deploying the feature and provided metrics showing the impact of vulnerability tracking on error reduction.

1. SourceWarp: A scalable, SCM-driven testing
and benchmarking approach to support
data-driven and agile decision making for
CI/CD tools and DevOps platforms
Vulnerability Research @ GitLab
Julian Thome, James Johnson, Isaac Dawson, Michael Henrisken, Dinesh Bolkensteyn, Mark Art
1

2. Vulnerability Research @ GitLab 🔗
2

3. Vulnerability Research @ GitLab
- Advance GitLab security offering towards
long-term vision.
- Improve GitLab efficacy of security features.
- Research/PoC/Benchmarking.
- Publish papers/Blog Posts/Participation @
Conferences.
3

4. Background
4

5. Background
- Agile: shorten the design phase of software; iterate quickly;
establish feedback cycle with user/customer.
- Development Security Operations (DevSecOps): Builds on
agile philosophy; focus on technical practice with
cross-functional teams.
- Continuous Integration/Continuous Delivery (CI/CD): focus
on tools; provide automation around software (managed by
Source Code Management Systems (SCM)).
- Data-Driven Decision Making (DDDM): Validating a feature to
make a qualified assessment about its quality/performance.
5

6. We see Agile being scaled with
70% of respondents indicating
an ambition to integrate both
Business and IT enabled Agile
transformation in the next 3
years.
With over 68% most
organisations state faster
product delivery as one of their
key drivers for agility, … .
KPMG Survey on Agility 2019 🔗
120 participants from 17
countries.
6

7. No other technology is as
widely used as Git.
Especially among
Professional Developers. But
for those learning to code,
17% still do not use a version
control system.
StackOverflow Dev Survey 🔗
Version Control Systems
7

8. Most Professional
Developers report having
CI/CD, DevOps, and
Automated Testing available
at their organization.
StackOverflow Dev Survey 🔗
Developer Experience:
Processes, tools, and
programs within an
organization
8

9. Background
- Agile Software Development is widely adopted.
- Rising popularity and adoption of SCMs in
combination with CI/CD have contributed to the
adoption of agile software development.
9

10. Motivation
10

11. Birds Eye View of a DevSecOps Platform
DevSecOps Platform
SCM Repository
Project(s)
Job 1 (CI/CD Tool A)
Job 2 (CI/CD Tool B)
Job 3 (CI/CD Tool C)
Job 4 (CI/CD Tool D)
Backend Service
Backend Database
Vulnerability
Management
Dependency
Management
License
Compliance
Code Quality Audit Events CI/CD Status …
API
11

12. Motivation
- DevSecOps Platforms are heterogeneous.
- Partially integration proof-of-concepts not
necessarily robust product feature may have a
negative impact on the stability of the entire
platform.
- A feature has to be shipped before its impact can
be measured.
12

13. Motivation
- Reduce Velocity?
- Not an option in an Agile setting.
- Feature Flags, A/B testing, Canary testing?
- Not designed to provide preliminary insights
before deployment and (ideally) while building
the feature.
- Not deeply integrated with SCM systems.
13

14. SourceWarp
- SCM driven testing and benchmarking approach.
- Run an experiment on DevSecOps and/or CI/CD
tools without having to deploy the feature to
staging or production; collect metrics;
Data-driven decision making (DDDM).
- Quickly (re-)run experiments parallel to the
feature development which is especially useful
in an Agile context.
14

15. Data-driven Decision Making for
DevSecOps Platforms in an
Industrial Setting
15

16. Data-Driven Decision Making
- Observability: Code in SCM changes over time.
- Reproducibility: Re(-run) the same testing and
benchmarking exercise on different systems.
- Scalability: Scale to large SCMs with long histories
(MLOC).
- Configurable Granularity: Cherry-picking
test-data and configuring the granularity with
which the tests are executed and metrics are
collected.
16

17. Approach
17

18. Approach
Record
Source SCM
Record
Store
Patch Sequence
Generation
Replay
Target SCM
Jobs
Jobs
Evaluation
Script
Patch Sequence
System Under
Test (SUT)
Monitor
SourceWarp
Results/Metrics
Report
18

19. Approach
Record
Source SCM
Record
Store
Patch Sequence
Generation
Replay
Target SCM
Jobs
Jobs
Evaluation
Script
Patch Sequence
System Under
Test (SUT)
Monitor
SourceWarp
Results/Metrics
Report
19

20. Record
Source Code Repository:
Commit Timestamps:
Time Interval:
Slice:
Cherry-pick relevant Commits.
20

21. Approach
Record
Source SCM
Record
Store
Patch Sequence
Generation
Replay
Target SCM
Jobs
Jobs
Evaluation
Script
Patch Sequence
System Under
Test (SUT)
Monitor
SourceWarp
Results/Metrics
Report
21

22. Patch Sequence Generation
Slice:
Patch Sequence:
Extract Patch Sequence to be replayed
Example:
22

23. Approach
Record
Source SCM
Record
Store
Patch Sequence
Generation
Replay
Target SCM
Jobs
Jobs
Evaluation
Script
Patch Sequence
System Under
Test (SUT)
Monitor
SourceWarp
Results/Metrics
Report
23

24. Monitor
1. Initialize new target SCM repository based on the
source SCM history (before the first patch that
includes the first relevant commit).
2. Replay patch sequence on the target SUT (chain of
CI/CD tools or pipeline, DevSecOps platform or a
combination of them).
3. Observe/Monitor the behaviour of target system
during the replay, check for errors (Testing) and
collect metrics (Benchmarking).
24

25. Evaluation - Industrial Case
Study - Vulnerability Tracking
25

26. Industrial Case Study - Vulnerability Tracking
- Vulnerability Tracking is an automated process
that helps deduplicating and tracking
vulnerabilities throughout the lifetime of a
software project.
- A vulnerability is identified by means of a
fingerprint.
26

27. Industrial Case Study - Vulnerability Tracking
- Code Volatility: Source code is volatile; as
developers make changes, source code may move
within files or between files.
- Double Reporting: Already reported vulnerabilities
that are being tracked in the Vulnerability Report.
If the code fragments are not tracked reliably as
they move, vulnerability management is harder
because the same vulnerability could be reported
again.
27

28. Industrial Case Study - Feature
- At GitLab, we have implemented a new
Vulnerability Tracking approach initially as a
proof-of-concept (PoC) to reduce the negative
effect (noise) of both double reporting as well as
code volatility.
- We needed evidence about the performance of the
feature before deploying it.
28

29. Evaluation Setup
- Equipped dockerized Ruby SAST tool brakeman
with vulnerability tracking: one version with
(brakeman+VT) and one version without tracking
(brakeman).
- GitLab source repository (3.7MLOC, history 200K
commit) between 2020-10-31 – 2020-12-31 and
allow-list of 60 relevant files.
29

30. Evaluation Setup
- We replayed the slice of the Git history on the two
target systems running brakeman (old
fingerprinting) and brakeman+VT using
SourceWarp.
30

31. Research Questions
- RQ1: Is SourceWarp effective in
supporting DDDM?
- RQ2: What is the impact of the allow-list
and the patch sampling number?
31

32. Results - Time
Record Time Replay Time Avg. Replay
Time Per Patch
Overall Time
brakeman 54m 30s 18m 19s 1m 24s 1h 12m 49s
brakeman+VT 52m 30s 17m 50s 1m 22s 1h 12m 20s
𝚫abs
0s 29s 2s 29s
𝚫rel
(%) 0 2.6 2.6 0.6
32

33. Results - Errors and Fingerprints
# Unique fingerprints for patchi
⚠
1 2 3 4 5 6 7 8 9 10 11 12 13
brakeman 0 94 94 97 97 102 102 118 118 125 125 128 128 132
brakeman+VT 0 83 83 84 84 84 84 91 91 91 91 91 91 92
𝚫abs
0 11 11 13 13 18 18 27 27 34 34 37 37 40
𝚫rel
(%) 0 11 11 13 13 17 17 22 22 27 27 28 28 30
33

34. Research Questions
- RQ1: Is SourceWarp effective in
supporting DDDM? Yes
- Performed experiment (to measure overhead,
cost savings and robustness) in a fully
automated and reproducible manner.
- Collected all the required data to make an
informed decision about the product
integration of Vulnerability Tracking.
34

35. Research Questions
- RQ2: What is the impact of the allow-list
and the patch sampling number?
- allow-list: (~61h) → (~3h)
- patch-sampling: (~18m)
35

36. Research Questions
- RQ2: What is the impact of the allow-list
and the patch sampling number?
- Reduce the benchmarking and testing time
from days to hours. This shows that these
heuristics enable SourceWarp to scale to
large, heterogeneous DevSecOps
platforms.
36

37. Summary
37

38. Blog Post/Tool Demo/Link to SourceWarp FOSS Tool
https://about.gitlab.com/blog/2023/04/13/data-driven-decision-making-with-sourcewarp/
38

39. 39