1.
An Empirical Analysis of
Technical Lag in npm Package
Dependencies
Ahmed Zerouali, Eleni Constantinou, Tom Mens,
Gregorio Robles and Jesus M. Gonzalez-Barahona
The 17th International Conference on Software Reuse
May 21-23, 2018 - Madrid

2.
/background
/aims
/method
/results
/limitations
/conclusion
Outline

3.
/background
Packages Releases Dependencies (runtime)
+700K +4.5M +20M
+145K +825K +2M
+130K +840K +2.3M
Libraries.io by March 2018
RubyGems

4.
/background
Open PRs Active Bugs
+2.3M -
+2M -
- +120K
by January 2018 - https://octoverse.github.com/

5.
/background
Technical lag*: the increasing difference between
deployed software packages and the available
upstream packages
Measurement: version updates, bugs, vulnerabilities,
line of code, commits, etc.
(*) Gonzalez-Barahona, et al. "Technical Lag in Software Compilations: Measuring How Outdated a Software Deployment Is."
IFIP International Conference on Open Source Systems. Springer, Cham, 2017.
Gold standard: stability, security, functionality, etc.

6.
/background
Example: different kinds of “gold standards” for Debian
Gold standard Scenario Candidate
Stability Isolated system, stable
functionality
Debian Stable
Functionality Cloud application Latest upstream
Security Reused containers Stable upstream

7.
/background

8.
/aims
Decan A, Mens T, Grosjean P. An empirical comparison of dependency network evolution in seven software packaging ecosystems. EMSE2017.

9.
/aims
Not update
Update
“How one developer just broke Node, Babel and thousands of projects in 11 lines of JavaScript” -
https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/

10.
/aims
Goal: Analyze technical lag in a wide ecosystem of
reused of packages.

11.
/aims
/ research questions
RQ0: How do packages manage their dependencies?
RQ1: How often do packages release new versions?
RQ2: What is the technical lag induced by outdated
dependencies?
RQ3: How often do dependencies inducing technical lag release
new versions?
RQ4: What is the appropriate moment to update dependencies?

12.
/method
/dataset

13.
/method
/dataset
Open Data:
- Libraries.io gathers data from 36 package managers and 3 source code
repositories.
- They track over 2.7m unique open source packages, 33m repositories and
235m interdependencies between them.

14.
/method
/dataset
- 610K packages
- 4.2M releases
- 44.9M dependencies
by Nov 2017
from

15.
/method
/semantic versioning
Examples: 0.0.1, 1.0.0, 1.2.3, 1.2.3-beta

16.
/method
/semantic versioning
Other: *, ==1.2.3, >1.2.3, <1.2.3, 1.2.x, 1.x.x

17.
/method
/technical lag
- Measurement = version updates, time
- version lag : version updates difference
- time lag: time difference
- Gold standard = being up to date.

18.
/method
/technical lag
1.0.1 1.1.0 2.0.01.2.1 2.1.0
Dependency: D
npm package
version
Technical lag

19.
/method
/technical lag
- time lag = date(latest) - date(used)
- version lag = (∆Major, ∆Minor, ∆Patch)

20.
/method
/technical lag
1.0.1 1.1.0 2.0.01.2.0 2.0.1
Dependency: D
npm package
version
Technical lag
- time lag= date(2.1.0) - date(1.1.0)

21.
/method
/technical lag
1.0.1 1.1.0 2.0.01.2.0 2.0.1
Dependency: D
npm package
version
Technical lag
1 minor
- time lag= date(2.1.0) - date(1.1.0)

22.
/method
/technical lag
1.0.1 1.1.0 2.0.01.2.0 2.0.1
Dependency: D
npm package
version
Technical lag
- time lag= date(2.1.0) - date(1.1.0)
1 minor
1 major

23.
/method
/technical lag
1.0.1 1.1.0 2.0.01.2.0 2.0.1
Dependency: D
npm package
version
Technical lag
1 minor
1 major 1 patch
- time lag= date(2.1.0) - date(1.1.0)
- version lag= (1,1,1)

24.
/method
/technical lag
1.0.1 1.2.0 2.0.1
3.6.0 4.1.04.0.0 5.0.0
2.0.0 2.1.0
npm package: P
dependency: D
^1.0.0
Technical lag
*
^1.0.0 ^2.0.0
^1.0.0 = [ 1.0.0, 2.0.0 [
allowed

25.
/method
/technical lag
1.0.1 1.2.0 2.0.1
3.6.0 4.1.04.0.0 5.0.0
2.0.0 2.1.0
npm package: P
dependency: D
^1.0.0
Technical lag
*
^1.0.0 ^2.0.0
allowed
^1.0.0 = [ 1.0.0, 2.0.0 [

26.
/method
/technical lag
1.0.1 1.2.0 2.0.1
3.6.0 4.1.04.0.0 5.0.0
2.0.0 2.1.0
npm package: P
dependency: D
^1.0.0
Technical lag = 0
*
^1.0.0 ^2.0.0
allowed
^1.0.0 = [ 1.0.0, 2.0.0 [

27.
/results
/RQ0: How do packages manage their dependencies?
68.2%
15.7%
7.8%
4.1%
- Developers are concerned with backward compatibility
- There is a potential of too strict dependency constraints leading to
technical lag.
4.3%

28.
/results
/RQ0: How do packages manage their dependencies?
- New dependencies are mainly added in major and minor releases.
- Dependencies are removed almost exclusively in major releases.
- Most packages do not appear to change their dependencies over time.

29.
/results
/RQ0: How do packages manage their dependencies?
Number of updated dependencies between package releases,
classified by release type of the update.

30.
/results
/RQ1: How often do packages release new versions?
- Patch: 80% - Minor: 16% - Major: 04%
- Dependent packages in npm mainly benefit from patch releases.
- Technical version lag is mainly occurring at the patch level.

31.
/results
/RQ1: How often do packages release new versions?
Time needed to update a package to a patch, minor or major release
- The average time to release a patch, minor and major versions
corresponds to 13 days, 1 month and 2 months respectively.

32.
/results
/RQ2: What is the technical lag induced by outdated dependencies?
- 27% of 44.1M dependencies are outdated.
- The outdated dependencies are used by 60% of all considered packages.
57%
28%
12%
3%

33.
/results
/RQ2: What is the technical lag induced by outdated dependencies?

34.
/results
/RQ2: What is the technical lag induced by outdated dependencies?
- Outdated dependencies induce a median of time lag of three months
and a half, and median version lag of one minor and two patch versions.

35.
/results
/RQ3: How often do dependencies inducing technical lag release new versions?
- Packages that are required as dependencies and are outdated have
more frequent releases than other required packages.

36.
/results
/RQ4: What is the appropriate moment to update dependencies?
- Developers should not start using newly available packages
immediately because they may still contain bugs that need new
patches.

37.
/limitations
- If the libraries.io dataset is incomplete, then there is a risk of underestimating technical lag.
- We did not differentiate between package characteristics, such as age, size, type, etc.
- The results are related to the measurement used to quantify for technical lag.
- npm semver had some issues in the past.

38.
/conclusion/
summary
Analyzed technical lag induced by outdated dependencies:
- A large number of packages suffer from technical lag.
- Outdated dependencies are several months behind the latest release.
- Technical lag caused by the specific use of dependency constraints,
- Maintainers should wait a few days before updating to the new patch dependency release.

39.
/conclusion/
future work
- Consider other measurements of technical lag and other gold standards.
- Validate the results with bug fixes, vulnerabilities and issues.
- Consider other ecosystems.
- Carry out cross-ecosystem comparisons.

40.
Questions