"When GitHub meets CRAN: An Analysis of Inter-Repository Package Dependency Problems" Presentation of a research article on software ecosystems at the International Conference on Software Analysis, Evolution and Reengineering (SANER 2016, Osaka, Japan). Joint work by Alexandre Decan, Tom Mens, Maelick Claes, Philippe Grosjean, COMPLEXYS Research Institute, University of Mons, Belgium

1. When GitHub meets CRAN :
An Analysis of Inter-Repository
Package Dependency Problems
Alexandre Decan Tom Mens
Maelick Claes Philippe Grosjean
So5ware Engineering Lab
University of Mons, Belgium
h<p://informa@que.umons.ac.be/genlog/projects/ecos

2. SANER – Osaka, Japan, March 2016
• A so5ware ecosystem is:
• “a collec@on of so5ware products that have some given degree of
symbio@c rela@onships.” [Messerschmi< & Szyperski 2003]
• “a collec@on of so5ware projects that are developed and evolve
together in the same environment.” [Lungu 2008]
• A package-based ecosystem
• is composed of interdependent so5ware packages …
• … that can be installed and distributed using a package manager
• Examples
Package-based
So5ware Ecosystems

3. SANER – Osaka, Japan, March 2016
The R ecosystem
• R = popular open source language and so5ware
environment for sta@s@cal compu@ng
www.r-project.org
• Language popularity
(IEEE Spectrum)
2015 2014

4. SANER – Osaka, Japan, March 2016
The R ecosystem
Where can we find R packages?
cran.r-project.org
www.bioconductor.org
r-forge.r-project.org
www.github.org
Number of packages
(March 2015)
6,411
997
1,883
5,150

5. SANER – Osaka, Japan, March 2016
CRAN
distribu@on plakorm
• But…
– “It used to be that puDng your package on CRAN also
gave it some exposure, but with >6000 packages,
that’s no longer quite true.” [Broman 2015]
– “The non-transparent nature of the CRAN submission /
rejecUon process is parUcularly at issue.
h<ps://github.com/stnava/ANTsR/issues/8
The main advantage to geDng your package on CRAN
is that it will be easier for users to install.
Your package will also be tested daily on mulUple systems.
“R package primer: a minimal tutorial”,2015. Karl Broman

6. SANER – Osaka, Japan, March 2016
CRAN
distribu@on plakorm
Credits: h<p://www.designandanaly@cs.com/cran-gephi/
Package dependency graph of CRAN
6,706 unarchived packages
on June 1, 2015

7. SANER – Osaka, Japan, March 2016
CRAN
distribu@on plakorm
• Con@nuous integra@on process based on
R CMD Check to ensure overall consistency
– Daily check, on mul@ple OS, of all CRAN packages
– Looks for common problems in package structure,
metadata, code, documenta@on, data, …
– Packages not passing the check need to be
updated to avoid gepng archived
– Packages can only depend on the latest version of
another package è if they depend on a failed
package they may also fail the check

8. SANER – Osaka, Japan, March 2016
CRAN
distribu@on plakorm
Automated daily CRAN package Check (2016-02-23): Summary of results

9. SANER – Osaka, Japan, March 2016
CRAN
distribu@on plakorm
Automated daily CRAN package Check (2016-02-23): Results by package

10. SANER – Osaka, Japan, March 2016
R on GitHub
githut.info Q4, 2014

11. SANER – Osaka, Japan, March 2016
How important is GitHub
for R packages?
• CRAN used to be the main distribu@on
plakorm for R packages
• With the appearance of GitHub, this is no
longer the case
Is GitHub star@ng to replace or complement
CRAN as package distribu@on medium?

12. SANER – Osaka, Japan, March 2016
How important is GitHub
for R packages?
• GitHub hosts many R packages
• Increasing number of new R packages on GitHub
June 1, 2015
monthly number of new packages

13. SANER – Osaka, Japan, March 2016
How important is GitHub
for R packages?
Is GitHub being used as a distribu(on pla/orm
for R packages?
• GitHub hosts thousands of R packages
• >40% of GitHub R packages are intended to be distributed
as they contain instruc@ons to install them from GitHub.
• An increasing number of R packages on GitHub is not
distributed elsewhere
• Packages like devtools facilitate distribu@on from GitHub

14. SANER – Osaka, Japan, March 2016
Managing R
package dependencies
Jeff Leek
Jeroen
Ooms
The number of packages on CRAN and other repositories has
increased beyond what might have been foreseen, and is revealing
some limitaUons of the current design. One such problem is the
general lack of dependency versioning in the infrastructure.
“Possible direc@ons for improving dependency versioning in R,”
R Journal 5(1): 197–206, June 2013
One of the best things about the R ecosystem is being able to
rely on other packages so that you don’t have to write
everything from scratch. But there is a hard balance to strike
with keeping the dependency list small.
“How I decide when to trust an R package,”
simplysta@s@cs.org/?p=4409, November 2015.

15. SANER – Osaka, Japan, March 2016
Managing R
package dependencies
In which repository are package dependencies saUsfied?
ES SUFFER
OBLEMS?
around R’s
allation of R
repositories
available R
ojects filled
uages. Even
with the R
R package.
nts devtools
Additionally,
repositories
dependency
s hosted on
cy problems
of packages
by GitHub
t frequently.
er-repository
on GitHub depend on at least one package in CRAN, while
85.3% of all declared dependencies in R packages on GitHub
are satisfied by CRAN packages. Note that, if a package is
hosted by both CRAN and GitHub, we count it as a CRAN
package, because for dependency satisfaction we privilege
the officially distributed package over its development ver-
sion.
Other repositories
CRAN GitHub
0.5
1.598.0
85.3
10.7
4.00.8
1.8
61.5
70.6
11.8
8.6
Figure 5. Inter-repository package dependencies for CRAN and GitHub.
In blue: percentage of packages. In red: percentage of dependencies.
Because of the constraints imposed by CRAN’s daily
R CMD check, CRAN is self-contained. 98.0% of its
Blue = % of packages in repository A that
have a dependency sa@sfied by a package
in repository B
Red = % of dependencies of packages in
repository A that are sa@sfied by a package
in repository B
(BioConductor, OmegaHat, …)
CRAN is self-contained, so it does
not suffer from inter-repository
dependency problems.
But what about GitHub?

16. SANER – Osaka, Japan, March 2016
Managing R
package dependencies
In which repository are package dependencies saUsfied?
ES SUFFER
OBLEMS?
around R’s
allation of R
repositories
available R
ojects filled
uages. Even
with the R
R package.
nts devtools
Additionally,
repositories
dependency
s hosted on
cy problems
of packages
by GitHub
t frequently.
er-repository
on GitHub depend on at least one package in CRAN, while
85.3% of all declared dependencies in R packages on GitHub
are satisfied by CRAN packages. Note that, if a package is
hosted by both CRAN and GitHub, we count it as a CRAN
package, because for dependency satisfaction we privilege
the officially distributed package over its development ver-
sion.
Other repositories
CRAN GitHub
0.5
1.598.0
85.3
10.7
4.00.8
1.8
61.5
70.6
11.8
8.6
Figure 5. Inter-repository package dependencies for CRAN and GitHub.
In blue: percentage of packages. In red: percentage of dependencies.
Because of the constraints imposed by CRAN’s daily
R CMD check, CRAN is self-contained. 98.0% of its
Blue = % of packages in repository A that
have a dependency sa@sfied by a package
in repository B
Red = % of dependencies of packages in
repository A that are sa@sfied by a package
in repository B
(BioConductor, OmegaHat, …)

17. SANER – Osaka, Japan, March 2016
Managing R
package dependencies
Conclusions :
• Most packages on GitHub rely on packages in CRAN …
• … but the number of packages on GitHub that rely on
packages outside of CRAN is increasing
Do R packages on GitHub suffer from
inter-repository dependency problems?

18. SANER – Osaka, Japan, March 2016
Inter-repository
dependency problems
Observa@ons
• CRAN’s policy requires to depend on the most recent
package version
• GitHub packages frequently depend on CRAN packages
• They cannot benefit from CRAN’s daily check
• But can use other con@nuous integra@on processes
such as Travis CI
• Although less than 25% actually do this…
I personally think it’s REALLY relevant to at least be ABLE to be
very specific and rigid with regard to your dependencies.
And I think the R universe could provide beaer tools to fit the
needs of developers and professionals out there in a beaer way.

19. SANER – Osaka, Japan, March 2016
Inter-repository
dependency problems
Observa@ons
• CRAN’s policy requires to depend on the most recent
package version
• GitHub packages frequently depend on CRAN packages
• They cannot benefit from CRAN’s daily check
• But can use other con@nuous integra@on processes
such as Travis CI
• Although less than 25% actually do this…
I personally think it’s REALLY relevant to at least be ABLE to be
very specific and rigid with regard to your dependencies.
And I think the R universe could provide beaer tools to fit the
needs of developers and professionals out there in a beaer way.
è CRAN package updates more likely to lead
to problems for GitHub than for CRAN
packages

20. SANER – Osaka, Japan, March 2016
Inter-repository
dependency problems
CRAN package updates more likely to lead to problems
for GitHub than for CRAN packages
Survival Analysis:
Probability that a CRAN
package is NOT updated
period 12/2014 – 6/2015
3,740 considered packages
If a CRAN package is required
by a GitHub package, it is more
likely to be updated.

21. SANER – Osaka, Japan, March 2016
Inter-repository
dependency problems
How ocen do package updates cause
backward incompaUble changes on CRAN?

22. SANER – Osaka, Japan, March 2016
Inter-repository
dependency problems
How ocen do package updates cause
backward incompaUble changes on CRAN?
• 41% of the errors in CRAN packages are caused by
backward incompa@ble changes
(analysis based on daily R CMD Check results over 2-year period)
• We observed backward incompa@ble changes
on average once every 20 updates
(period december 2014 à June 2015)

23. SANER – Osaka, Japan, March 2016
Inter-repository
dependency problems
How ocen do package updates cause
backward incompaUble changes?
R package maintainers share this concern:
It is more and more of a pain if the
package I’m depending on breaks
If I have to load a Depends package, it adds a significant burden:
I have to check for conflicts every Ume I take a dependency
on a new package.

24. SANER – Osaka, Japan, March 2016
Inter-repository
dependency problems
How ocen do package updates cause
backward incompaUble changes?
R package maintainers share this concern:
The risk of things breaking at some point due to the
fact that a version of a dependency has changed
without you knowing about it is immense.
I had one case where my package heavily depended
on another package and acer a while that package
was removed from CRAN and stopped being maintained.

25. SANER – Osaka, Japan, March 2016
Wrap-up
• GitHub is becoming increasingly used to
develop and distribute R packages
• Package updates o5en lead to incompa@bili@es
in depending packages
• GitHub packages cannot benefit from CRAN’s
con@nuous integra@on process
Jeroen
Ooms
One recent example was the forced roll-back of the ggplot2 update to version
0.9.0, because the introduced changes caused several other packages to break.
“Possible direc@ons for improving dependency versioning in R,”
R Journal 5(1): 197–206, June 2013

26. SANER – Osaka, Japan, March 2016
Wrap-up
• R community needs be<er tool support for
managing dependencies and dealing with package
versions.
• Such tools are star@ng to emerge:
– devtools, drat, packrat, miniCRAN, checkpoint, …
– R-Hub: ongoing project accepted by R consor@um
• Infrastructure for developing, building, tes@ng, valida@on,
distribu@on, con@nuous integra@on of R packages.
• A service that is complementary to CRAN and R-Forge

27. SANER – Osaka, Japan, March 2016
• On the maintainability of CRAN packages
[CSMR-WCRE 2014]
• maintaineR, a web-based dashboard for maintainers of CRAN
packages
[ICSME 2015]
• An empirical study of idenUcal funcUon clones in CRAN
[IWSC 2015 @ SANER]
• Anonymized e-mail interviews with R package maintainers
acUve on CRAN and GitHub
[Technical Report]
Website:
informa@que.umons.ac.be/genlog/projects/ecos/
27
References

28. SANER – Osaka, Japan, March 2016
References
/var/folders/lm/
qbg6frqd6435jxm9w70y7h480000gn/T/
com.apple.Preview/
com.apple.Preview.PasteboardItems/
csmr13-140314140935-phpapp02
(dragged).pdf
/var/folders/lm/
qbg6frqd6435jxm9w70y7h480000gn/T/
com.apple.Preview/
com.apple.Preview.PasteboardItems/
csmr13-140314140935-phpapp02
(dragged).pdf
The Evolution of the R Software Ecosystem
Daniel M. German
University ofVictoria
Bram Adams
École Polytechnique
de Montréal
Ahmed E. Hassan
Queen's University
CSMR 2013