The document discusses the importance of reproducibility in research, particularly focusing on the R programming environment. It proposes changes to R's package management system to enhance reproducibility, such as local installation of packages and daily snapshots of CRAN, aiming to make it easier to ensure that specific versions of packages are used in research. Additionally, it addresses challenges related to package updates and dependencies that can hinder reproducibility in research projects.

1. R and Reproducibility
A Proposal
David Smith
useR! 2014

2. What is Reproducibility?
“The goal of reproducible research is to tie
specific instructions to data analysis and
experimental data so that scholarship can be
recreated, better understood and verified.”
CRAN Task View on Reproducible Research (Kuhn)
• Method + Environment
-> Results
• A process for:
– Sharing the method
– Describing the environment
– Recreating the results
2 xkcd.com/242/

3. Why care about reproducibility?
Academic / Research
• Verify results
• Advance Research
Business
• Production code
• Reliability
• Reusability
• Regulation
3
www.nytimes.com/2011/07/08/health/research/08genes.html
http://arxiv.org/pdf/1010.1092.pdf

4. R and Reproducibility
4
Results
Interfaces
Platform
Packages
R Engine
• Hand-assembled
• Sweave/knitr/DeployR/Shiny
• R GUI / DevelopR / RStudio
• Batch / Web Services
• OS / Virtualization
• Hardware Architecture
• CRAN
• BioConductor / GitHub / …
• R Version
• Base + Recommended pkgs

5. Observations
• R versions are pretty manageable
– Major versions just once a year
– Patches rarely introduce incompatible changes
• Good solutions for literate programming
– Interfaces help
• OS/Hardware not the major cause of
problems
• The big problem is with packages
– CRAN is in a state of continual flux
5

6. Package Problem #1 : The User
http://xkcd.com/234/6
I heard you need to create a
TPS Report. Here, I’ve got an
R script that does that
already.
Oh, you need to
download these 5
packages first.
I already
did, and it
still
doesn’t
work!
Well, it worked when I
wrote it 3 weeks ago.
YOUR
Grr.
Package
updates…

7. Package Problem #2: The Author
http://xkcd.com/970/7
Time to update
my package on
CRAN!
>> Dependent
packages that
now fail to build:
67
>> Resubmit
your package
and try again
Crap.

8. Package Problem #3 : The Update
http://xkcd.com/664/8
3 days later…
Woot! A new version of R
is out! I have 10 minutes
now, time to download
and install!
… package not found …
… can’t install package…
… error …

9. The Proposal
• Change the default way R handles packages
– Install packages local to projects
• “Snapshot” CRAN daily
– Make it easy to get & use package versions used in script
development
Not a new idea!
– Ooms, “Possible Directions for Improving Dependency
Versioning in R”, R Journal 5/1
– BioConductor Project
– Revolution R Enterprise
– Linux distros
9

10. Example
• R script file using 6 most popular packages
10

11. Sharing a script reproducibly
… and simply
# Run with R 3.1.0
require(RRT)
mran_set(snapshot="2014-06-27")
# find packages used in this project
# get package versions used by script author
# install locally to this project
require(ggplot2)
require(data.table)
require(knitr) …
11

12. RRT: The R Reproducibility Toolkit
• Open Source R Package (GPLv2)
• From an R project folder:
– Detect packages & dependencies used in project
– Download and install from MRAN
– Versions selected according to script date
– Find and use packages from local install
github.com/RevolutionAnalytics/RRT
12

13. MRAN - Implementation
A downstream CRAN mirror with daily snapshots
• Use rsync to mirror CRAN daily
– Only downloads changed packages
• Use zfs to store incremental snapshots
– Storage only required for new packages
• Organize snapshots into a labelled hierarchy
– Access package versions by date of use
• CRAN snapshot server hosted by cloud provider
– Provisioned for availability and latency
13

14. Future work
• Just getting started!
• Snapshot binaries and source packages
• Other repos (BioConductor, GitHub, user)
• Institution-level package duplication
– CRAN “behind the firewall”
• User-defined package versions
• Checks on R versions
• Suggestions welcome!
github.com/RevolutionAnalytics/RRT
14

15. Thank You!
David Smith
david@revolutionanalytics.com
blog.revolutionanalytics.com

16. Possible Solution
• Bundle all packages with scripts
• Packrat solves this very well
– Project + package dependencies stored in Github
• But:
– Contributes to package fragmentation
– Adds friction to the sharing process
– Doesn’t address the problem for R generally
16

17. CRAN vs Github
CRAN
• “Repository of Record”
– Default for R users
• Strict quality checking
• Handles dependencies
• Binaries built
– But only current versions
saved
• Manual update process
• Dependent on volunteer
support
Github
• Frictionless publishing /
updates
– RStudio integration
• Social development
– Pull requests FTW
• Ease of updates
• Fragmented – no unified
directory of packages
• Permanence – accounts
closed / repos deleted
17

18. A downstream CRAN solution?
“I don't see why CRAN needs to be involved in
this effort at all. A third party could take
snapshots of CRAN at R release dates, and make
those available to package users in a separate
repository. It is not hard to set a different
repository than CRAN as the default location
from which to obtain packages.”
-- R-core member, r-devel, March 2014
18

19. Snapshot CRAN repository :
requirements
• Availability
• Latency
• Bandwidth
• Storage
• Binary package archives
• Other enhancements?
19

20. Proposal
“Development Branch” “Stable Branch”
Defaults are important!!20
MRANCRAN Downstram
Reproducible