As the number of packages available for R continues to grow, maintaining and testing these packages becomes more difficult. This difficulty is compounded as independent implementations of the R language, such as TIBCO Enterprise Runtime for R (TERR), are developed. To address this, we have created a test automation framework for testing packages with both TERR and R. We will describe how the framework automatically creates tests from a package's source files. Issues with testing on multiple platforms will be discussed. Suggestions for improving packages with tests will also be presented.

1. Software Testing and the R Language
useR! 2014, U.C.L.A.
Stephen Kaluzny
TIBCO Software Inc.
July 3, 2014
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 1 / 21

2. Introduction
1 Testing R engines
TIBCO’s approach
Not validating R (Andy Nichols talk on Wednesday)
2 Testing R packages
How does one test a package?
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 2 / 21

3. Motivation and Background
Company has been working on S/R engines for 26 years
Originally S and S-PLUS with StatSci / MathSoft / Insightful / TIBCO
Now working with TIBCO Enterprise Runtime for R (TERR)
TERR is an embed-able R engine for enterprises
the analytics engine in TIBCO Spotfire visualization software
”Develop in R, Deploy in TERR”
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 3 / 21

4. Importance of Testing
Need to ensure correct results from the software
Is R giving the correct results?
R Core has this covered fairly well
Do your results match R?
There are now several R engines that need testing, including:
TERR
Renjin (jvm)
ORE Statistics Engine (Oracle)
Test with use of different software libraries e.g.
linear algebra
math functions and statistical distributions (Intel Math Library)
R on special hardware
GPU for linear algebra
multi-core systems
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 4 / 21

5. Testing the TERR Engine
Developers write unit tests as features (functions) are developed
in C++ and R
Quality Control group write tests for each feature
often converting tests developed for S-PLUS
Quality Control group write regression tests for bugs that are fixed
these tests often cover complex areas of the code
We have created several packages to assist with development and
testing
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 5 / 21

6. Assertion Test Framework
Based on loop test idea for testing from John Chambers for S version3
Each individual test is an S expression that evaluate to true if
successful
Collection of tests for a function are grouped into a file
Each file is run in clean environment
Metadata at the top of the file is used by the test framework
Entire test system is controlled by a Jenkins job
can request only particular tests
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 6 / 21

7. Assertion Test Metadata
## TestName: glm.t
## Package: stats
## FunctionalArea: statistical models
## Tags: models, glm, model.frame
## Description: Basic glm function
## RequiredPackages:
## UsesRandom: FALSE
## Author: ctaylor, meriksso, nmockler
## TestType: unit
## TestFormat: assertion
## NotBeforeVersion:
## NotAfterVersion:
## ExcludePlatforms:
## RunsInSnext: TRUE
## RunsInR: TRUE
## RunsInSplus:
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 7 / 21

8. assertionTest Package
Created to support Assertion Test Framework
Not unlike the testthat package
Key Functions:
assertionTest
the main function, run on a file of expressions
atExpectStop and atExpectWarnings
atUsesRandom
resets seed after each expression
error if .Random.seed changes outside of this function
atSource
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 8 / 21

9. Assertion Test Stats
Tests are run after every nightly build
3 build platforms (Win-32, Win-64, Linux-64)
4 test platforms (Win7-32, Win7-64, Win8-64, Linux-64)
Two compilers on each platform (Intel and gcc or Microsoft)
Two builds of each compiler/platform combination (Release, Debug)
Results are written to a MySQL database
Over 1000 test files for 16 ”core” packages in TERR
Developer ”unit” assertion tests are run during continous builds
build runs every 10 minutes if there has been a checkin
build results email goes to any developer who made a checkin for that
build
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 9 / 21

10. Other Tests for TERR
Manual Tests
run by QC group before a release
Performance Tests
spiff test - diff tests (ala Rout.save) with a numerical tolerance
mostly adapted from S-PLUS
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 10 / 21

11. RinR Package
Evaluates R expressions in a different R interpreter
Includes evaluator functions that evaluate expressions in particular
interpreter
REvaluator - R interpreter
TERREvaluator - TERR interpreter
LocalEvaluator - current interpreter
makeREvaluator - creates evaluators e.g. R2.15.3Evaluator
multiREvaluator - evaluate an expression in multiple interpreters
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 11 / 21

12. RinR Examples
# From within TERR:
> library(RinR)
> REvaluate(version)
platform x86_64-unknown-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 3
minor 1.0
year 2014
month 04
day 10
svn rev 65387
language R
version.string R version 3.1.0 (2014-04-10)
nickname Spring Dance
> .libPaths(c(.libPaths(), REvaluate(.libPaths())))
> .libPaths()
[1] "/home/TERR/TERR-intel.140620/library"
[2] "/home/R/R-3.1.0/lib/R/library"
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 12 / 21

13. RinR Examples (page 2)
RCompare is often used during development
> compareQuant <- RCompare(quantile(c(0,1,1,2,3,5,8,13)))
> class(compareQuant)
[1] "sideBySide"
> compareQuant
R version 3.1.0 (2014-04-10) TERR version 3.0.0 (2014-06-20)
[1,] 0% 25% 50% 75% 100% 0% 25% 50% 75% 100%
[2,] 0.00 1.00 2.50 5.75 13.00 0.00 1.00 2.50 5.75 13.00
[3,]
[4,]
$all.equal
$all.equal$R version 3.1.0 (2014-04-10) vs. TERR version 3.0.0 (2014-06-2
[1] TRUE
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 13 / 21

14. Future TERR Testing
Add new tests as new features are added
Incorporating tests from core R
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 14 / 21

15. Testing Packages
One of our biggest challenges
First question from customers: Does package xyzzy work with
TERR?
What does it mean for a package to work?
How do you test over 5000 packages?
Package versions and dependencies add to the challenge
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 15 / 21

16. Package Testing with R
R CMD check does some testing
runs files in tests
compare to Rout.save files, if the exist
runs examples from help files
compare to Rout.save files, if they exist in tests/Examples
Some packages formal tests that can be run, using these packages:
RUnit : R Unit test framework
scriptests : Transcript-based unit tests that are easy to create and
maintain
svUnit : SciViews GUI API - Unit testing
testit : A simple package for testing R packages
testthat : Testthat code. Tools to make testing fun :)
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 16 / 21

17. CRAN Test Statistics
CRAN packages from early June, 2014 - 5566 packages
961 (17.3%) have at least one tests directory
230 (4.1%) have target files for comparing test results (‘*.Rout.save‘)
394 of the 961 (7.1%) have actual software tests:
262 (4.7%) use testthat
113 (2.0%) use RUnit
10 (0.2%) use svUnit
5 (0.09%) use testit
4 (0.07%) use scriptests
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 17 / 21

18. Package Tests
Without creating our own specific tests, from a package we can run:
tests (if they exist)
example code from the help files
code extracted from any vignettes
Problems with this approach
package tests do not always include output to check results
help file code examples seldom have output to compare
help file code examples show usage, not necessarily testing the code
vignette is also to illustrate usage, not for testing the code
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 18 / 21

19. Testing Packages with TERR
Run tests, help file examples and vignette examples in R and TERR
Compare results of each each expression
Instrumented in a package (‘teaextract‘) that is run from R
use the parallel package to divide up the testing
runs now down to 23 hours from over 6 days
Currently: 1795 CRAN packages run without any errors in TERR
830 give exactly the same results as R
Many packages have a small fraction that do not run
missing a minor function in TERR
Failures with examples that use random numbers, timestamps
Improvement of this system is a priority in the QC group
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 19 / 21

20. Summary and the Future
Testing R Engines
well covered by R and TIBCO
Testing packages - a big challenge
limited tests in packages, there should be more
use of testthat is a good sign
testCoverage package from Mango Solutions would help
does not solve the problem
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 20 / 21

21. Questions?
Stephen Kaluzny (TIBCO Software Inc.) Software Testing July 3, 2014 21 / 21