This document discusses tools for promoting reproducibility in computational research. It summarizes a project that tested Sumatra and CoRR for managing simulations and recording their execution contexts. A benchmark phase-field problem was used to evaluate these tools. The analysis found that Sumatra and CoRR were effective for autonomous recording and versioning of simulations, but more work is still needed in these areas and computational reproducibility in general.

1. U n i v e rs i t y LO G O
Testing and Developing Tools to Promote
the Reproducibility of Computational
Research
Andrey Moskalenko
Center for Theoretical and Computational Materials Science
Daniel Wheeler | Faical Yannick P. Congo

2. Reproducible Research
• Main Areas:
• Computational
• Experimental

3. •Context of the Project
• Simulation Management
• Sumatra and CoRR
• Benchmark Phase Field Problem
• Conclusion
Table of Contents

4. U n i v e rs i t y LO G O
• Context of the Project
•Simulation Management
• Sumatra and CoRR
• Benchmark Phase Field Problem
• Conclusion
Table of Contents

5. U n i v e rs i t y LO G O
Simulation Management
The GoalComputational Research Now

6. U n i v e rs i t y LO G O
Current available tools
Robust
Command line
Web integration
Highly collaborative
Not suitable for
capturing execution
context
Suitable for recording
stable automated
executions
Provides log, search and
view of execution history
Capture entire
simulation context
Version environments
Collaborative
Not collaborative
with current tools
Not robust or
ubiquitous
Not suitable for log,
search and view of
history
Suitable for building
pipelines of distinct
tasks
Enables a clear
division of tasks for
non-experts
Black box design for
each section of the
pipeline
Monolithic in nature
encouraging isolated
ecosystem of tools

7. • Context of the Project
• Simulation Management
•Sumatra and CoRR
• Benchmark Phase Field Problem
• Conclusion
Table of Contents

8. • Context of the Project
• Simulation Management
•Sumatra and CoRR
• Benchmark Phase Field Problem
• Environment and Examples
• Conclusion
Table of Contents
• Context of the Project
• Simulation Management
•Sumatra and CoRR
• Benchmark Phase Field Problem
• Conclusion
Table of Contents

9. U n i v e rs i t y LO G O
Sumatra and CoRR
- What is it good for?
1
- What are the limitations?

10. U n i v e rs i t y LO G O
Sumatra and CoRR
- What is it good for?
1
- What are the limitations?
- Autonomous
- Local and cloud storage
- Continuously recording
- Compatible
- click-and-run
2

11. Sumatra and CoRR
dt = 1
Equation = f()
while elapsed_time is less than desired_duration:
result1 = equation.solve(dt = dt, solver = LinearPCG)
result2 = equation.solve(dt = small_dt, solver = LinearPCG)
if result1 does not meet tolerance * result2:
decrease dt and solve again
else:
increase dt and solve again
Extract data

12. U n i v e rs i t y LO G O
Environment
Workflow
 Definition
 Jupyter Notebook aka iPython Notebook
 libraries
 GitHub
 Cluster

13. • Context of the Project
• Simulation Management
• Sumatra and CoRR
•Benchmark Phase Field
Problem
• Conclusion
Table of Contents

14. • Context of the Project
• Simulation Management
• Sumatra and CoRR
•Benchmark Phase Field
Problem
• Conclusion
Table of Contents

15. U n i v e rs i t y LO G O
Analysis – phase-field model
2 Test CoRR and Sumatra functionality
1 Performance evaluation
3 Results
1 Performance evaluation

16. U n i v e rs i t y LO G O
Analysis – phase-field model
Results

17. U n i v e rs i t y LO G O
Why is reproducibility a difficult task?
• Versions and updates
• Legality
• Hardware
• Python libraries and dependencies
• Time drain

18. U n i v e rs i t y LO G O
• Context of the Project
• Simulation Management
• Sumatra and CoRR
• Benchmark Phase Field Problem
•Conclusion
Table of Contents

19. U n i v e rs i t y LO G O
Conclusion
2
Problem: CHiMaD benchmark problem
Solution: CoRR
1 Could you reproduce our phase-field results?
3 More work to be done in both areas

20. U n i v e rs i t y LO G O
Acknowledgements
2 MML Thermodynamics and Kinetics group
1
Mentors
Daniel Wheeler, Ph.D
Faical Yannick P. Congo, Ph.D
3 Anushka Dasgupta
4 All who made NIST SURF possible