2013 10-30-sbc361-reproducible designsandsustainablesoftware


Published on

Queen Mary U London SBC361
Experimental Design
Reproducible Research
Sustainable Software

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2013 10-30-sbc361-reproducible designsandsustainablesoftware

  1. 1. Programming in R Quick refresher
  2. 2. • creating a vector • three synonyms: > myvector > myvector > myvector > myvector [1] 5 6 <- 5:11 <- seq(from=5, to=11, by=1) <- c(5, 6, 7, 8, 9, 10, 11) 7 8 9 10 11 • accessing a subset • of a vector > bigvector <- 150:100 > bigvector [1] 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 1 [20] 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 1 [39] 112 111 110 109 108 107 106 105 104 103 102 101 100 > mysubset <- bigvector[myvector] > mysubset [1] 146 145 144 143 142 141 140 > subset(bigvector, bigvector > 120) [1] 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 1 [20] 131 130 129 128 127 126 125 124 123 122 121
  3. 3. Regular expressions: Text search on steroids. Regular expression David Dav(e|id) Dav(e|id|ide|o) At{1,2}enborough Atte[nm]borough At{1,2}[ei][nm]bo{0,1}ro(ugh){0,1} Finds David David, Dave David, Dave, Davide, Davo Attenborough, Atenborough Attenborough, Attemborough Atimbro, attenbrough, etc. Easy counting, replacing all with “Sir David Attenborough”
  4. 4. • for subsetting/counting: grep() • for replacing: gsub()
  5. 5. Functions •R has many. e.g.: plot(), t.test() • Making your own: tree_age_estimate <- function(diameter, species) { [...do the magic... # maybe something like: growth.rate <- growth.rates[ species ] age.estimate <- diameter / growth.rate ...] return(age.estimate) } > + > + tree_age_estimate(25, "White Oak") 66 tree_age_estimate(60, "Carya ovata") 190
  6. 6. “for” Loop > possible_colours <- c('blue', 'cyan', 'sky-blue', 'navy blue', 'steel blue', 'royal blue', 'slate blue', 'light blue', 'dark blue', 'prussian blue', 'indigo', 'baby blue', 'electric blue') > possible_colours [1] "blue" "cyan" "sky-blue" [5] "steel blue" "royal blue" "slate blue" [9] "dark blue" "prussian blue" "indigo" [13] "electric blue" > for (colour in possible_colours) { + print(paste("The sky is oh so, so", colour)) + } [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] "The "The "The "The "The "The "The "The "The "The "The "The sky sky sky sky sky sky sky sky sky sky sky sky is is is is is is is is is is is is so, so, so, so, so, so, so, so, so, so, so, so, oh oh oh oh oh oh oh oh oh oh oh oh so so so so so so so so so so so so blue" cyan" sky-blue" navy blue" steel blue" royal blue" slate blue" light blue" dark blue" prussian blue" indigo" baby blue" "navy blue" "light blue" "baby blue"
  7. 7. Experimental design Reproducible research & Scientific computing.
  8. 8. Why consider experimental design? • If you’re performing experiments • Cost • Time • for experiment • for analysis • Ethics • If you’re deciding to fund? to buy? to approve? to compete? • are the results real? • can you trust the data?
  9. 9. Main potential problems • Insufficient data/power • Inappropriate statistics • Pseudoreplication • Confounding factors Inaccurate & Misleading Wrong
  10. 10. Example: deer parasites • Do red deer that feed in woodland have more parasites than deer that feed on moorland? • Find a woodland + a highland; collect faecal samples from 20 deer in each. • Conclusion? • But: • pseudoreplication: (n = 1 not 20!): • shared environment (influence each other) • relatedness • many confounding factors: (e.g. altitude...)
  11. 11. Your turn: small & big Pheidole workers. • Is there a genetic predisposition for becoming a larger worker? • Design an experiment alone. • Exchange ideas with your neighbor.
  12. 12. e.g.: John.
  13. 13. Your turn again: protein production • Large amounts of potential superdrug takeItEasyProtein™ required for Phase II trials. • 10 cell lines can produce takeItEasyProtein™. • You have 5 possible growth media. • Optimization question: Which combination of temperature, cell line, and growth medium will perform best? • Constraints: • each assay takes 4 days. • access to 2 incubators (each can contain 1-100 growth tubes). • large scale production starts in 2 weeks • Design an experiment alone. • Exchange ideas with your neighbor.
  14. 14. Reproducible Research & Scientific Computing
  15. 15. Why care?
  16. 16. Some sources of inspiration
  17. 17. (steve@practicalcomputing.org),†† University of Wisconsin (khuff@cae.w Mary University of London (mark.plumbley@eecs.qmul.ac.uk),¶¶ Unive University D.A. Aruliah † , C. Titus Brown ‡ , Neil P. ChueUniversityDavisWisconsin Guy , (ethan@weecology.org), and ††† Hong § , Matt of ¶ , Richard T. (wils ∗ Greg Wilson , Best Practices for Scientific Computing Steven H.D. Haddock ∗∗ , Katy Huff †† , Ian M. Mitchell ‡‡ , Mark D. Plumbley §§ , Ben Waugh ¶¶ , Ethan P. White ∗∗∗ , Paul Wilson ††† Software Carpentry (gvwilson@software-carpentry.org),† University of Ontario Institute of Technology (Dhavide.Aru State University (ctb@msu.edu),§ Software Sustainability Institute (N.ChueHong@epcc.ed.ac.uk),¶ Space Telescope (mrdavis@stsci.edu), University of Toronto (guy@cs.utoronto.ca),∗∗ Monterey Bay Aquarium Research Institute (steve@practicalcomputing.org),†† University of Wisconsin (khuff@cae.wisc.edu),‡‡ University of British Columbia (mi Mary University of London (mark.plumbley@eecs.qmul.ac.uk),¶¶ University College London (b.waugh@ucl.ac.uk),∗∗ University (ethan@weecology.org), and ††† University of Wisconsin (wilsonp@engr.wisc.edu) ∗ arXiv:1210.0530v3 [cs.MS] 29 Nov 2012 Scientists spend an increasing amount of time building and using a software. However, most scientists are never taught how to do this i efficiently. As a result, many are unaware of tools and practices that d would allow them to write more reliable and maintainable code with p less effort. We describe a set of best practices for scientific software m Scientists spend an increasing amount of time building and using research and software development [61 and open source experience, development that have solid foundations in ical studies of scientific computing [4, 31, software. However, most scientists are never taught how to do this e efficiently. As a improve are unaware of tools and practices thatand the reliability of their and that result, many scientists’ productivity development in general (summarized in would allow them to write more reliable and maintainable code with software. describe a set of best practices for scientific software practices will guarantee efficient, error-frt less effort. We ment, but used in concert they will red f development that have solid foundations in research and experience, and that improve scientists’ productivitypeople, reliability of their and the not computers. errors in scientific software, make it easie 1. Write programs for the authors of the software time and effo software. Software is as important to modern focusing on the underlying scientific ques scientific research as 2. Automate repetitive tasks. 3. Use important to tubes. From groups the test modern scientific research telescopesasand computer to record history. as that work exclusively Software is 1 telescopes andMaketubes. From groups that work exclusively test incremental changes. 4. on computationalto traditional laboratory and field 1. laboratory andpeople, not c problems, to traditional Write programs for field on computational problems, control. 5. Use version Scientists writing software need to writeS scientists, more and more of the daily operation of science re- operation of science rescientists, more and more of the daily cutes correctly and can be easily read and 6. computers. This includes the development of volves aroundDon’t repeat yourself (or others). c programmers (especially the author’s fut volves 7. Plan for mistakes. around computers. This includes the development of new algorithms, managing and analyzing the large amounts cannot be easily read and understood it is p of data algorithms, managing andworksand that are generated in single research projects, correctly.the large amounts new 8. Optimize software only after it analyzingknow that it is actually doing what it i to combining disparate datasets to assess synthetic problems. c 9. Document the designown software single research projects, and must t and purpose ofthese rather than itssoftware developers code be productive, mechanics. of Scientists that are generated in for data typically develop their aspects of human cognition into account t 10. Conduct requires substantial domain-specific purposes because doing so code reviews. human working memory is limited, huma
  18. 18. Education A Quick Guide to Organizing Computational Biology Projects William Stafford Noble1,2* 1 Department of Genome Sciences, School of Medicine, University of Washington, Seattle, Washington, United States of America, 2 Department of Computer Science and Engineering, University of Washington, Seattle, Washington, United States of America Introduction under a common root directory. The understanding your work or who may be exception to this rule is source code or evaluating your research skills. Most comMost bioinformatics coursework focusscripts that are used in multiple projects. monly, however, that ‘‘someone’’ is you. A es on algorithms, with perhaps some Each such program might have a project few months from now, you may not components devoted to learning prodirectory of its own. remember what you were up to when you gramming skills and learning how to Within a given project, I use a top-level created a particular set of files, or you may use existing bioinformatics software. Unorganization that is logical, with chrononot remember what conclusions you drew. fortunately, for students who are preparlogical organization at the next level, and You will either have to then spend time ing for a research career, this type of logical organization below that. A sample reconstructing your previous experiments curriculum fails to address many of the project, called msms, is shown in Figure 1. or lose whatever insights you gained from day-to-day organizational challenges asAt the root of most of my projects, I have a those experiments. sociated with performing computational data directory for storing fixed data sets, a This leads to the second principle, experiments. In practice, the principles results directory for tracking computawhich is actually more like a version of Figure names are typeface, and filenames are behind organizing and documenting 1. Directory structure for a sample project. Directorydo, youin large tional experiments in smaller typeface. Only a subset of Murphy’s that the dates are formatted ,year.-,month.-,day. so that they can bepeformed on that data, the files are shown here. NoteLaw: Everything you sorted in chronological order. The computational experiments are often code src/ms-analysis.c have to to do over again. and is documented in doc/ms-analysis.html. The README source is compiled create bin/ms-analysis a doc directory with one subdirectory per will probably files in what date. The driver script results/2009-01-15/runall learned on the fly, and this learning is the data directories specify who downloaded the data files from what URL on manuscript, and directories such as src automatically Inevitably, you will discover some flaw split3, corresponding to three cross-validation splits. The bin/parsegenerates the three subdirectories split1, split2, and in sqt.py strongly influenced by personal predilec- script is called by bothpreparation driverthe data being for source code and bin for compiled your initial of the runall of scripts. doi:10.1371/journal.pcbi.1000424.g001 tions as well as by chance interactions binaries or scripts. analyzed, or you will get access to new with collaborators or colleagues. Within the data and results a complete data, the distinction be- The your paramThese types of entries provide directowith this approach,or you will decide that Lab Notebook The purpose of this article is to describe data and results may of a particular model was not picture of the development a similar, tween not be useful. ries, it is often tempting to apply of the project eterization In parallel with this chronological over time. Instead, could one good strategy for carrying out com- onebroad imagine a top-level means structure,the find itlogical toorganization. For example, you enough. This directory that I useful directory called something like experiIn practice, I ask members of my putational experiments. I will not describe , with subdirectories with names like last week, chronologically organizedhave two or group to data sets notebooks maintain a or even may lab research three put their lab against ments experiment you did notebook. This is a document that resides 2008-12-19. Optionally, the directory profound issues such as how to formulate which plan to password protection if the set of experiments you’veroot of the results directory andyou online, behind benchmark your in the been workname also include a or two necessary. When I meet with a member hypotheses, design experiments, or draw might ing on over word past month, will probably that records your progress algorithms, ofso lab or a could team, we can one in detail. indicating the topic of the the experiment my you project create refer Entries in the notebook conclusions. Rather, I will focus therein. In practice,to single experiment you have organized should be dated, for each of lab notebook, focusing on on directory need a be redone. If and they should be relatively verbose, with to the online them under data. will often require more than one day of the current entry but scrolling up to relatively mundane issues such as organizthis and documented your work clearly, thenimages In my experience, entries approach is risky, links or embedded or tables work, and so you may end up working a previous as necessary. The URL ing files and directories and documenting or repeating creating a new displaying the results of the experiments the can also be provided toof yourcollabobecause logical structure remote final few days more before the experiment with the new In each results folder: •script getResults.rb or WHATIDID.txt or MyAnalysis.Rnw •intermediates •output
  19. 19. Take notes in Markdown “compile” to html, pdf,
  20. 20. knitr (sweave)Analyzing & Reporting in a single file. MyFile.Rnw documentclass{article} usepackage[sc]{mathpazo} usepackage[T1]{fontenc} usepackage{url} begin{document} Also works with Markdown instead of LaTeX! ### in R: library(knitr) knit(“MyFile.Rnw”) # --> creates MyFile.tex <<setup, include=FALSE, cache=FALSE, echo=FALSE>>= # this is equivalent to SweaveOpts{...} opts_chunk$set(fig.path='figure/minimal-', fig.align='center', fig.show='hold') options(replace.assign=TRUE,width=90) @ title{A Minimal Demo of knitr} ### in shell: pdflatex MyFile.tex # --> creates MyFile.pdf author{Yihui Xie} A Minimal Demo of knitr maketitle You can test if textbf{knitr} works with this minimal demo. OK, let's get started with some boring random numbers: Yihui Xie February 26, 2012 <<boring-random,echo=TRUE,cache=TRUE>>= set.seed(1121) (x=rnorm(20)) mean(x);var(x) @ You can test if knitr works with this minimal demo. OK, let’s get started with s numbers: The first element of texttt{x} is Sexpr{x[1]}. Boring boxplots and histograms recorded by the PDF device: set.seed(1121) (x <- rnorm(20)) <<boring-plots,cache=TRUE,echo=TRUE>>= ## two plots side by side par(mar=c(4,4,.1,.1),cex.lab=.95,cex.axis=.9,mgp=c(2,.7,0),tcl=-.3,las=1) boxplot(x) hist(x,main='') @ Do the above chunks work? You should be able to compile the TeX{} ## [1] 0.14496 0.43832 ## [10] -0.02531 0.15088 ## [19] 0.13272 -0.15594 mean(x) ## [1] 0.3217 var(x) 0.15319 0.11008 1.08494 1.99954 -0.81188 1.35968 -0.32699 -0.71638 0.16027 1.80977 0 0
  21. 21. Choosing a programming language Excel R Unix command-line (i.e., shell, i.e., bash) Perl Java Python Ruby Javascript
  22. 22. Ruby. “Friends don’t let friends do Perl” - reddit user example: reverse the contents of each line in a file ### in PERL: open INFILE, "my_file.txt"; while (defined ($line = <INFILE>)) { chomp($line); @letters = split(//, $line); @reverse_letters = reverse(@letters); $reverse_string = join("", @reverse_letters); print $reverse_string, "n"; } ### in Ruby: File.open("my_file.txt").each do |line| puts line.chomp.reverse end
  23. 23. More ruby examples. 5.times do puts "Hello world" end # Sorting people people_sorted_by_age = people.sort_by{ |person| person.age}
  24. 24. Getting help. • In real life: Make friends with people. Talk to them. • Online: • Specific discussion mailing lists (e.g.: R, Stacks, bioruby, MAKER...) • Programming: http://stackoverflow.com • Bioinformatics: http://www.biostars.org • Sequencing-related: http://seqanswers.com • Stats: http://stats.stackexchange.com
  25. 25. • Online reputation is good: • forums • “citizen science”