Download to read offline
![mymacbook:~/2015-‐02-‐01-‐myproject>
abyss-‐pe
k=25
reads.fastq.gz
zsh:
command
not
found:
abyss-‐pe
mymacbook:~/2015-‐02-‐01-‐myproject>
oswitch
-‐l
yeban/biolinux:8
ubuntu:14.04
ontouchstart/texlive-‐full
ipython/ipython
mymacbook:~/2015-‐02-‐01-‐myproject>
oswitch
yeban/biolinux
######
You
are
now
running:
biolinux
in
container
biolinux-‐7187.
######
biolinux-‐7187:~/2015-‐02-‐01-‐myproject>
abyss-‐pe
k=25
reads.fastq.gz
[...
just
works
on
your
files
where
they
are…]
biolinux-‐7187:~/2015-‐02-‐01-‐myproject>
exit
mymacbook:~/2015-‐02-‐01-‐myproject>
[...
output
is
where
you
expect
it
to
be
...]
https://github.com/yeban/oswitch
oSwitch
One-line access to other operating systems.](https://image.slidesharecdn.com/2015-03-25-cw15-150428051511-conversion-gate02/85/Sustainable-software-institute-Collaboration-workshop-15-320.jpg)
![Algorithm discovery by protein folding game players
Firas Khatiba
, Seth Cooperb
, Michael D. Tykaa
, Kefan Xub
, Ilya Makedonb
, Zoran Popovićb
,
David Bakera,c,1
, and Foldit Players
a
Department of Biochemistry; b
Department of Computer Science and Engineering; and c
Howard Hughes Medical Institute, University of Washington,
Box 357370, Seattle, WA 98195
Contributed by David Baker, October 5, 2011 (sent for review June 29, 2011)
Foldit is a multiplayer online game in which players collaborate
and compete to create accurate protein structure models. For spe-
cific hard problems, Foldit player solutions can in some cases out-
perform state-of-the-art computational methods. However, very
little is known about how collaborative gameplay produces these
results and whether Foldit player strategies can be formalized and
structured so that they can be used by computers. To determine
whether high performing player strategies could be collectively
codified, we augmented the Foldit gameplay mechanics with tools
for players to encode their folding strategies as “recipes” and to
share their recipes with other players, who are able to further mod-
ify and redistribute them. Here we describe the rapid social evolu-
tion of player-developed folding algorithms that took place in the
year following the introduction of these tools. Players developed
over 5,400 different recipes, both by creating new algorithms and
by modifying and recombining successful recipes developed by
other players. The most successful recipes rapidly spread through
the Foldit player population, and two of the recipes became parti-
cularly dominant. Examination of the algorithms encoded in these
two recipes revealed a striking similarity to an unpublished algo-
rithm developed by scientists over the same period. Benchmark
calculations show that the new algorithm independently discov-
ered by scientists and by Foldit players outperforms previously
published methods. Thus, online scientific game frameworks have
the potential not only to solve hard scientific problems, but also to
discover and formalize effective new strategies and algorithms.
citizen science ∣ crowd-sourcing ∣ optimization ∣ structure prediction ∣
strategy
Citizen science is an approach to leveraging natural human
abilities for scientific purposes. Most such efforts involve
visual tasks such as tagging images or locating image features
(1–3). In contrast, Foldit is a multiplayer online scientific discovery
game, in which players become highly skilled at creating accurate
protein structure models through extended game play (4, 5). Foldit
recruits online gamers to optimize the computed Rosetta energy
using human spatial problem-solving skills. Players manipulate
protein structures with a palette of interactive tools and manipula-
tions. Through their interactive exploration Foldit players also uti-
lize user-friendly versions of algorithms from the Rosetta structure
prediction methodology (6) such as wiggle (gradient-based energy
minimization) and shake (combinatorial side chain rotamer pack-
ing). The potential of gamers to solve more complex scientific pro-
blems was recently highlighted by the solution of a long-standing
protein structure determination problem by Foldit players (7).
One of the key strengths of game-based human problem ex-
ploration is the human ability to search over the space of possible
strategies and adapt those strategies to the type of problem and
stage of problem solving (5). The variability of tactics and
strategies stems from the individuality of each player as well as
multiple methods of sharing and evolution within the game
(group play, game chat), and outside of the game [wiki pages (8)].
One way to arrive at algorithmic methods underlying successful
human Foldit play would be to apply machine learning techniques
to the detailed logs of expert Foldit players (9). We chose instead
to rely on a superior learning machine: Foldit players themselves.
As the players themselves understand their strategies better than
anyone, we decided to allow them to codify their algorithms
directly, rather than attempting to automatically learn approxi-
mations. We augmented standard Foldit play with the ability to
create, edit, share, and rate gameplay macros, referred to as
“recipes” within the Foldit game (10). In the game each player
has their own “cookbook” of such recipes, from which they can
invoke a variety of interactive automated strategies. Players can
share recipes they write with the rest of the Foldit community or
they can choose to keep their creations to themselves.
In this paper we describe the quite unexpected evolution of
recipes in the year after they were released, and the striking con-
vergence of this very short evolution on an algorithm very similar
to an unpublished algorithm recently developed independently
by scientific experts that improves over previous methods.
Results
In the social development environment provided by Foldit,
players evolved a wide variety of recipes to codify their diverse
strategies to problem solving. During the three and a half month
study period (see Materials and Methods), 721 Foldit players ran
5,488 unique recipes 158,682 times and 568 players wrote 5,202
recipes. We studied these algorithms and found that they fell
into four main categories: (i) perturb and minimize, (ii) aggressive
rebuilding, (iii) local optimize, and (iv) set constraints. The first
category goes beyond the deterministic minimize function
provided to Foldit players, which has the disadvantage of readily
being trapped in local minima, by adding in perturbations to lead
the minimizer in different directions (11). The second category
uses the rebuild tool, which performs fragment insertion with
loop closure, to search different areas of conformation space;
these recipes are often run for long periods of time as they are
designed to rebuild entire regions of a protein rather than just
refining them (Fig. S1). The third category of recipes performs
local minimizations along the protein backbone in order to im-
prove the Rosetta energy for every segment of a protein. The final
category of recipes assigns constraints between beta strands or
pairs of residues (rubber bands), or changes the secondary struc-
ture assignment to guide subsequent optimization.
Different algorithms were used with very different frequencies
during the experiment. Some are designated by the authors as
public and are available for use by all Foldit players, whereas
others are private and available only to their creator or their
Foldit team. The distribution of recipe usage among different
players is shown in Fig. 1 for the 26 recipes that were run over
1,000 times. Some recipes, such as the one represented by the
leftmost bar, were used many times by many different players,
while others, such as the one represented by the pink bar in the
Author contributions: F.K., S.C., Z.P., and D.B. designed research; F.K., S.C., M.D.T., and
F.P. performed research; F.K., S.C., M.D.T., K.X., and I.M. analyzed data; and F.K., S.C., Z.P.,
and D.B. wrote the paper.
The authors declare no conflict of interest.
Freely available online through the PNAS open access option.
1
To whom correspondence should be addressed. E-mail: dabaker@u.washington.edu.
This article contains supporting information online at www.pnas.org/lookup/suppl/
doi:10.1073/pnas.1115898108/-/DCSupplemental.
BIOPHYSICSAND
COMPUTATIONALBIOLOGY
PSYCHOLOGICALAND
COGNITIVESCIENCES
http://Fold.it](https://image.slidesharecdn.com/2015-03-25-cw15-150428051511-conversion-gate02/85/Sustainable-software-institute-Collaboration-workshop-27-320.jpg)
Uploaded byYannick Wurm
Sustainable software institute Collaboration workshop
The document discusses tools for analyzing biological data. It summarizes four tools: 1. SequenceServer - A simple web interface for BLAST that handles formatting and installing BLAST locally. 2. oSwitch - Allows rapidly switching between operating systems and container environments to access specific bioinformatics software without installation. 3. GeneValidator - Helps curate gene predictions by identifying problematic predictions, choosing best alternative models, and aiding manual curation of individual genes. 4. Afra - A crowdsourcing platform that aims to crowdsource the visual inspection and correction of gene models by recruiting and training students, ensuring quality through tutorials, redundancy and senior review, and creating small, simple initial tasks.
More Related Content
Similar to Sustainable software institute Collaboration workshop
Sustainable software institute Collaboration workshop
- 1. @yannick__ http://wurmlab.github.io Opportunities to overcomehuman & computational challenges in biology. 2015-03-25-CW15
- 2. This changes everything.454 Illumina Solid... Any labcan sequence anything!
- 3. • Biology/life iscomplex • Field is young. • Biologists lack computational training. • Generally, analysis tools suck. • badly written • badly tested • hard to install • output quality… often questionable. • Understanding/visualizing/massaging data is hard. • Datasets continue to grow! Genomics is hard.
- 4.
- 5.
- 6. Four tools suck less. (hopefully) Fourtools that
- 7.
- 8. “Can you BLASTthis for me?” BLAST is the most commonly used tool: >100,000 citations But: •convoluted interface •challenging on custom data
- 9. http://www.sequenceserver.com/ If no configfile:Asks interactive setup questions. If needed: Downloads BLAST binaries If needed: Formats FASTA into BLAST database. 1. Installing gem install sequenceserver ### Launched SequenceServer at: http://0.0.0.0:4567 2. Launch sequenceserver Demo Anurag Priyam - @yeban
- 12.
- 13. Bioinformatics software challenges •Software hard to install 😖 • Central versions on cluster • “I need it now” - slow system administrators? → delays • Impossible to install? • AmazonVirtual Machine - copy there - run analysis - copy back→😕 • Consistent version for specific project? One project dozens of tools?
- 14. Miracle solution? Too complicated& too much copying back & forth
- 15. mymacbook:~/2015-‐02-‐01-‐myproject> abyss-‐pe k=25 reads.fastq.gz zsh: command not found: abyss-‐pe mymacbook:~/2015-‐02-‐01-‐myproject> oswitch -‐l yeban/biolinux:8 ubuntu:14.04 ontouchstart/texlive-‐full ipython/ipython mymacbook:~/2015-‐02-‐01-‐myproject> oswitch yeban/biolinux ###### You are now running: biolinux in container biolinux-‐7187. ###### biolinux-‐7187:~/2015-‐02-‐01-‐myproject> abyss-‐pe k=25 reads.fastq.gz [... just works on your files where they are…] biolinux-‐7187:~/2015-‐02-‐01-‐myproject> exit mymacbook:~/2015-‐02-‐01-‐myproject> [... output is where you expect it to be ...] https://github.com/yeban/oswitch oSwitch One-line access to other operating systems.
- 16. oSwitch One-line access toother operating systems. Things feel (largely) unchanged: • Current working directory is maintained. • User name, uid and gid are maintained. • Login shell (bash/zsh/fish) is maintained. • Home directory is maintained (thus all .dotfiles and config files are maintained). • read/write permissions are maintained. • Paths are maintained whenever possible.Thus volumes (external drives, NAS, USB) mounted on the host are available in the container at the same path. https://github.com/yeban/oswitch
- 17. Working with Genepredictions
- 18. Gene prediction Dozens ofsoftware algorithms: dozens of predictions 20% failure rate: •missing pieces •extra pieces •incorrect merging •incorrect splitting Visual inspection... and manual fixing required. 1 gene = 5 minutes to 3 days Yandell&Ence2013NRG GTTTTtACCTGTTTTtGAAAAGGTAATTTTCTTTAGATATATTATGTTGAATaTTAGGGTTTTTATAAAGAATGTGTATATTGUTTACAATATAAAAGACACAATTGCAAACTAGCATGATTGTAAACAATTGCTAAACGGATCAATATAAATTAAAATTGTAATATTAAGTATCAAACCGATAATTTTTA Evidence Consensus:
- 20.
- 21.
- 22.
- 23. GeneValidator Run on: ★whole geneset:identify most problematic predictions ★alternative models for a gene (choose best) ★individual genes (while manually curating)
- 24. 3.Afra: Crowdsourcing gene modelcuration
- 25. Gene prediction Dozens ofsoftware algorithms: dozens of predictions 20% failure rate: •missing pieces •extra pieces •incorrect merging •incorrect splitting Visual inspection... and manual fixing required. 1 gene = 20 minutes to 3 days 15,000 genes * 20 species = impossible. Yandell&Ence2013NRG TTTTtACCTGTTTTtGAAAAGGTAATTTTCTTTAGATATATACAGTTTGTAATaTTAGGTATTTTATAAACAGTGTGTATATTTCTTACAATATAAAAGACACAATTGCAAACTAGCATGATTGTAAACAATTGCTAAACGGATCAATATAAATTAAAATTGTAATATTAAGTATCAAACCGATAATTTTT Evidence Consensus:
- 27. Algorithm discovery byprotein folding game players Firas Khatiba , Seth Cooperb , Michael D. Tykaa , Kefan Xub , Ilya Makedonb , Zoran Popovićb , David Bakera,c,1 , and Foldit Players a Department of Biochemistry; b Department of Computer Science and Engineering; and c Howard Hughes Medical Institute, University of Washington, Box 357370, Seattle, WA 98195 Contributed by David Baker, October 5, 2011 (sent for review June 29, 2011) Foldit is a multiplayer online game in which players collaborate and compete to create accurate protein structure models. For spe- cific hard problems, Foldit player solutions can in some cases out- perform state-of-the-art computational methods. However, very little is known about how collaborative gameplay produces these results and whether Foldit player strategies can be formalized and structured so that they can be used by computers. To determine whether high performing player strategies could be collectively codified, we augmented the Foldit gameplay mechanics with tools for players to encode their folding strategies as “recipes” and to share their recipes with other players, who are able to further mod- ify and redistribute them. Here we describe the rapid social evolu- tion of player-developed folding algorithms that took place in the year following the introduction of these tools. Players developed over 5,400 different recipes, both by creating new algorithms and by modifying and recombining successful recipes developed by other players. The most successful recipes rapidly spread through the Foldit player population, and two of the recipes became parti- cularly dominant. Examination of the algorithms encoded in these two recipes revealed a striking similarity to an unpublished algo- rithm developed by scientists over the same period. Benchmark calculations show that the new algorithm independently discov- ered by scientists and by Foldit players outperforms previously published methods. Thus, online scientific game frameworks have the potential not only to solve hard scientific problems, but also to discover and formalize effective new strategies and algorithms. citizen science ∣ crowd-sourcing ∣ optimization ∣ structure prediction ∣ strategy Citizen science is an approach to leveraging natural human abilities for scientific purposes. Most such efforts involve visual tasks such as tagging images or locating image features (1–3). In contrast, Foldit is a multiplayer online scientific discovery game, in which players become highly skilled at creating accurate protein structure models through extended game play (4, 5). Foldit recruits online gamers to optimize the computed Rosetta energy using human spatial problem-solving skills. Players manipulate protein structures with a palette of interactive tools and manipula- tions. Through their interactive exploration Foldit players also uti- lize user-friendly versions of algorithms from the Rosetta structure prediction methodology (6) such as wiggle (gradient-based energy minimization) and shake (combinatorial side chain rotamer pack- ing). The potential of gamers to solve more complex scientific pro- blems was recently highlighted by the solution of a long-standing protein structure determination problem by Foldit players (7). One of the key strengths of game-based human problem ex- ploration is the human ability to search over the space of possible strategies and adapt those strategies to the type of problem and stage of problem solving (5). The variability of tactics and strategies stems from the individuality of each player as well as multiple methods of sharing and evolution within the game (group play, game chat), and outside of the game [wiki pages (8)]. One way to arrive at algorithmic methods underlying successful human Foldit play would be to apply machine learning techniques to the detailed logs of expert Foldit players (9). We chose instead to rely on a superior learning machine: Foldit players themselves. As the players themselves understand their strategies better than anyone, we decided to allow them to codify their algorithms directly, rather than attempting to automatically learn approxi- mations. We augmented standard Foldit play with the ability to create, edit, share, and rate gameplay macros, referred to as “recipes” within the Foldit game (10). In the game each player has their own “cookbook” of such recipes, from which they can invoke a variety of interactive automated strategies. Players can share recipes they write with the rest of the Foldit community or they can choose to keep their creations to themselves. In this paper we describe the quite unexpected evolution of recipes in the year after they were released, and the striking con- vergence of this very short evolution on an algorithm very similar to an unpublished algorithm recently developed independently by scientific experts that improves over previous methods. Results In the social development environment provided by Foldit, players evolved a wide variety of recipes to codify their diverse strategies to problem solving. During the three and a half month study period (see Materials and Methods), 721 Foldit players ran 5,488 unique recipes 158,682 times and 568 players wrote 5,202 recipes. We studied these algorithms and found that they fell into four main categories: (i) perturb and minimize, (ii) aggressive rebuilding, (iii) local optimize, and (iv) set constraints. The first category goes beyond the deterministic minimize function provided to Foldit players, which has the disadvantage of readily being trapped in local minima, by adding in perturbations to lead the minimizer in different directions (11). The second category uses the rebuild tool, which performs fragment insertion with loop closure, to search different areas of conformation space; these recipes are often run for long periods of time as they are designed to rebuild entire regions of a protein rather than just refining them (Fig. S1). The third category of recipes performs local minimizations along the protein backbone in order to im- prove the Rosetta energy for every segment of a protein. The final category of recipes assigns constraints between beta strands or pairs of residues (rubber bands), or changes the secondary struc- ture assignment to guide subsequent optimization. Different algorithms were used with very different frequencies during the experiment. Some are designated by the authors as public and are available for use by all Foldit players, whereas others are private and available only to their creator or their Foldit team. The distribution of recipe usage among different players is shown in Fig. 1 for the 26 recipes that were run over 1,000 times. Some recipes, such as the one represented by the leftmost bar, were used many times by many different players, while others, such as the one represented by the pink bar in the Author contributions: F.K., S.C., Z.P., and D.B. designed research; F.K., S.C., M.D.T., and F.P. performed research; F.K., S.C., M.D.T., K.X., and I.M. analyzed data; and F.K., S.C., Z.P., and D.B. wrote the paper. The authors declare no conflict of interest. Freely available online through the PNAS open access option. 1 To whom correspondence should be addressed. E-mail: dabaker@u.washington.edu. This article contains supporting information online at www.pnas.org/lookup/suppl/ doi:10.1073/pnas.1115898108/-/DCSupplemental. BIOPHYSICSAND COMPUTATIONALBIOLOGY PSYCHOLOGICALAND COGNITIVESCIENCES http://Fold.it
- 28. • Recruiting &retaining contributors Crowd-sourcing the visual inspection + correction of gene models. Challenges
- 29. Recruiting & retainingcontributors Plan A: get students. • Increase accessibility: • Make tasks small & simple • Need excellent tutorials & training • Need an intelligent “mothering” user interface. • Provide rewards: • Better grades • Learning experience • Good karma (helping science) • Prestige & pride (on facebook; points & badges “leaderboard”, with certificates, in publications) • Opportunities to develop expertise & responsibilities
- 30. Crowd-sourcing the visualinspection + correction of gene models. Challenges • Recruiting & retaining contributors • Ensuring quality
- 31. Ensuring quality • Excellenttutorials/training • Make tasks small & simple • Redundancy • Review of conflicts by senior users. Begin Needs curation Create initial tasks Being curated Curate Being curated Curate Being curated Curate Submit Submit Submit Auto-check Done Inconsistent: create “review” task Consistent: create next required task
- 32. Crowd-sourcing the visualinspection + correction. Challenges http://afra.sbcs.qmul.ac.ukAnurag Priyam http://github.com/yeban/afra • Recruiting & retaining contributors • Ensuring quality
- 33.
- 37. Timelines • Rolled outto: • 8 MSc students • 20 3rd year students • Need to improve tutorials/guidance/documentation/outputs • Roll out to 200 first years (autumn) • Expand
- 38. 5. Bionode… staytuned.
- 39. Thanks! y.wurm@qmul.ac.uk @yannick__ http://wurmlab.github.io Colleagues & Collaborators @QMUL & UNIL Anurag Priyam @yeban Monica Dragan Ismail Moghul Vivek Rai Bruno Vieira @bmpvieira
- 40. Minimally guided demo:) 1. Set up a custom BLAST server (Sequenceserver) 2. Set up oswitch to rapidly switch to yeban/biolinux @yannick__