The document discusses the development of collaborative workflows for scientific data management using tools like Dat and Bionode. It addresses challenges in bioinformatics workflows, such as reproducibility and data accessibility, and emphasizes the use of Node.js for building scalable and efficient pipelines. The speaker highlights the importance of open-source tools and community collaboration in advancing scientific research.

1. Building
collaborative
workflows for
scientific data
bmpvieira.com/orcambridge14

2. |
Phd Student @
Bioinformatics and
Population Genomics
Supervisor:
Yannick Wurm |
Before:
Bruno Vieira @bmpvieira
@yannick__
© 2014 Bruno Vieira CC-BY 4.0

3. Sequencing cost drops

4. Sequencing data rises

5. Goodbye Excel/Windows

6. Hello command line

7. Hello super computers

8. Programming

9. Programming

10. Programming

11. Programming

12. Programming

13. Reproducibility crisis

14. Losing data

15. Reproducibility layers
Code
Data
Workflow
Environment

16. Code

17. The GitHub for Science...
is GitHub!

18. Code as a research output

19. Reproducibility layers
Code
Data
Workflow
Environment

20. Data

21. Dat
open source tool for sharing and
collaborating on data
started august '13, we are grant funded
and 100% open source
public
on freenode
dat-data.com
#dat
gitter.im/datproject/discussions
Dat Community Call #1

23. Dat - "git for data"
npm install -g dat
dat init
collect-data | dat import
dat listen

25. Dat
dat clone
dat pull --live
dat blobs put mygenome data.fasta
dat cat | transform
dat cat | docker run -i transform
http://eukaryota.dathub.org

26. Dat
Planned
dat checkout revision
dat diff
dat branch
multi master replication
sync to databases
registry

27. Data stored locally in leveldb, but can use
other backends such as
Postgres
Redis
etc
Files stored in blob-stores
s3
local-fs
bitorrent
ftp
etc

28. Dat features
auto schema generation
free REST API
all APIs are streaming

29. Dat workshop
maxogden.github.io/get-dat

30. Dat quick deploy
github.com/bmpvieira/heroku-dat-template

31. Reproducibility layers
Code
Data
Workflow
Environment

32. Workflow

33. Bionode
open source project for modular and
universal bioinformatics
started january '14
bionode.io

34. Some problems I faced
during my research:
Difficulty getting relevant descriptions and
datasets from NCBI API using bio* libs
For web projects, needed to implement
the same functionality on browser and
server
Difficulty writing scalable, reproducible
and complex bioinformatic pipelines

35. Bionode also collaborates with BioJS

36. Bionode
npm install -g bionode
bionode ncbi download gff bacteria
bionode ncbi download sra arthropoda |
bionode sra fastq-dump
npm install -g bionode-ncbi
bionode-ncbi search assembly formicidae |
dat import --json

37. Bionode - list of modules
Name Type Status People
Data
access
status production
Parser status production
Wrangling status production
Data
access
status production
Parser status production
ncbi
fasta
seq IM
ensembl
blast-
parser

38. Bionode - list of modules
Name Type Status People
Documentation status production
Documentation status production
Documentation status production
Documentation status production
template
JS pipeline
Gasket
pipeline
Dat/Bionode
workshop

39. Bionode - list of modules
Name Type Status People
Wrappers status development
Wrappers status development
Wrappers status development
Parser status development
sra
bwa
sam
bbi

40. Bionode - list of modules
status request
Name Type People
Data access
Data access
Parser
Parser
Wrappers
Wrappers
Wrappers
ebi
semantic
vcf
gff
bowtie
sge badryan
blast

41. Bionode - list of modules
Name Type People
Wrappers
Wrappers
Wrappers
Wrappers
Wrappers
Wrappers
vsearch
khmer
rsem
gmap
star
go badryan

42. Bionode - Why wrappers?
Same interface between modules
(Streams and NDJSON)
Easy installation with NPM
Semantic versioning
Add tests
Abstract complexity / More user friendly

43. Bionode - Why Node.js?
Same code client/server side

44. Need to reimplement the same code on
browser and server.
Solution: JavaScript everywhere
->
-> ,
->
->
Afra bionode-seq
GeneValidator seq fasta
SequenceServer
BioJS collaborating for code reuse
Biodalliance converting to bionode

45. Bionode - Why Node.js?

46. Reusable, small and tested
modules

47. Benefit from other JS
projects
Dat BioJS NoFlo

51. Difficulty getting relevant description and
datasets from NCBI API using bio* libs
Python example: URL for the Achromyrmex
assembly?
Solution:
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA_000188075.1_Si_gnG
import xml.etree.ElementTree as ET
from Bio import Entrez
Entrez.email = "mail@bmpvieira.com"
esearch_handle = Entrez.esearch(db="assembly", term="Achromyrmex")
esearch_record = Entrez.read(esearch_handle)
for id in esearch_record['IdList']:
esummary_handle = Entrez.esummary(db="assembly", id=id)
esummary_record = Entrez.read(esummary_handle)
documentSummarySet = esummary_record['DocumentSummarySet']
document = documentSummarySet['DocumentSummary'][0]
metadata_XML = document['Meta'].encode('utf-8')
metadata = ET.fromstring('' + metadata_XML + '')
for entry in Metadata[1]:
print entry.text
bionode-ncbi

52. Difficulty getting relevant description and
datasets from NCBI API using bio* libs
Example: URL for the Achromyrmex
assembly?
JavaScript
http://ftp.ncbi.nlm.nih.gov/genomes/all/GCA_000204515.1_Aech_3.9/GCA_00020
4515.1_Aech_3.9_genomic.fna.gz
var bio = require('bionode')
bio.ncbi.urls('assembly', 'Acromyrmex', function(urls) {
console.log(urls[0].genomic.fna)
})
bio.ncbi.urls('assembly', 'Acromyrmex').on('data', printGenomeURL)
function printGenomeURL(urls) {
console.log(urls[0].genomic.fna)
})

53. Difficulty getting relevant description and
datasets from NCBI API using bio* libs
Example: URL for the Achromyrmex
assembly?
JavaScript
BASH
http://ftp.ncbi.nlm.nih.gov/genomes/all/GCA_000204515.1_Aech_3.9/GCA_00020
4515.1_Aech_3.9_genomic.fna.gz
var ncbi = require('bionode-ncbi')
var ndjson = require('ndjson')
ncbi.urls('assembly', 'Acromyrmex')
.pipe(ndjson.stringify())
.pipe(process.stdout)
bionode-ncbi urls assembly Acromyrmex |
tool-stream extractProperty genomic.fna

54. Difficulty writing scalable, reproducible and
complex bioinformatic pipelines.
Solution: Node.js Streams everywhere
var ncbi = require('bionode-ncbi')
var tool = require('tool-stream')
var through = require('through2')
var fork1 = through.obj()
var fork2 = through.obj()

55. Difficulty writing scalable, reproducible and
complex bioinformatic pipelines.
Solution: Node.js Streams everywhere
ncbi
.search('sra', 'Solenopsis invicta')
.pipe(fork1)
.pipe(dat.reads)
fork1
.pipe(tool.extractProperty('expxml.Biosample.id'))
.pipe(ncbi.search('biosample'))
.pipe(dat.samples)
fork1
.pipe(tool.extractProperty('uid'))
.pipe(ncbi.link('sra', 'pubmed'))
.pipe(ncbi.search('pubmed'))
.pipe(fork2)
.pipe(dat.papers)

58. Difficulty writing scalable, reproducible and
complex bioinformatic pipelines.
bionode-ncbi search genome Guillardia theta |
tool-stream extractProperty assemblyid |
bionode-ncbi download assembly |
tool-stream collectMatch status completed |
tool-stream extractProperty uid|
bionode-ncbi link assembly bioproject |
tool-stream extractProperty destUID |
bionode-ncbi link bioproject sra |
tool-stream extractProperty destUID |
bionode-ncbi download sra |
bionode-sra fastq-dump |
tool-stream extractProperty destFile |
bionode-bwa mem 503988/GCA_000315625.1_Guith1_genomic.fna.gz |
tool-stream collectMatch status finished|
tool-stream extractProperty sam|
bionode-sam

59. Difficulty writing scalable, reproducible and
complex bioinformatic pipelines.
bionode-example-dat-gasket
get-dat workshop
get-dat bionode gasket example

60. Difficulty writing scalable, reproducible and
complex bioinformatic pipelines.
{
"import-data": [
"bionode-ncbi search genome eukaryota",
"dat import --json --primary=uid"
],
"search-ncbi": [
"dat cat",
"grep Guillardia",
"tool-stream extractProperty assemblyid",
"bionode-ncbi download assembly -",
"tool-stream collectMatch status completed",
"tool-stream extractProperty uid",
"bionode-ncbi link assembly bioproject -",
"tool-stream extractProperty destUID",
"bionode-ncbi link bioproject sra -",
"tool-stream extractProperty destUID",
"grep 35526",
"bionode-ncbi download sra -",
"tool-stream collectMatch status completed",
"tee > metadata.json"
],

61. Difficulty writing scalable, reproducible and
complex bioinformatic pipelines.
"index-and-align": [
"cat metadata.json",
"bionode-sra fastq-dump -",
"tool-stream extractProperty destFile",
"bionode-bwa mem **/*fna.gz"
],
"convert-to-bam": [
"bionode-sam 35526/SRR070675.sam"
]
}

62. Difficulty writing scalable, reproducible and
complex bioinformatic pipelines.
datscript
pipeline main
run pipeline import
pipeline import
run foobar | run dat import --json
bmpvieira example
ekg example

63. Reproducibility layers
Code
Data
Workflow
Environment

64. Environment

65. Docker for reproducible
science
docker run bmpvieira/thesis

66. - Modular and universal bioinformatics
Pipeable UNIX command line tools and
JavaScript / Node.js APIs for bioinformatic
analysis workflows on the server and browser.
- Build data pipelines
Provides a streaming interface between every file
format and data storage backend. "git for data"
Bionode.io
#bionode
gitter.im/bionode/bionode
Dat-data.com
#dat
gitter.im/datproject/discussions

67. Acknowledgements
@yannick__
@maxogden
@mafintosh
@erikgarrison
@QM_SBCS
@opendata
Bionode contributors

68. Thanks!
"Science should work as an
Open Source project"
dat-data.com
bionode.io