SlideShare a Scribd company logo
BioMake
Ian Holmes & Christopher Mungall
UC Berkeley,
Berkeley Lab
http://biomake.org/
Bioinformatics analysis pipelines involves
chains of dependencies
GNU make
CC = gcc
%.o: %.c
$(CC) -c -o $@ $<
Rules, patterns, variables, automatic variables
Compiling a C program
Makefile
GNU make
BWAMEM = bwa
REF = ref.fasta
%.sam: %.fastq
$(BWA) mem $(REF) $< >$@
Aligning reads
Rules, patterns, variables, automatic variables
Makefile
Dependencies
BWAMEM = bwa
REF = ref.fasta
%.sam: %.fastq $(REF).bwt
$(BWA) mem $(REF) $< >$@
%.bwt: %
$(BWA) index $<
Timestamps, dependency graphs
Functions
MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
MAKEFILE_DIR := $(dir $(MAKEFILE_PATH))
test1:
echo $(shell ls $(MAKEFILE_DIR))
FUNC = echo $1 is $2
test2:
$(call FUNC,make,cool)
Special variables, functions, user-defined functions
Issues with GNU make
• Only one wildcard per rule
• Poor support for parallelism
• Timestamps are fragile, especially on NFS
• Can’t extend build logic, e.g. add qualifiers to rules
Biomake fixes these!
Multiple wildcards per rule
$X-$Y.sam: $X.fa.bwt $Y.fastq
bwa mem $X.fa $Y.fastq >$@
biomake myref-myreads.sam
MD5 signatures
$X-$Y.sam: $X.fa.bwt $Y.fastq
bwa mem $X.fa $Y.fastq >$@
biomake --md5-hash myref-myreads.sam
md5_hash("myref-myreads.sam",12,"0f723ae7f9bf07744445e93ac5595156").
md5_valid("myref-myreads.sam",12,"0f723ae7f9bf07744445e93ac5595156",X) :-
md5_check("myref.fa.bwt",6,"b1946ac92492d2347c6235b4d2611184",X),
md5_check("myreads.fastq",6,"591785b794601e212b260e25925636fd",X).
.biomake/md5/myref-myreads.sam
Multiple queue engines
$X-$Y.sam: $X.fa.bwt $Y.fastq
bwa mem $X.fa $Y.fastq >$@
biomake -Q sge myref-myreads.sam
biomake -Q slurm myref-myreads.sam
biomake -Q pbs myref-myreads.sam
biomake -Q poolq myref-myreads.sam
Logic extensions
$X-$Y.sam: $X.fa.bwt $Y { size_file(Y,S),
S < 1000000000 }
bwa mem $X.fa $Y >$@
Can embed Prolog in Makefiles,
or auto-translate entire Makefile to Prolog
(Some of) The Competition
Language
GNU-
compatible?
MD5? Clusters?
Erlang make Erlang No No No
omake OCaML Somewhat Yes No
makepp Perl Yes Yes No
qmake C Yes No SGE
What Biomake is not(yet)
• A massively parallel data analysis framework
(try Apache Spark)
• A heavy-duty cloud-oriented workflow language
(try Common Workflow Language)
• A web application for managing jobs
(try Galaxy)
What Biomake is
A simple, drop-in replacement for GNU make that
allows you to…
• ramp up your Makefile-driven workflows to cluster
scale
• avoid costly rebuilds triggered by file copying &
unsynchronized clocks
• extend rules with wildcards and logical tests
http://www.biorxiv.org/content/early/2016/12/12/093245
https://github.com/evoldoers/biomake
http://biomake.org/
• Ian Holmes
• Contributors & testers
• Mahesh Panchal
• Markus Triska
• Jan Wielemaker - creator of SWI-Prolog

More Related Content

What's hot

Introduction To Lamp
Introduction To LampIntroduction To Lamp
Introduction To Lamp
Amzad Hossain
 
RESTful web services
RESTful web servicesRESTful web services
RESTful web services
Tudor Constantin
 
DrupalCafe4 Kiev Services
DrupalCafe4 Kiev ServicesDrupalCafe4 Kiev Services
DrupalCafe4 Kiev Services
Yuriy Gerasimov
 
Otimizando Aplicações em Rails
Otimizando Aplicações em RailsOtimizando Aplicações em Rails
Otimizando Aplicações em Rails
Juan Maiz
 
Slim RedBeanPHP and Knockout
Slim RedBeanPHP and KnockoutSlim RedBeanPHP and Knockout
Slim RedBeanPHP and Knockout
Vic Metcalfe
 
Perl Dancer for Python programmers
Perl Dancer for Python programmersPerl Dancer for Python programmers
Perl Dancer for Python programmers
xSawyer
 
Distributed work with Gearman
Distributed work with GearmanDistributed work with Gearman
Distributed work with Gearman
Dominik Jungowski
 
Object Oriented JavaScript
Object Oriented JavaScriptObject Oriented JavaScript
Object Oriented JavaScript
techwhizbang
 
chef loves windows
chef loves windowschef loves windows
chef loves windows
Mat Schaffer
 
Modern Perl Web Development with Dancer
Modern Perl Web Development with DancerModern Perl Web Development with Dancer
Modern Perl Web Development with Dancer
Dave Cross
 
Keeping it small - Getting to know the Slim PHP micro framework
Keeping it small - Getting to know the Slim PHP micro frameworkKeeping it small - Getting to know the Slim PHP micro framework
Keeping it small - Getting to know the Slim PHP micro framework
Jeremy Kendall
 
PerlDancer for Perlers (FOSDEM 2011)
PerlDancer for Perlers (FOSDEM 2011)PerlDancer for Perlers (FOSDEM 2011)
PerlDancer for Perlers (FOSDEM 2011)
xSawyer
 
Loadrunner
LoadrunnerLoadrunner
Loadrunner
danwrong
 
Inside Bokete: Web Application with Mojolicious and others
Inside Bokete:  Web Application with Mojolicious and othersInside Bokete:  Web Application with Mojolicious and others
Inside Bokete: Web Application with Mojolicious and others
Yusuke Wada
 

What's hot (14)

Introduction To Lamp
Introduction To LampIntroduction To Lamp
Introduction To Lamp
 
RESTful web services
RESTful web servicesRESTful web services
RESTful web services
 
DrupalCafe4 Kiev Services
DrupalCafe4 Kiev ServicesDrupalCafe4 Kiev Services
DrupalCafe4 Kiev Services
 
Otimizando Aplicações em Rails
Otimizando Aplicações em RailsOtimizando Aplicações em Rails
Otimizando Aplicações em Rails
 
Slim RedBeanPHP and Knockout
Slim RedBeanPHP and KnockoutSlim RedBeanPHP and Knockout
Slim RedBeanPHP and Knockout
 
Perl Dancer for Python programmers
Perl Dancer for Python programmersPerl Dancer for Python programmers
Perl Dancer for Python programmers
 
Distributed work with Gearman
Distributed work with GearmanDistributed work with Gearman
Distributed work with Gearman
 
Object Oriented JavaScript
Object Oriented JavaScriptObject Oriented JavaScript
Object Oriented JavaScript
 
chef loves windows
chef loves windowschef loves windows
chef loves windows
 
Modern Perl Web Development with Dancer
Modern Perl Web Development with DancerModern Perl Web Development with Dancer
Modern Perl Web Development with Dancer
 
Keeping it small - Getting to know the Slim PHP micro framework
Keeping it small - Getting to know the Slim PHP micro frameworkKeeping it small - Getting to know the Slim PHP micro framework
Keeping it small - Getting to know the Slim PHP micro framework
 
PerlDancer for Perlers (FOSDEM 2011)
PerlDancer for Perlers (FOSDEM 2011)PerlDancer for Perlers (FOSDEM 2011)
PerlDancer for Perlers (FOSDEM 2011)
 
Loadrunner
LoadrunnerLoadrunner
Loadrunner
 
Inside Bokete: Web Application with Mojolicious and others
Inside Bokete:  Web Application with Mojolicious and othersInside Bokete:  Web Application with Mojolicious and others
Inside Bokete: Web Application with Mojolicious and others
 

Similar to BioMake PAG 2017

Introduction to GNU Make Programming Language
Introduction to GNU Make Programming LanguageIntroduction to GNU Make Programming Language
Introduction to GNU Make Programming Language
Shih-Hsiang Lin
 
Erubis徹底解説
Erubis徹底解説Erubis徹底解説
Erubis徹底解説
kwatch
 
All about Erubis (English)
All about Erubis (English)All about Erubis (English)
All about Erubis (English)
kwatch
 
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamMalo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Flink Forward
 
unix-editors.pdf
unix-editors.pdfunix-editors.pdf
unix-editors.pdf
samuelEstevam3
 
Revoke-Obfuscation
Revoke-ObfuscationRevoke-Obfuscation
Revoke-Obfuscation
Daniel Bohannon
 
Twig Templating
Twig TemplatingTwig Templating
Twig Templating
Rj Bautista
 
JavaOne 2008 - TS-5793 - Groovy and Grails, changing the landscape of Java EE...
JavaOne 2008 - TS-5793 - Groovy and Grails, changing the landscape of Java EE...JavaOne 2008 - TS-5793 - Groovy and Grails, changing the landscape of Java EE...
JavaOne 2008 - TS-5793 - Groovy and Grails, changing the landscape of Java EE...
Guillaume Laforge
 
LOSS_C11- Programming Linux 20221006.pdf
LOSS_C11- Programming Linux 20221006.pdfLOSS_C11- Programming Linux 20221006.pdf
LOSS_C11- Programming Linux 20221006.pdf
Thninh2
 
Practical catalyst
Practical catalystPractical catalyst
Practical catalyst
dwm042
 
BioMake BOSC 2004
BioMake BOSC 2004BioMake BOSC 2004
BioMake BOSC 2004
Chris Mungall
 
2005_Structures and functions of Makefile
2005_Structures and functions of Makefile2005_Structures and functions of Makefile
2005_Structures and functions of Makefile
NakCheon Jung
 
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
Flink Forward SF 2017: Malo Deniélou -  No shard left behind: Dynamic work re...Flink Forward SF 2017: Malo Deniélou -  No shard left behind: Dynamic work re...
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
Flink Forward
 
Software Build processes and Git
Software Build processes and GitSoftware Build processes and Git
Software Build processes and Git
Alec Clews
 
Laurens Van Den Oever Xopus Presentation
Laurens Van Den Oever Xopus PresentationLaurens Van Den Oever Xopus Presentation
Laurens Van Den Oever Xopus Presentation
Ajax Experience 2009
 
Xopus Application Framework
Xopus Application FrameworkXopus Application Framework
Xopus Application Framework
Jady Yang
 
CMake Tutorial
CMake TutorialCMake Tutorial
CMake Tutorial
Fu Haiping
 
Cより速いRubyプログラム
Cより速いRubyプログラムCより速いRubyプログラム
Cより速いRubyプログラム
kwatch
 
Content-Driven Web Applications with Magnolia CMS and Ruby on Rails
Content-Driven Web Applications with Magnolia CMS and Ruby on RailsContent-Driven Web Applications with Magnolia CMS and Ruby on Rails
Content-Driven Web Applications with Magnolia CMS and Ruby on Rails
bkraft
 
Why scala is not my ideal language and what I can do with this
Why scala is not my ideal language and what I can do with thisWhy scala is not my ideal language and what I can do with this
Why scala is not my ideal language and what I can do with this
Ruslan Shevchenko
 

Similar to BioMake PAG 2017 (20)

Introduction to GNU Make Programming Language
Introduction to GNU Make Programming LanguageIntroduction to GNU Make Programming Language
Introduction to GNU Make Programming Language
 
Erubis徹底解説
Erubis徹底解説Erubis徹底解説
Erubis徹底解説
 
All about Erubis (English)
All about Erubis (English)All about Erubis (English)
All about Erubis (English)
 
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamMalo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
 
unix-editors.pdf
unix-editors.pdfunix-editors.pdf
unix-editors.pdf
 
Revoke-Obfuscation
Revoke-ObfuscationRevoke-Obfuscation
Revoke-Obfuscation
 
Twig Templating
Twig TemplatingTwig Templating
Twig Templating
 
JavaOne 2008 - TS-5793 - Groovy and Grails, changing the landscape of Java EE...
JavaOne 2008 - TS-5793 - Groovy and Grails, changing the landscape of Java EE...JavaOne 2008 - TS-5793 - Groovy and Grails, changing the landscape of Java EE...
JavaOne 2008 - TS-5793 - Groovy and Grails, changing the landscape of Java EE...
 
LOSS_C11- Programming Linux 20221006.pdf
LOSS_C11- Programming Linux 20221006.pdfLOSS_C11- Programming Linux 20221006.pdf
LOSS_C11- Programming Linux 20221006.pdf
 
Practical catalyst
Practical catalystPractical catalyst
Practical catalyst
 
BioMake BOSC 2004
BioMake BOSC 2004BioMake BOSC 2004
BioMake BOSC 2004
 
2005_Structures and functions of Makefile
2005_Structures and functions of Makefile2005_Structures and functions of Makefile
2005_Structures and functions of Makefile
 
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
Flink Forward SF 2017: Malo Deniélou -  No shard left behind: Dynamic work re...Flink Forward SF 2017: Malo Deniélou -  No shard left behind: Dynamic work re...
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
 
Software Build processes and Git
Software Build processes and GitSoftware Build processes and Git
Software Build processes and Git
 
Laurens Van Den Oever Xopus Presentation
Laurens Van Den Oever Xopus PresentationLaurens Van Den Oever Xopus Presentation
Laurens Van Den Oever Xopus Presentation
 
Xopus Application Framework
Xopus Application FrameworkXopus Application Framework
Xopus Application Framework
 
CMake Tutorial
CMake TutorialCMake Tutorial
CMake Tutorial
 
Cより速いRubyプログラム
Cより速いRubyプログラムCより速いRubyプログラム
Cより速いRubyプログラム
 
Content-Driven Web Applications with Magnolia CMS and Ruby on Rails
Content-Driven Web Applications with Magnolia CMS and Ruby on RailsContent-Driven Web Applications with Magnolia CMS and Ruby on Rails
Content-Driven Web Applications with Magnolia CMS and Ruby on Rails
 
Why scala is not my ideal language and what I can do with this
Why scala is not my ideal language and what I can do with thisWhy scala is not my ideal language and what I can do with this
Why scala is not my ideal language and what I can do with this
 

More from Chris Mungall

MADICES Mungall 2022.pptx
MADICES Mungall 2022.pptxMADICES Mungall 2022.pptx
MADICES Mungall 2022.pptx
Chris Mungall
 
Scaling up semantics; lessons learned across the life sciences
Scaling up semantics; lessons learned across the life sciencesScaling up semantics; lessons learned across the life sciences
Scaling up semantics; lessons learned across the life sciences
Chris Mungall
 
LinkML Intro July 2022.pptx PLEASE VIEW THIS ON ZENODO
LinkML Intro July 2022.pptx PLEASE VIEW THIS ON ZENODOLinkML Intro July 2022.pptx PLEASE VIEW THIS ON ZENODO
LinkML Intro July 2022.pptx PLEASE VIEW THIS ON ZENODO
Chris Mungall
 
Ontology Access Kit_ Workshop Intro Slides.pptx
Ontology Access Kit_ Workshop Intro Slides.pptxOntology Access Kit_ Workshop Intro Slides.pptx
Ontology Access Kit_ Workshop Intro Slides.pptx
Chris Mungall
 
LinkML Intro (for Monarch devs)
LinkML Intro (for Monarch devs)LinkML Intro (for Monarch devs)
LinkML Intro (for Monarch devs)
Chris Mungall
 
LinkML presentation to Yosemite Group
LinkML presentation to Yosemite GroupLinkML presentation to Yosemite Group
LinkML presentation to Yosemite Group
Chris Mungall
 
Experiences in the biosciences with the open biological ontologies foundry an...
Experiences in the biosciences with the open biological ontologies foundry an...Experiences in the biosciences with the open biological ontologies foundry an...
Experiences in the biosciences with the open biological ontologies foundry an...
Chris Mungall
 
All together now: piecing together the knowledge graph of life
All together now: piecing together the knowledge graph of lifeAll together now: piecing together the knowledge graph of life
All together now: piecing together the knowledge graph of life
Chris Mungall
 
Collaboratively Creating the Knowledge Graph of Life
Collaboratively Creating the Knowledge Graph of LifeCollaboratively Creating the Knowledge Graph of Life
Collaboratively Creating the Knowledge Graph of Life
Chris Mungall
 
Representation of kidney structures in Uberon
Representation of kidney structures in UberonRepresentation of kidney structures in Uberon
Representation of kidney structures in Uberon
Chris Mungall
 
SparqlProg (BioHackathon 2019)
SparqlProg (BioHackathon 2019)SparqlProg (BioHackathon 2019)
SparqlProg (BioHackathon 2019)
Chris Mungall
 
Ontology Development Kit: Bio-Ontologies 2019
Ontology Development Kit: Bio-Ontologies 2019Ontology Development Kit: Bio-Ontologies 2019
Ontology Development Kit: Bio-Ontologies 2019
Chris Mungall
 
US2TS: Reasoning over multiple open bio-ontologies to make machines and human...
US2TS: Reasoning over multiple open bio-ontologies to make machines and human...US2TS: Reasoning over multiple open bio-ontologies to make machines and human...
US2TS: Reasoning over multiple open bio-ontologies to make machines and human...
Chris Mungall
 
Uberon: opening up to community contributions
Uberon: opening up to community contributionsUberon: opening up to community contributions
Uberon: opening up to community contributions
Chris Mungall
 
Modeling exposure events and adverse outcome pathways using ontologies
Modeling exposure events and adverse outcome pathways using ontologiesModeling exposure events and adverse outcome pathways using ontologies
Modeling exposure events and adverse outcome pathways using ontologies
Chris Mungall
 
Causal reasoning using the Relation Ontology
Causal reasoning using the Relation OntologyCausal reasoning using the Relation Ontology
Causal reasoning using the Relation Ontology
Chris Mungall
 
US2TS presentation on Gene Ontology
US2TS presentation on Gene OntologyUS2TS presentation on Gene Ontology
US2TS presentation on Gene Ontology
Chris Mungall
 
Introduction to the BioLink datamodel
Introduction to the BioLink datamodelIntroduction to the BioLink datamodel
Introduction to the BioLink datamodel
Chris Mungall
 
Computing on Phenotypes AMP 2015
Computing on Phenotypes AMP 2015Computing on Phenotypes AMP 2015
Computing on Phenotypes AMP 2015
Chris Mungall
 
ENVO GSC 2015
ENVO GSC 2015ENVO GSC 2015
ENVO GSC 2015
Chris Mungall
 

More from Chris Mungall (20)

MADICES Mungall 2022.pptx
MADICES Mungall 2022.pptxMADICES Mungall 2022.pptx
MADICES Mungall 2022.pptx
 
Scaling up semantics; lessons learned across the life sciences
Scaling up semantics; lessons learned across the life sciencesScaling up semantics; lessons learned across the life sciences
Scaling up semantics; lessons learned across the life sciences
 
LinkML Intro July 2022.pptx PLEASE VIEW THIS ON ZENODO
LinkML Intro July 2022.pptx PLEASE VIEW THIS ON ZENODOLinkML Intro July 2022.pptx PLEASE VIEW THIS ON ZENODO
LinkML Intro July 2022.pptx PLEASE VIEW THIS ON ZENODO
 
Ontology Access Kit_ Workshop Intro Slides.pptx
Ontology Access Kit_ Workshop Intro Slides.pptxOntology Access Kit_ Workshop Intro Slides.pptx
Ontology Access Kit_ Workshop Intro Slides.pptx
 
LinkML Intro (for Monarch devs)
LinkML Intro (for Monarch devs)LinkML Intro (for Monarch devs)
LinkML Intro (for Monarch devs)
 
LinkML presentation to Yosemite Group
LinkML presentation to Yosemite GroupLinkML presentation to Yosemite Group
LinkML presentation to Yosemite Group
 
Experiences in the biosciences with the open biological ontologies foundry an...
Experiences in the biosciences with the open biological ontologies foundry an...Experiences in the biosciences with the open biological ontologies foundry an...
Experiences in the biosciences with the open biological ontologies foundry an...
 
All together now: piecing together the knowledge graph of life
All together now: piecing together the knowledge graph of lifeAll together now: piecing together the knowledge graph of life
All together now: piecing together the knowledge graph of life
 
Collaboratively Creating the Knowledge Graph of Life
Collaboratively Creating the Knowledge Graph of LifeCollaboratively Creating the Knowledge Graph of Life
Collaboratively Creating the Knowledge Graph of Life
 
Representation of kidney structures in Uberon
Representation of kidney structures in UberonRepresentation of kidney structures in Uberon
Representation of kidney structures in Uberon
 
SparqlProg (BioHackathon 2019)
SparqlProg (BioHackathon 2019)SparqlProg (BioHackathon 2019)
SparqlProg (BioHackathon 2019)
 
Ontology Development Kit: Bio-Ontologies 2019
Ontology Development Kit: Bio-Ontologies 2019Ontology Development Kit: Bio-Ontologies 2019
Ontology Development Kit: Bio-Ontologies 2019
 
US2TS: Reasoning over multiple open bio-ontologies to make machines and human...
US2TS: Reasoning over multiple open bio-ontologies to make machines and human...US2TS: Reasoning over multiple open bio-ontologies to make machines and human...
US2TS: Reasoning over multiple open bio-ontologies to make machines and human...
 
Uberon: opening up to community contributions
Uberon: opening up to community contributionsUberon: opening up to community contributions
Uberon: opening up to community contributions
 
Modeling exposure events and adverse outcome pathways using ontologies
Modeling exposure events and adverse outcome pathways using ontologiesModeling exposure events and adverse outcome pathways using ontologies
Modeling exposure events and adverse outcome pathways using ontologies
 
Causal reasoning using the Relation Ontology
Causal reasoning using the Relation OntologyCausal reasoning using the Relation Ontology
Causal reasoning using the Relation Ontology
 
US2TS presentation on Gene Ontology
US2TS presentation on Gene OntologyUS2TS presentation on Gene Ontology
US2TS presentation on Gene Ontology
 
Introduction to the BioLink datamodel
Introduction to the BioLink datamodelIntroduction to the BioLink datamodel
Introduction to the BioLink datamodel
 
Computing on Phenotypes AMP 2015
Computing on Phenotypes AMP 2015Computing on Phenotypes AMP 2015
Computing on Phenotypes AMP 2015
 
ENVO GSC 2015
ENVO GSC 2015ENVO GSC 2015
ENVO GSC 2015
 

Recently uploaded

Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
XfilesPro
 
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
dakas1
 
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
widenerjobeyrl638
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
OnePlan Solutions
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
Marcin Chrost
 
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
Paul Brebner
 
42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert
vaishalijagtap12
 
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyesAssure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Paul Brebner
 
TMU毕业证书精仿办理
TMU毕业证书精仿办理TMU毕业证书精仿办理
TMU毕业证书精仿办理
aeeva
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Peter Caitens
 
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
kalichargn70th171
 
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
Luigi Fugaro
 
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
kalichargn70th171
 
Photoshop Tutorial for Beginners (2024 Edition)
Photoshop Tutorial for Beginners (2024 Edition)Photoshop Tutorial for Beginners (2024 Edition)
Photoshop Tutorial for Beginners (2024 Edition)
alowpalsadig
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
What’s New in Odoo 17 – A Complete Roadmap
What’s New in Odoo 17 – A Complete RoadmapWhat’s New in Odoo 17 – A Complete Roadmap
What’s New in Odoo 17 – A Complete Roadmap
Envertis Software Solutions
 

Recently uploaded (20)

Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
 
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
 
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...Transforming Product Development using OnePlan To Boost Efficiency and Innova...
Transforming Product Development using OnePlan To Boost Efficiency and Innova...
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
 
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
 
42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert
 
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyesAssure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
 
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...
 
TMU毕业证书精仿办理
TMU毕业证书精仿办理TMU毕业证书精仿办理
TMU毕业证书精仿办理
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
 
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
The Power of Visual Regression Testing_ Why It Is Critical for Enterprise App...
 
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
WMF 2024 - Unlocking the Future of Data Powering Next-Gen AI with Vector Data...
 
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
 
Photoshop Tutorial for Beginners (2024 Edition)
Photoshop Tutorial for Beginners (2024 Edition)Photoshop Tutorial for Beginners (2024 Edition)
Photoshop Tutorial for Beginners (2024 Edition)
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
What’s New in Odoo 17 – A Complete Roadmap
What’s New in Odoo 17 – A Complete RoadmapWhat’s New in Odoo 17 – A Complete Roadmap
What’s New in Odoo 17 – A Complete Roadmap
 

BioMake PAG 2017

  • 1. BioMake Ian Holmes & Christopher Mungall UC Berkeley, Berkeley Lab http://biomake.org/
  • 2. Bioinformatics analysis pipelines involves chains of dependencies
  • 3. GNU make CC = gcc %.o: %.c $(CC) -c -o $@ $< Rules, patterns, variables, automatic variables Compiling a C program Makefile
  • 4. GNU make BWAMEM = bwa REF = ref.fasta %.sam: %.fastq $(BWA) mem $(REF) $< >$@ Aligning reads Rules, patterns, variables, automatic variables Makefile
  • 5. Dependencies BWAMEM = bwa REF = ref.fasta %.sam: %.fastq $(REF).bwt $(BWA) mem $(REF) $< >$@ %.bwt: % $(BWA) index $< Timestamps, dependency graphs
  • 6. Functions MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST))) MAKEFILE_DIR := $(dir $(MAKEFILE_PATH)) test1: echo $(shell ls $(MAKEFILE_DIR)) FUNC = echo $1 is $2 test2: $(call FUNC,make,cool) Special variables, functions, user-defined functions
  • 7. Issues with GNU make • Only one wildcard per rule • Poor support for parallelism • Timestamps are fragile, especially on NFS • Can’t extend build logic, e.g. add qualifiers to rules Biomake fixes these!
  • 8. Multiple wildcards per rule $X-$Y.sam: $X.fa.bwt $Y.fastq bwa mem $X.fa $Y.fastq >$@ biomake myref-myreads.sam
  • 9. MD5 signatures $X-$Y.sam: $X.fa.bwt $Y.fastq bwa mem $X.fa $Y.fastq >$@ biomake --md5-hash myref-myreads.sam md5_hash("myref-myreads.sam",12,"0f723ae7f9bf07744445e93ac5595156"). md5_valid("myref-myreads.sam",12,"0f723ae7f9bf07744445e93ac5595156",X) :- md5_check("myref.fa.bwt",6,"b1946ac92492d2347c6235b4d2611184",X), md5_check("myreads.fastq",6,"591785b794601e212b260e25925636fd",X). .biomake/md5/myref-myreads.sam
  • 10. Multiple queue engines $X-$Y.sam: $X.fa.bwt $Y.fastq bwa mem $X.fa $Y.fastq >$@ biomake -Q sge myref-myreads.sam biomake -Q slurm myref-myreads.sam biomake -Q pbs myref-myreads.sam biomake -Q poolq myref-myreads.sam
  • 11. Logic extensions $X-$Y.sam: $X.fa.bwt $Y { size_file(Y,S), S < 1000000000 } bwa mem $X.fa $Y >$@ Can embed Prolog in Makefiles, or auto-translate entire Makefile to Prolog
  • 12. (Some of) The Competition Language GNU- compatible? MD5? Clusters? Erlang make Erlang No No No omake OCaML Somewhat Yes No makepp Perl Yes Yes No qmake C Yes No SGE
  • 13. What Biomake is not(yet) • A massively parallel data analysis framework (try Apache Spark) • A heavy-duty cloud-oriented workflow language (try Common Workflow Language) • A web application for managing jobs (try Galaxy)
  • 14. What Biomake is A simple, drop-in replacement for GNU make that allows you to… • ramp up your Makefile-driven workflows to cluster scale • avoid costly rebuilds triggered by file copying & unsynchronized clocks • extend rules with wildcards and logical tests http://www.biorxiv.org/content/early/2016/12/12/093245 https://github.com/evoldoers/biomake
  • 15. http://biomake.org/ • Ian Holmes • Contributors & testers • Mahesh Panchal • Markus Triska • Jan Wielemaker - creator of SWI-Prolog