SlideShare a Scribd company logo
IGA Workflow
Alessandro Gervaso   Vittorio Zamboni
Introduction
IGA Workflow is a web-based tool created
using the Django Framework.
It is intended as a management tool for the IGA
wet-lab.

We use it to track the lifecycle of biological
samples, from vial to file.
Overview
● Laboratory management, from sample to
  flowcell
● Bioinformatic analyses
● Pipelines management
● Technology
● Other applications and future developments
Biology for dummies
Technology - Basics
Database server
● Postgresql
● Redis
Workers
● Celery
Web server
● nginx + uWSGI
Overview - Lab
● SAMPLE: The basic unit in the lab

● LIBRARY: a treated sample with an attached
  chemical TAG.

● POOL: a set of libraries, ready to be placed
  on a flowcell's lane.
Overview - Lab
Overview - Lab
The main challenge was to replace notebooks
with a tool that allows to:
● insert samples, libraries, pools, and edit
   them;
● create lanes and runs and the configuration
   files for the physical sequencer;
● collect the sequencer results and map them
   in a easy way
Almost done!
Overview - Lab
We started using the basic Django admin
                     BUT

● the page loading was slow
● due to the admin's nature we lacked
  flexibility
● we were forcing the lab people procedures
● management was cumbersome
Overview - Analyses
After the physical sequencing the raw data
(basecalls) must be converted in FASTQ files.

The FASTQ files are FASTA files with some
embedded quality stats.

They are the starting point for almost every
genomic analysis.
Overview - Analyses
To optimize time and resources we use a
cluster of Celery workers.

● we track the software packages used
● we track their parameters
● we create a set of useful stats
Technology
Additions:

● Informational celery tasks (list directories
  content, copy files between devices with
  dbus plugin and UDisks instead of ugly
  hacks, ...)
Overview - Pipelines
FASTQ files     alignments and assemblies

Each analysis use different software in
sequence or in parallel.
Using hundreds of samples, the analyses can't
be handled manually.
Pipelines
The results of each pipelines (like previous
analyses) must be tracked.

Since CLI based software is not user-friendly,
we develop a graphical pipeline builder.

Users are able to choose and combine different
softwares to perform their own analyses.
Pipeline
● Workers have different queues in order to
  satisfy different tasks

● Worker's tasks talk each other with Redis to
  avoid inconsistencies and to improve
  performances
Pipelines
Different queues (stored in Redis dbs):

  ○   available
  ○   queued
  ○   active
  ○   completed
  ○   error
Pipelines
Video demonstration of a working pipeline with
use of different kind of steps.
(1 video: 1 minute)
Under development
● simple interface that allow customers to:
  ○ insert their samples directly
  ○ watch the results of their pipeline in a genome
      browser - also made with Django (see below)!

● barcodes

● genome browser (like GMOD GBrowse, but
  with the greatness of Python instead of the
  confusion of Perl)
Genome browser
An application that allows browsing a genome's
annotations (like genes, or where reads are
aligned).

Actually, the best web genome browser is
GMOD Gbrowse.
Genome browser
The challenge is to develop a genome browser
that has a set of basic features and could
accept plugins for particular type of data - like
GMOD Gbrowse.

In addition, it must be quick and easy to
manage - NOT like GMOD Gbrowse.
Genome browser
Video demonstration of the genome browser.
(2 videos: 1 minute + 1 minute)
Acknowledgments
● The wet-lab     Teams that developed:
  ladies at IGA   ● Django
                  ● JQuery
                  ● nginx
● WEBdeBS         ● uWSGI
                  ● Celery
                  ● Redis
                  ● pip and virtualenv
                  ● PostgreSQL
                  ● All the open source projects involved

More Related Content

What's hot

Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Develcz
 
Bgoug 2019.11 test your pl sql - not your patience
Bgoug 2019.11   test your pl sql - not your patienceBgoug 2019.11   test your pl sql - not your patience
Bgoug 2019.11 test your pl sql - not your patience
Jacek Gebal
 
Grunt - The JavaScript Task Runner
Grunt - The JavaScript Task RunnerGrunt - The JavaScript Task Runner
Grunt - The JavaScript Task Runner
Mohammed Arif
 
Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014
Puppet
 
Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!
Puppet
 
What grunt?
What grunt?What grunt?
What grunt?
Lucio Martinez
 
OpenCms Days 2012 - Developing OpenCms with Gradle
OpenCms Days 2012 - Developing OpenCms with GradleOpenCms Days 2012 - Developing OpenCms with Gradle
OpenCms Days 2012 - Developing OpenCms with Gradle
Alkacon Software GmbH & Co. KG
 
Groovy and noteworthy
Groovy and noteworthyGroovy and noteworthy
Groovy and noteworthy
Izzet Mustafaiev
 
Sandstorm.io and OpenDocument
Sandstorm.io and OpenDocumentSandstorm.io and OpenDocument
Sandstorm.io and OpenDocument
Audrey Tang
 
React Django Presentation
React Django PresentationReact Django Presentation
React Django Presentation
Allison DiNapoli
 

What's hot (10)

Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
 
Bgoug 2019.11 test your pl sql - not your patience
Bgoug 2019.11   test your pl sql - not your patienceBgoug 2019.11   test your pl sql - not your patience
Bgoug 2019.11 test your pl sql - not your patience
 
Grunt - The JavaScript Task Runner
Grunt - The JavaScript Task RunnerGrunt - The JavaScript Task Runner
Grunt - The JavaScript Task Runner
 
Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014
 
Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!
 
What grunt?
What grunt?What grunt?
What grunt?
 
OpenCms Days 2012 - Developing OpenCms with Gradle
OpenCms Days 2012 - Developing OpenCms with GradleOpenCms Days 2012 - Developing OpenCms with Gradle
OpenCms Days 2012 - Developing OpenCms with Gradle
 
Groovy and noteworthy
Groovy and noteworthyGroovy and noteworthy
Groovy and noteworthy
 
Sandstorm.io and OpenDocument
Sandstorm.io and OpenDocumentSandstorm.io and OpenDocument
Sandstorm.io and OpenDocument
 
React Django Presentation
React Django PresentationReact Django Presentation
React Django Presentation
 

Similar to Iga workflow

Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Sauce Labs
 
DevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsDevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsFedir RYKHTIK
 
SciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programming
Samuel Lampa
 
ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest Project
 
Reproducibility in artificial intelligence
Reproducibility in artificial intelligenceReproducibility in artificial intelligence
Reproducibility in artificial intelligence
Carlos Toxtli
 
2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_MicroservicesJason Varghese
 
Software Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery ApproachSoftware Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery Approach
Giovanni Toraldo
 
ElasTest Webinar
ElasTest WebinarElasTest Webinar
ElasTest Webinar
ElasTest Project
 
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Ohad Basan
 
TDD with Python and App Engine
TDD with Python and App EngineTDD with Python and App Engine
TDD with Python and App Engine
Ricardo Bánffy
 
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest Project
 
Go at Skroutz
Go at SkroutzGo at Skroutz
Go at Skroutz
AgisAnastasopoulos
 
Introduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibusIntroduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibus
Davide Fella
 
Cypress Testing.pptx
Cypress Testing.pptxCypress Testing.pptx
Cypress Testing.pptx
JasmeenShrestha
 
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie..."How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
Fwdays
 
Integration testing - A&BP CC
Integration testing - A&BP CCIntegration testing - A&BP CC
Integration testing - A&BP CC
JWORKS powered by Ordina
 
Mono Repo
Mono RepoMono Repo
Mono Repo
Zacky Pickholz
 
Cypress for Testing
Cypress for TestingCypress for Testing
Cypress for Testing
PoojaSingh1123
 
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Борис Зора
 
OpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load TestsOpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load Tests
jhierrot
 

Similar to Iga workflow (20)

Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
 
DevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsDevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and Projects
 
SciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programming
 
ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)
 
Reproducibility in artificial intelligence
Reproducibility in artificial intelligenceReproducibility in artificial intelligence
Reproducibility in artificial intelligence
 
2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices
 
Software Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery ApproachSoftware Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery Approach
 
ElasTest Webinar
ElasTest WebinarElasTest Webinar
ElasTest Webinar
 
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
 
TDD with Python and App Engine
TDD with Python and App EngineTDD with Python and App Engine
TDD with Python and App Engine
 
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
 
Go at Skroutz
Go at SkroutzGo at Skroutz
Go at Skroutz
 
Introduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibusIntroduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibus
 
Cypress Testing.pptx
Cypress Testing.pptxCypress Testing.pptx
Cypress Testing.pptx
 
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie..."How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
 
Integration testing - A&BP CC
Integration testing - A&BP CCIntegration testing - A&BP CC
Integration testing - A&BP CC
 
Mono Repo
Mono RepoMono Repo
Mono Repo
 
Cypress for Testing
Cypress for TestingCypress for Testing
Cypress for Testing
 
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
 
OpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load TestsOpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load Tests
 

More from WEBdeBS

Legal Pirlo
Legal PirloLegal Pirlo
Legal Pirlo
WEBdeBS
 
Nodejsconf 2012
Nodejsconf 2012Nodejsconf 2012
Nodejsconf 2012
WEBdeBS
 
Mockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzioneMockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzione
WEBdeBS
 
Djangoday lt 20120420
Djangoday lt 20120420Djangoday lt 20120420
Djangoday lt 20120420WEBdeBS
 
Unbit djangoday 20120419
Unbit djangoday 20120419Unbit djangoday 20120419
Unbit djangoday 20120419WEBdeBS
 
Django mongodb -djangoday_
Django mongodb -djangoday_Django mongodb -djangoday_
Django mongodb -djangoday_WEBdeBS
 
Virtualenv
VirtualenvVirtualenv
Virtualenv
WEBdeBS
 
Fagungis
FagungisFagungis
Fagungis
WEBdeBS
 
Django e il Rap Elia Contini
Django e il Rap Elia ContiniDjango e il Rap Elia Contini
Django e il Rap Elia ContiniWEBdeBS
 
PepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaPepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaWEBdeBS
 
Peppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaPeppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaWEBdeBS
 
NoSql Day - Chiusura
NoSql Day - ChiusuraNoSql Day - Chiusura
NoSql Day - Chiusura
WEBdeBS
 
NoSql Day - Apertura
NoSql Day - AperturaNoSql Day - Apertura
NoSql Day - Apertura
WEBdeBS
 

More from WEBdeBS (14)

Legal Pirlo
Legal PirloLegal Pirlo
Legal Pirlo
 
Nodejsconf 2012
Nodejsconf 2012Nodejsconf 2012
Nodejsconf 2012
 
Mockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzioneMockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzione
 
Djangoday lt 20120420
Djangoday lt 20120420Djangoday lt 20120420
Djangoday lt 20120420
 
Unbit djangoday 20120419
Unbit djangoday 20120419Unbit djangoday 20120419
Unbit djangoday 20120419
 
Geodjango
GeodjangoGeodjango
Geodjango
 
Django mongodb -djangoday_
Django mongodb -djangoday_Django mongodb -djangoday_
Django mongodb -djangoday_
 
Virtualenv
VirtualenvVirtualenv
Virtualenv
 
Fagungis
FagungisFagungis
Fagungis
 
Django e il Rap Elia Contini
Django e il Rap Elia ContiniDjango e il Rap Elia Contini
Django e il Rap Elia Contini
 
PepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaPepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend Brescia
 
Peppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaPeppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend Brescia
 
NoSql Day - Chiusura
NoSql Day - ChiusuraNoSql Day - Chiusura
NoSql Day - Chiusura
 
NoSql Day - Apertura
NoSql Day - AperturaNoSql Day - Apertura
NoSql Day - Apertura
 

Recently uploaded

Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
Generating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using SmithyGenerating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using Smithy
g2nightmarescribd
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
Cheryl Hung
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Dorra BARTAGUIZ
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
Paul Groth
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 

Recently uploaded (20)

Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
Generating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using SmithyGenerating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using Smithy
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 

Iga workflow

  • 2. Introduction IGA Workflow is a web-based tool created using the Django Framework. It is intended as a management tool for the IGA wet-lab. We use it to track the lifecycle of biological samples, from vial to file.
  • 3. Overview ● Laboratory management, from sample to flowcell ● Bioinformatic analyses ● Pipelines management ● Technology ● Other applications and future developments
  • 5. Technology - Basics Database server ● Postgresql ● Redis Workers ● Celery Web server ● nginx + uWSGI
  • 6. Overview - Lab ● SAMPLE: The basic unit in the lab ● LIBRARY: a treated sample with an attached chemical TAG. ● POOL: a set of libraries, ready to be placed on a flowcell's lane.
  • 8. Overview - Lab The main challenge was to replace notebooks with a tool that allows to: ● insert samples, libraries, pools, and edit them; ● create lanes and runs and the configuration files for the physical sequencer; ● collect the sequencer results and map them in a easy way Almost done!
  • 9. Overview - Lab We started using the basic Django admin BUT ● the page loading was slow ● due to the admin's nature we lacked flexibility ● we were forcing the lab people procedures ● management was cumbersome
  • 10.
  • 11.
  • 12. Overview - Analyses After the physical sequencing the raw data (basecalls) must be converted in FASTQ files. The FASTQ files are FASTA files with some embedded quality stats. They are the starting point for almost every genomic analysis.
  • 13. Overview - Analyses To optimize time and resources we use a cluster of Celery workers. ● we track the software packages used ● we track their parameters ● we create a set of useful stats
  • 14.
  • 15. Technology Additions: ● Informational celery tasks (list directories content, copy files between devices with dbus plugin and UDisks instead of ugly hacks, ...)
  • 16. Overview - Pipelines FASTQ files alignments and assemblies Each analysis use different software in sequence or in parallel. Using hundreds of samples, the analyses can't be handled manually.
  • 17. Pipelines The results of each pipelines (like previous analyses) must be tracked. Since CLI based software is not user-friendly, we develop a graphical pipeline builder. Users are able to choose and combine different softwares to perform their own analyses.
  • 18. Pipeline ● Workers have different queues in order to satisfy different tasks ● Worker's tasks talk each other with Redis to avoid inconsistencies and to improve performances
  • 19. Pipelines Different queues (stored in Redis dbs): ○ available ○ queued ○ active ○ completed ○ error
  • 20. Pipelines Video demonstration of a working pipeline with use of different kind of steps. (1 video: 1 minute)
  • 21. Under development ● simple interface that allow customers to: ○ insert their samples directly ○ watch the results of their pipeline in a genome browser - also made with Django (see below)! ● barcodes ● genome browser (like GMOD GBrowse, but with the greatness of Python instead of the confusion of Perl)
  • 22. Genome browser An application that allows browsing a genome's annotations (like genes, or where reads are aligned). Actually, the best web genome browser is GMOD Gbrowse.
  • 23. Genome browser The challenge is to develop a genome browser that has a set of basic features and could accept plugins for particular type of data - like GMOD Gbrowse. In addition, it must be quick and easy to manage - NOT like GMOD Gbrowse.
  • 24. Genome browser Video demonstration of the genome browser. (2 videos: 1 minute + 1 minute)
  • 25. Acknowledgments ● The wet-lab Teams that developed: ladies at IGA ● Django ● JQuery ● nginx ● WEBdeBS ● uWSGI ● Celery ● Redis ● pip and virtualenv ● PostgreSQL ● All the open source projects involved