The document outlines the efforts of the Belgian startup Data Fellas in providing scalable and interoperable data services for genomics, focusing on the Big Data challenges represented by genomic data. It highlights various projects and training initiatives related to Apache Spark and distributed machine learning, emphasizing the importance of metadata and service discovery in genomic research. Future plans include expanding their services to incorporate medical data trading and geospatial modules.

1. by Data Fellas,
Data Enthusiasts v 4.0 (July, 13th ‘15)
Scalable and Interoperable data services
Applied to Genomics

2. Young Belgian Startup
The Data Fellas Startup
Data Science
Xavier Tordoir
@xtordoir
Andy Petrella
@noootsab
Data Processing
Scalable Machine Learning
Micro Services oriented

3. Data Fellas Ecosystem
We’ve worked with

4. Data Fellas: Evangelizing
Training
Scala
Apache Spark (BE, in September)
http://spark4devs.data-fellas.guru/
Distributed Machine Learning
Pipeline (Oakland, August)
http://bigdatascala.bythebay.io/training.html
Apache Spark
(SFO with BoldRadius, August)
Talks
Scala IO, Devoxx Belgium,
Devoxx France, Scala Days, KTH,
KUL, Spark Meetup London, …
more to come (Italy, …)
PMC Member at Strata NY
PMC member at Devoxx
PMC Member at Foss4G

5. First: Data Science
Analysis
Spark Notebook

6. First: Data Science
Analysis
Production
Project Generator
Mesos / C* / DCOS

7. First: Data Science
Analysis
Production
Distribution
Micro Service /
Binary format
Marathon

8. First: Data Science
Analysis
Production
DistributionRendering
SChema for output
GG / D3 …

9. First: Data Science
Analysis
Production
DistributionRendering
Discovery
Service Metadata
SOLR , …

10. First: Data Science
Analysis
Production
DistributionRendering
Discovery
Catalog
Spark Notebook
using Services
too

11. First: Data Science
Analysis
Production
DistributionRendering
Discovery
Share
Analyses
Share
Results
Share
Datasets

12. First: Data Science
Project Code Name:
Shar3

13. Next: Applied TO Genomics
Genomics data is pretty big
● 100,000’s genomes in 2015
● 1,000,000’s …
● 100,000,000’s …
● …

14. Next: Applied TO Genomics
Genomics data is pretty big and of High dimensionality
One genome:
○ 3 billions bases (basic DNA component) sequence
○ 30 - 60 x coverage for quality
○ 10’s to 100’s millions variants (variable bases
from one individual to the next)

15. Next: Applied TO Genomics
e.g. 1000genomes project:
● 200TB compressed data
● organised in files/directories
● data formatted following specs in a … PDF
Data and services schemas are required

16. What we do with genomics data?
Lots of Querying and Learning:
E.G.
● Population structure is a fundamental basis
● Querying relationships between genomes and other
biological features
Hey… no one has all data!
Metadata

17. What we do with genomics data?
Lots of Querying and Learning:
E.G.
● We do some specific Modelling on some data…
Hey… no two serve the same computations!
Service Discovery

18. Interoperability
So, no one has all data …
BUT all should be able to talk…
Interoperability (GA4GH)

19. Interoperable…
Analysis
Production
DistributionRendering
Discovery
Share
Analyses
Share
Results
Share
Datasets

20. Interoperable & scalable…
GA4GH + Shar3 = Med@Scale
+ ADAM & spark
+ In Memory optimization (Tachyon)
+ Deployment (e.g. DCOS)

21. Wrap-UP
Follow us @DataFellas and get notified about our
+ sharing platform at scale: Shar3
+ Google Genomics At Home (^.^): Med@Scale
+ future plans: modules for Trading, Geospatial,
other medical data, …

22. References
Adam: https://github.com/bigdatagenomics/adam
Bdg-Formats: https://github.com/bigdatagenomics/bdg-formats
GA4GH website: http://genomicsandhealth.org/
GA4GH data working group: http://ga4gh.org/
@Spark-Notebook: https://github.com/andypetrella/spark-notebook/
Med-At-Scale: https://github.com/med-at-scale/high-health
Data Fellas: http://data-fellas.guru/
Training: http://spark4devs.data-fellas.guru/

23. Q/A
THANKS!