Data Enthusiasts London: Scalable and Interoperable data services. Applied to Genomics

•

1 like•1,254 views

Data science requires so many skills, people and time before the results can be accessed. Moreover, these results cannot be static anymore. And finally, the Big Data comes to the plate and the whole tool chain needs to change. In this talk Data Fellas introduces Shar3, a tool kit aiming to bridged the gaps to build a interactive distributed data processing pipeline, or loop! Then the talk covers genomics nowadays problems including data types, processing, discovery by introducing the GA4GH initiative and its implementation using Shar3.

by Data Fellas,
Data Enthusiasts v 4.0 (July, 13th ‘15)
Scalable and Interoperable data services
Applied to Genomics

Young Belgian Startup
The Data Fellas Startup
Data Science
Xavier Tordoir
@xtordoir
Andy Petrella
@noootsab
Data Processing
Scalable Machine Learning
Micro Services oriented

Data Fellas: Evangelizing
Training
Scala
Apache Spark (BE, in September)
http://spark4devs.data-fellas.guru/
Distributed Machine Learning
Pipeline (Oakland, August)
http://bigdatascala.bythebay.io/training.html
Apache Spark
(SFO with BoldRadius, August)
Talks
Scala IO, Devoxx Belgium,
Devoxx France, Scala Days, KTH,
KUL, Spark Meetup London, …
more to come (Italy, …)
PMC Member at Strata NY
PMC member at Devoxx
PMC Member at Foss4G

First: Data Science
Analysis
Spark Notebook

First: Data Science
Analysis
Production
Project Generator
Mesos / C* / DCOS

First: Data Science
Analysis
Production
Distribution
Micro Service /
Binary format
Marathon

First: Data Science
Analysis
Production
DistributionRendering
SChema for output
GG / D3 …

First: Data Science
Analysis
Production
DistributionRendering
Discovery
Service Metadata
SOLR , …

First: Data Science
Analysis
Production
DistributionRendering
Discovery
Catalog
Spark Notebook
using Services
too

First: Data Science
Analysis
Production
DistributionRendering
Discovery
Share
Analyses
Share
Results
Share
Datasets

First: Data Science
Project Code Name:
Shar3

Next: Applied TO Genomics
Genomics data is pretty big
● 100,000’s genomes in 2015
● 1,000,000’s …
● 100,000,000’s …
● …

Next: Applied TO Genomics
Genomics data is pretty big and of High dimensionality
One genome:
○ 3 billions bases (basic DNA component) sequence
○ 30 - 60 x coverage for quality
○ 10’s to 100’s millions variants (variable bases
from one individual to the next)

Next: Applied TO Genomics
e.g. 1000genomes project:
● 200TB compressed data
● organised in files/directories
● data formatted following specs in a … PDF
Data and services schemas are required

What we do with genomics data?
Lots of Querying and Learning:
E.G.
● Population structure is a fundamental basis
● Querying relationships between genomes and other
biological features
Hey… no one has all data!
Metadata

What we do with genomics data?
Lots of Querying and Learning:
E.G.
● We do some specific Modelling on some data…
Hey… no two serve the same computations!
Service Discovery

Interoperability
So, no one has all data …
BUT all should be able to talk…
Interoperability (GA4GH)

Interoperable…
Analysis
Production
DistributionRendering
Discovery
Share
Analyses
Share
Results
Share
Datasets

Interoperable & scalable…
GA4GH + Shar3 = Med@Scale
+ ADAM & spark
+ In Memory optimization (Tachyon)
+ Deployment (e.g. DCOS)

Wrap-UP
Follow us @DataFellas and get notified about our
+ sharing platform at scale: Shar3
+ Google Genomics At Home (^.^): Med@Scale
+ future plans: modules for Trading, Geospatial,
other medical data, …

References
Adam: https://github.com/bigdatagenomics/adam
Bdg-Formats: https://github.com/bigdatagenomics/bdg-formats
GA4GH website: http://genomicsandhealth.org/
GA4GH data working group: http://ga4gh.org/
@Spark-Notebook: https://github.com/andypetrella/spark-notebook/
Med-At-Scale: https://github.com/med-at-scale/high-health
Data Fellas: http://data-fellas.guru/
Training: http://spark4devs.data-fellas.guru/

We are at a time where biotech allow us to get personal genomes for $1000. Tremendous progress since the 70s in DNA sequencing have been done, e.g. more samples in an experiment, more genomic coverages at higher speeds. Genomic analysis standards that have been developed over the years weren't designed with scalability and adaptability in mind. In this talk, we’ll present a game changing technology in this area, ADAM, initiated by the AMPLab at Berkeley. ADAM is framework based on Apache Spark and the Parquet storage. We’ll see how it can speed up a sequence reconstruction to a factor 150.

BioBankCloud: Machine Learning on Genomics + GA4GH @ Med at Scale

Andy Petrella

Scalable Genome Analysis With ADAM

fnothaft

Managing Genomes At Scale: What We Learned - StampedeCon 2014

StampedeCon

At StampedeCon 2014, Rob Long (Monstanto) presented "Managing Genomes At Scale: What We Learned." Monsanto generates large amounts of genomic sequence data every year. Agronomists and other scientists use this data as input for predictive analytics to aid breeding and the discovery of new traits such as disease or drought resistance. In order to enable the broadest use possible of this valuable data, scientists would like to query genomic data by species, chromosome, position, and myriad other categories. We present our solutions to these problems, as realized on top of HBase here at Monsanto.We will be discussing our particular learnings around: flat/wide vs tall/narrow HBase schema design, preprocessing and caching windows of data for use in web based visualizations, approaches to complex multi-join queries across deep data sets, and distributed indexing via SolrCloud.

"Spark, DeepLearning and Life Sciences, Systems Biology in the Big Data age" Dev Lakhani, Founder of Batch Insights YouTube Link: https://www.youtube.com/watch?v=z6aTv0ZKndQ Watch more from Data Natives 2015 here: http://bit.ly/1OVkK2J Visit the conference website to learn more: www.datanatives.io Follow Data Natives: https://www.facebook.com/DataNatives https://twitter.com/DataNativesConf Stay Connected to Data Natives by Email: Subscribe to our newsletter to get the news first about Data Natives 2016: http://bit.ly/1WMJAqS About the author: Dev Lakhani has a background in Software Engineering and Computational Statistics and is a founder of Batch Insights, a Big Data consultancy that has worked on numerous Big Data architectures and data science projects in Tier 1 banking, global telecoms, retail, media and fashion. Dev has been actively working with the Hadoop infrastructure since it’s inception and is currently researching and contributing to the Apache Spark and Tachyon community.

Fast Variant Calling with ADAM and avocado

fnothaft

Spark meetup london share and analyse genomic data at scale with spark, adam...

Andy Petrella

Genomics and Health data is nowadays one of the hot topics requiring lots of computations and specially machine learning. This helps science with a very relevant societal impact to get even better outcome. That is why Apache Spark and its ADAM library is a must have. This talk will be twofold. First, we'll show how Apache Spark, MLlib and ADAM can be plugged all together to extract information from even huge and wide genomics dataset. Everything will be packed into examples from the Spark Notebook, showing how bio-scientists can work interactively with such a system. Second, we'll explain how these methodologies and even the datasets themselves can be shared at very large scale between remote entities like hospitals or laboratories using micro services leveraging Apache Spark, ADAM, Play Framework 2, Avro and Tachyon.

Scaling up genomic analysis with ADAM

fnothaft

Why is Bioinformatics a Good Fit for Spark?

Timothy Danford

DNA sequencing is producing a wave of data which will change the way that drugs are developed, patients diagnosed, and our understanding of human biology. To fulfill this promise, however, the tools for interpretation and analysis must scale to match the quantity and diversity of "big data genomics." ADAM is an open-source genomics processing engine, built using Spark, Apache Avro, and Parquet. This talk will discuss some of the advantages that the Spark platform brings to genomics, the benefits of using technologies like Parquet in conjunction with Spark, and the challenges of adapting new technologies for existing tools in bioinformatics. These are slides for a talk given at the Apache Spark Meetup in Boston on October 20, 2014.

ADAM—Spark Summit, 2014fnothaft

Spark Summit East 2015

Timothy Danford

Strata-Hadoop 2015 Presentation

Timothy Danford

Ga4 gh meeting at the the sanger instituteMatt Massie

Hadoop for Bioinformatics: Building a Scalable Variant Store

Uri Laserson

Genomics Is Not Special: Towards Data Intensive Biology

Uri Laserson

Spark, Deep Learning and Life Sciences, Systems Biology in the Big Data Age

batchinsights

Democratizing Big Semantic Data management

WU (Vienna University of Economics and Business)

From Genomics to Medicine: Advancing Healthcare at Scale

Databricks

With the exponential growth of genomic data sets, healthcare practitioners now have the opportunity to improve human outcomes at an unprecedented pace. These outcomes are difficult to realize in the existing ecosystem of genomic tools, where biostatisticians regularly chain together command-line interfaces based on a single-node setup on premise. The Databricks Unified Analytics Platform for Genomics empowers users to perform end-to-end analysis on our massively scalable platform in the cloud: in only minutes, a data scientist can visualize an individual’s disease risk based on their raw genomic data. Built on Apache Spark, we provide click-button implementations of accepted best practice workflows, as well as low-level Spark SQL optimizations for common genomics operations.

20160818 Semantics and Linkage of Archived Catalogs

andrea huang

Learning Systems for Science

Ian Foster

New learning technologies seem likely to transform much of science, as they are already doing for many areas of industry and society. We can expect these technologies to be used, for example, to obtain new insights from massive scientific data and to automate research processes. However, success in such endeavors will require new learning systems: scientific computing platforms, methods, and software that enable the large-scale application of learning technologies. These systems will need to enable learning from extremely large quantities of data; the management of large and complex data, models, and workflows; and the delivery of learning capabilities to many thousands of scientists. In this talk, I review these challenges and opportunities and describe systems that my colleagues and I are developing to enable the application of learning throughout the research process, from data acquisition to analysis.

eScience: A Transformed Scientific Method

Duncan Hull

Big Data Science with H2O in R

Anqi Fu

2010 03 Lodoxf OpenflydataJun Zhao

MongoDB and the Connectivity Map: Making Connections Between Genetics and Dis...

MongoDB

The Broad Institute has developed a novel high-throughput gene-expression profiling technology and has used it to build an open-source catalog of over a million profiles that captures the functional states of cells when treated with drugs and other types of perturbations. Referred to as the Connectivity Map (or CMap), these data when paired with pattern matching algorithms, facilitate the discovery of connections between drugs, genes and diseases. We wished to expose this resource to scientists around the world via an API that is easily accessible to programmers and biologists alike. We required a database solution that could handle a variety of data types and handle frequent changes to the schema. We realized that a relational database did not fit our needs, and gravitated towards MongoDB for its ease of use, support for dynamic schema, complex data structures and expressive query syntax. In this talk, we’ll walk through how we built the CMap library. We’ll discuss why we chose MongoDB, the various schema design iterations and tradeoffs we’ve made, how people are using the API, and what we’re planning for the next generation of biomedical data.

Tds — big science dec 2021

Gérard Dupont

https://bigscience.huggingface.co/ EN: Presentation of the BigScience project: a research initiative launched by HuggingFace and aiming to build a large language model (inspired by OpenAI and GPTx) over multiple languages and a very large processing cluster. The participants plan to investigate the dataset and the model from all angles: bias, social impact, capabilities, limitations, ethics, potential improvements, specific domain performances, carbon impact, general AI/cognitive research landscape. FR : Présentation du projet Bigscience : un projet de recherche ouvert lancé par HuggingFace et qui a pour objectif de contruire un modèle de langue (ie un peu comme openAI et GPT-3) mais en explorant les problèmes liés au jeux de données et au modèle selon les angles des biais cognitifs, de l'impact social et environemental, des limites éthiques, des possibles gain de performance et de l'impact général de ce type d'approche lorsque le but n'est pas seulement "d'avoir un plus gros modèle".

Hadoop @ Sara & BiG GridEvert Lammerts

What's hot

"Spark, Deep Learning and Life Sciences, Systems Biology in the Big Data Age"...

Dataconomy Media

Fast Variant Calling with ADAM and avocado

fnothaft

Spark meetup london share and analyse genomic data at scale with spark, adam...

Andy Petrella

Scaling up genomic analysis with ADAM

fnothaft

Why is Bioinformatics a Good Fit for Spark?

Timothy Danford

ADAM—Spark Summit, 2014fnothaft

Spark Summit East 2015

Timothy Danford

Strata-Hadoop 2015 Presentation

Timothy Danford

Ga4 gh meeting at the the sanger instituteMatt Massie

Hadoop for Bioinformatics: Building a Scalable Variant Store

Uri Laserson

Genomics Is Not Special: Towards Data Intensive Biology

Uri Laserson

Spark, Deep Learning and Life Sciences, Systems Biology in the Big Data Age

batchinsights

Democratizing Big Semantic Data management

WU (Vienna University of Economics and Business)

From Genomics to Medicine: Advancing Healthcare at Scale

Databricks

20160818 Semantics and Linkage of Archived Catalogs

andrea huang

Learning Systems for Science

Ian Foster

eScience: A Transformed Scientific Method

Duncan Hull

Big Data Science with H2O in R

Anqi Fu

2010 03 Lodoxf OpenflydataJun Zhao

MongoDB and the Connectivity Map: Making Connections Between Genetics and Dis...

MongoDB

What's hot (20)

"Spark, Deep Learning and Life Sciences, Systems Biology in the Big Data Age"...

Fast Variant Calling with ADAM and avocado

Spark meetup london share and analyse genomic data at scale with spark, adam...

Scaling up genomic analysis with ADAM

Why is Bioinformatics a Good Fit for Spark?

ADAM—Spark Summit, 2014

Spark Summit East 2015

Strata-Hadoop 2015 Presentation

Ga4 gh meeting at the the sanger institute

Hadoop for Bioinformatics: Building a Scalable Variant Store

Genomics Is Not Special: Towards Data Intensive Biology

Spark, Deep Learning and Life Sciences, Systems Biology in the Big Data Age

Democratizing Big Semantic Data management

From Genomics to Medicine: Advancing Healthcare at Scale

20160818 Semantics and Linkage of Archived Catalogs

Learning Systems for Science

eScience: A Transformed Scientific Method

Big Data Science with H2O in R

2010 03 Lodoxf Openflydata

MongoDB and the Connectivity Map: Making Connections Between Genetics and Dis...

Similar to Data Enthusiasts London: Scalable and Interoperable data services. Applied to Genomics

Tds — big science dec 2021

Gérard Dupont

Hadoop @ Sara & BiG GridEvert Lammerts

(Big) Data (Science) Skills

Oscar Corcho

Scaling the (evolving) web data –at low cost-

WU (Vienna University of Economics and Business)

Vital AI: Big Data Modeling

Vital.AI

Video: https://www.youtube.com/watch?v=Rt2oHibJT4k Technologies such as Hadoop have addressed the "Volume" problem of Big Data, and technologies such as Spark have recently addressed the "Velocity" problem – but the "Variety" problem is largely unaddressed – there is a lot of manual "data wrangling" to mange data models. These manual processes do not scale well. Not only is the variety of data increasing, also the rate of change in the data definitions is increasing. We can’t keep up. NoSQL data repositories can handle storage, but we need effective models of the data to fully utilize it. This talk will present tools and a methodology to manage Big Data Models in a rapidly changing world. This talk covers: Creating Semantic Metadata Models of Big Data Resources Graphical UI Tools for Big Data Models Tools to synchronize Big Data Models and Application Code Using NoSQL Databases, such as Amazon DynamoDB, with Big Data Models Using Big Data Models with Hadoop, Storm, Spark, Giraph, and Inference Using Big Data Models with Machine Learning to generate Predictive Models Developer Collaborative/Coordination processes using Big Data Models and Git Managing change – Big Data Models with rapidly changing Data Resources

AMP Camp 5 Intro

jeykottalam

Big Data, Beyond the Data Center

Gilles Fedak

Big Data, Beyond the Data Center Increasingly the next scientific discoveries and the next industrial innovative breakthroughs will depend on the capacity to extract knowledge and sense from gigantic amount of information. Examples vary from processing data provided by scientific instruments such as the CERN’s LHC; collecting data from large-scale sensor networks; grabbing, indexing and nearly instantaneously mining and searching the Web; building and traversing the billion-edges social network graphs; anticipating market and customer trends through multiple channels of information. Collecting information from various sources, recognizing patterns and distilling insights constitutes what is called the Big Data challenge. However, As the volume of data grows exponentially, the management of these data becomes more complex in proportion. A key challenge is to handle the complexity of data management on Hybrid distributed infrastructures, i.e assemblage of Cloud, Grid or Desktop Grids. In this talk, I will overview our works in this research area; starting with BitDew, a middleware for large scale data management on Clouds and Desktop Grids. Then I will present our approach to enable MapReduce on Desktop Grids. Finally, I will present our latest results around Active Data, a programming model for managing data life cycle on heterogeneous systems and infrastructures.

Towards a rebirth of data science (by Data Fellas)

Andy Petrella

Nowadays, Data Science is buzzing all over the place. But what is a, so-called, Data Scientist? Some will argue that a Data Scientist is a person able to report and present insights in a data set. Others will say that a Data Scientist can handle a high throughput of values and expose them in services. Yet another definition includes the capacity to create meaningful visualizations on the data. However, we enter an age where velocity is a key. Not only the velocity of your data is high, but the time to market is shortened. Hence, the time separating the moment you receive a set of data and the time you’ll be able to deliver added value is crucial. In this talk, we’ll review the legacy Data Science methodologies, what it meant in terms of delivered work and results. Afterwards, we’ll slightly move towards different concepts, techniques and tools that Data Scientists will have to learn and appropriate in order to accomplish their tasks in the age of Big Data. The dissertation is closed by exposing the Data Fellas view on a solution to the challenges, specially thanks to the Spark Notebook and the Shar3 product we develop.

Big Data Meetup #7

Paul Lo

BigData: My Learnings from data analytics at Uber Reference (highly recommended): * Designing Data-Intensive Applications http://bit.ly/big_data_architecture * Big Data and Machine Learning using Python tools http://bit.ly/big_data_machine_learning * Uber Engineering Blog http://eng.uber.com * Hadoop: The Definitive Guide: Storage and Analysis at Internet Scale http://bit.ly/hadoop_guide_bigdata

FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...

FIWARE

A Generic Scientific Data Model and Ontology for Representation of Chemical Data

Stuart Chalk

The current movement toward openness and sharing of data is likely to have a profound effect on the speed of scientific research and the complexity of questions we can answer. However, a fundamental problem with currently available datasets (and their metadata) is heterogeneity in terms of implementation, organization, and representation. To address this issue we have developed a generic scientific data model (SDM) to organize and annotate raw and processed data, and the associated metadata. This paper will present the current status of the SDM, implementation of the SDM in JSON-LD, and the associated scientific data model ontology (SDMO). Example usage of the SDM to store data from a variety of sources with be discussed along with future plans for the work.

INF2190_W1_2016_publicAttila Barta

NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...

Paolo Nesi

Abstract—The recent growth of the World Wide Web at increasing rate and speed and the number of online available resources populating Internet represent a massive source of knowledge for various research and business interests. Such knowledge is, for the most part, embedded in the textual content of web pages and documents, which is largely represented as unstructured natural language formats. In order to automatically ingest and process such huge amounts of data, single-machine, non-distributed architectures are proving to be inefficient for tasks like Big Data mining and intensive text processing and analysis. Current Natural Language Processing (NLP) systems are growing in complexity, and computational power needs have been significantly increased, requiring solutions such as distributed frameworks and parallel computing programming paradigms. This paper presents a distributed framework for executing NLP related tasks in a parallel environment. This has been achieved by integrating the APIs of the widespread GATE open source NLP platform in a multi-node cluster, built upon the open source Apache Hadoop file system. The proposed framework has been evaluated against a real corpus of web pages and documents.

Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir

Spark Summit

GeoKettle: A powerful open source spatial ETL tool

Thierry Badard

DataHub

Aditya Parameswaran

What is a distributed data science pipeline. how with apache spark and friends.

Andy Petrella

Data FAIRport Prototype & Demo - Presentation to Elsevier, Jul 10, 2015

Mark Wilkinson

The Dendro research data management platform: Applying ontologies to long-ter...

João Rocha da Silva

It has been shown that data management should start as early as possible in the research workflow to minimize the risks of data loss. Given the large numbers of datasets produced every day, curators may be unable to describe them all, so researchers should take an active part in the process. However, since they are not data management experts, they must be provided with user-friendly but powerful tools to capture the context information necessary for others to interpret and reuse their datasets. In this paper, we present Dendro, a fully ontology-based collaborative platform for research data management. Its graph data model innovates in the sense that it allows domain-specific lightweight ontologies to be used in resource description, acting as a staging area for later deposit in long-term preservation solutions.

A Gen3 Perspective of Disparate Data

Robert Grossman

Similar to Data Enthusiasts London: Scalable and Interoperable data services. Applied to Genomics (20)

Tds — big science dec 2021

Hadoop @ Sara & BiG Grid

(Big) Data (Science) Skills

Scaling the (evolving) web data –at low cost-

Vital AI: Big Data Modeling

AMP Camp 5 Intro

Big Data, Beyond the Data Center

Towards a rebirth of data science (by Data Fellas)

Big Data Meetup #7

FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...

A Generic Scientific Data Model and Ontology for Representation of Chemical Data

INF2190_W1_2016_public

NLP on Hadoop: A Distributed Framework for NLP-Based Keyword and Keyphrase Ex...

Share and analyze geonomic data at scale by Andy Petrella and Xavier Tordoir

GeoKettle: A powerful open source spatial ETL tool

DataHub

What is a distributed data science pipeline. how with apache spark and friends.

Data FAIRport Prototype & Demo - Presentation to Elsevier, Jul 10, 2015

The Dendro research data management platform: Applying ontologies to long-ter...

A Gen3 Perspective of Disparate Data

More from Andy Petrella

Data Observability Best Pracices

Andy Petrella

How to Build a Global Data Mapping

Andy Petrella

Interactive notebooks

Andy Petrella

Governance compliance

Andy Petrella

Data science governance and GDPR

Andy Petrella

Data science governance : what and how

Andy Petrella

Scala: the unpredicted lingua franca for data science

Andy Petrella

Agile data science with scala

Andy Petrella

Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...

Andy Petrella

Distributed Data Science… * A genomics use case * Spark Notebook * Interactive Distributed Data Science Distributed Data Science… Pipeline * Pipeline: productizing Data Science * Demo of Distributed Pipeline (ADAM, Akka, Cassandra, Parquet, Spark) * Why Micro Services? * Painful points: * Data science is Discontiguous * Context Lost in Translation * Solution: Data Fellas’ Agile Data Science Toolkit

Distributed machine learning 101 using apache spark from a browser devoxx.b...

Andy Petrella

Leveraging mesos as the ultimate distributed data science platform

Andy Petrella

Distributed machine learning 101 using apache spark from the browser

Andy Petrella

Liège créative: Open Science

Andy Petrella

What is Distributed Computing, Why we use Apache Spark

Andy Petrella

Spark devoxx2014Andy Petrella

Machine Learning and GraphXAndy Petrella

Quanti-litative Revolution in GIS

Andy Petrella

Scala and-fp-in-big-data

Andy Petrella

Software Crafted And Libraries Available

Andy Petrella

Fp and entrepreneurship

Andy Petrella

More from Andy Petrella (20)

Data Observability Best Pracices

How to Build a Global Data Mapping

Interactive notebooks

Governance compliance

Data science governance and GDPR

Data science governance : what and how

Scala: the unpredicted lingua franca for data science

Agile data science with scala

Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...

Distributed machine learning 101 using apache spark from a browser devoxx.b...

Leveraging mesos as the ultimate distributed data science platform

Distributed machine learning 101 using apache spark from the browser

Liège créative: Open Science

What is Distributed Computing, Why we use Apache Spark

Spark devoxx2014

Machine Learning and GraphX

Quanti-litative Revolution in GIS

Scala and-fp-in-big-data

Software Crafted And Libraries Available

Fp and entrepreneurship

Recently uploaded

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Product School

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Knowledge engineering: from people to machines and back

Elena Simperl

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

Generating a custom Ruby SDK for your web service or Rails API using Smithy

g2nightmarescribd

Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

Recently uploaded (20)

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Leading Change strategies and insights for effective change management pdf 1.pdf

Knowledge engineering: from people to machines and back

Monitoring Java Application Security with JDK Tools and JFR Events

Generating a custom Ruby SDK for your web service or Rails API using Smithy

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

How world-class product teams are winning in the AI era by CEO and Founder, P...

Elevating Tactical DDD Patterns Through Object Calisthenics

PCI PIN Basics Webinar from the Controlcase Team

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Key Trends Shaping the Future of Infrastructure.pdf

Assuring Contact Center Experiences for Your Customers With ThousandEyes

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Essentials of Automations: Optimizing FME Workflows with Parameters

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Data Enthusiasts London: Scalable and Interoperable data services. Applied to Genomics

1. by Data Fellas, Data Enthusiasts v 4.0 (July, 13th ‘15) Scalable and Interoperable data services Applied to Genomics

2. Young Belgian Startup The Data Fellas Startup Data Science Xavier Tordoir @xtordoir Andy Petrella @noootsab Data Processing Scalable Machine Learning Micro Services oriented

3. Data Fellas Ecosystem We’ve worked with

4. Data Fellas: Evangelizing Training Scala Apache Spark (BE, in September) http://spark4devs.data-fellas.guru/ Distributed Machine Learning Pipeline (Oakland, August) http://bigdatascala.bythebay.io/training.html Apache Spark (SFO with BoldRadius, August) Talks Scala IO, Devoxx Belgium, Devoxx France, Scala Days, KTH, KUL, Spark Meetup London, … more to come (Italy, …) PMC Member at Strata NY PMC member at Devoxx PMC Member at Foss4G

5. First: Data Science Analysis Spark Notebook

6. First: Data Science Analysis Production Project Generator Mesos / C* / DCOS

7. First: Data Science Analysis Production Distribution Micro Service / Binary format Marathon

8. First: Data Science Analysis Production DistributionRendering SChema for output GG / D3 …

9. First: Data Science Analysis Production DistributionRendering Discovery Service Metadata SOLR , …

10. First: Data Science Analysis Production DistributionRendering Discovery Catalog Spark Notebook using Services too

11. First: Data Science Analysis Production DistributionRendering Discovery Share Analyses Share Results Share Datasets

12. First: Data Science Project Code Name: Shar3

13. Next: Applied TO Genomics Genomics data is pretty big ● 100,000’s genomes in 2015 ● 1,000,000’s … ● 100,000,000’s … ● …

14. Next: Applied TO Genomics Genomics data is pretty big and of High dimensionality One genome: ○ 3 billions bases (basic DNA component) sequence ○ 30 - 60 x coverage for quality ○ 10’s to 100’s millions variants (variable bases from one individual to the next)

15. Next: Applied TO Genomics e.g. 1000genomes project: ● 200TB compressed data ● organised in files/directories ● data formatted following specs in a … PDF Data and services schemas are required

16. What we do with genomics data? Lots of Querying and Learning: E.G. ● Population structure is a fundamental basis ● Querying relationships between genomes and other biological features Hey… no one has all data! Metadata

17. What we do with genomics data? Lots of Querying and Learning: E.G. ● We do some specific Modelling on some data… Hey… no two serve the same computations! Service Discovery

18. Interoperability So, no one has all data … BUT all should be able to talk… Interoperability (GA4GH)

19. Interoperable… Analysis Production DistributionRendering Discovery Share Analyses Share Results Share Datasets

20. Interoperable & scalable… GA4GH + Shar3 = Med@Scale + ADAM & spark + In Memory optimization (Tachyon) + Deployment (e.g. DCOS)

21. Wrap-UP Follow us @DataFellas and get notified about our + sharing platform at scale: Shar3 + Google Genomics At Home (^.^): Med@Scale + future plans: modules for Trading, Geospatial, other medical data, …

22. References Adam: https://github.com/bigdatagenomics/adam Bdg-Formats: https://github.com/bigdatagenomics/bdg-formats GA4GH website: http://genomicsandhealth.org/ GA4GH data working group: http://ga4gh.org/ @Spark-Notebook: https://github.com/andypetrella/spark-notebook/ Med-At-Scale: https://github.com/med-at-scale/high-health Data Fellas: http://data-fellas.guru/ Training: http://spark4devs.data-fellas.guru/

23. Q/A THANKS!

Data Enthusiasts London: Scalable and Interoperable data services. Applied to Genomics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data Enthusiasts London: Scalable and Interoperable data services. Applied to Genomics

Similar to Data Enthusiasts London: Scalable and Interoperable data services. Applied to Genomics (20)

More from Andy Petrella

More from Andy Petrella (20)

Recently uploaded

Recently uploaded (20)

Data Enthusiasts London: Scalable and Interoperable data services. Applied to Genomics