Deep learning on a mixed cluster with deeplearning4j and spark

•

1 like•736 views

Deep learning models can be distributed across a cluster to speed up training time and handle large datasets. Deeplearning4j is an open-source deep learning library for Java that runs on Spark, allowing models to be trained in a distributed fashion across a Spark cluster. Training a model involves distributing stochastic gradient descent (SGD) across nodes, with the key challenge being efficient all-reduce communication between nodes. Engineering high performance distributed training, such as with parameter servers, is important to reduce bottlenecks.

Software

Deep learning on a mixed
cluster with Deeplearning4j
and Spark
Barcelona Spark meetup, Dec 9, 2016
(right after NIPS)
francois@garillot.net @huitseeker

Agenda
Intro
Why Deep Learning on a
Cluster
Big Data Architecture
Deeplearning4j
Spark challenges

Introduction : Deep Learning
in the trenches today

The bad thing about doing a
talk right after NIPS
you guys are scary.

The good thing about doing a
talk right after NIPS
You guys don't need to be told SkyNet is a fantasy (for now).

Paying algorithms
Anomaly detection in many forms (bad guys / predictive
maintenance / market rally)
Fraud detection
Network intrusion
Fintech secutiries churn prediction
Video object detection (security)

Models that are being
neglected in benchmarks and
implementation efforts
LSTMs
Autoencoders

How to deal with this in the
Spark world ?
experiment with trained model application: Tensorframes,
what are the deep learning frameworks that let you train
?

Practically ... let's look at benchmarks

Training, but how ?
New Amazon GPU instances

Cluster training in the
enterprise
it's really about multi-tenancy & economies of scale
a big bunch of machines shared among everybody shares
better
if only because you can reuse it for other workloads
Minor reasons
enterprises may not have
GPUs

Distributing training
basically distributing SGD (R)
challenge is AllReduce Communication
Sparse updates, async
communications

Distributing training : good
engineering matters

Cluster training in your
(experimentor) case ?
it's a fun problem : AllReduce
Ultimately solved for people with a large amount of images
that solution is not open-source (but at Facebook, Google,
Amazon, Microsoft¹, Baidu)
¹: 1-bit SGD is under non-commercial license in CNTK 2.0

With Spark
Spark does the initial ETL
Spark ingests the nal result
In the middle : parameter
server.

Spark cluster modes
Mesos GPU support merged
devices cgroups !
YARN GPU support through
tags
Spark Standalone : ?

Deeplearning4j
the rst commercial-grade, open-source, distributed deep-
learning library written for Java and Scala
Skymind its commercial support arm

Scienti c computing on the JVM
libnd4j : Vectorization, 32-bit addressing, linalg (BLAS!)
JavaCPP: generates JNI bindings to your CPP libs
ND4J : numpy for the JVM, native superfast arrays
Datavec : one-stop interface to an NDArray
DeepLearning4J: orchestration, backprop, layer de nition
ScalNet: gateway drug, inspired from (and closely following)
Keras
RL4J : Reinforcement learning for the JVM

With Spark
JavaSparkContent sc = ...;
JavaRDD<DataSet> trainingData = ...;
MultiLayerConfiguration networkConfig = ...;
//Create the TrainingMaster instance
int examplesPerDataSetObject = 1;
TrainingMaster trainingMaster =
new ParameterAveragingTrainingMaster.Builder(examplesPerDataSetObjec
.(other configuration options)
.build();
//Create the SparkDl4jMultiLayer instance
SparkDl4jMultiLayer sparkNetwork =
new SparkDl4jMultiLayer(sc, networkConfig, trainingMaster);
//Fit the network using the training data:
sparkNetwork.fit(trainingData);

Even if you don't care about Deep
learning
(from Kazuaki Ishizaki @ IBM Japan)
SPARK-6442 : better linear algebra than
breeze
ND4J will have sparse representations soon

Even if you don't care about Deep
learning II
Meta-
RDDs

Killing the bottlenecks
Spark has already changed its networking backend once.
better support for parameters servers and their fault
tolerance.

A Last Word (from Andrew Y. Ng)
get involved !
don't just read papers, reproduce research
results
Also
We're happy to mentor contributions, and there's a book !

At the recent Spark & Machine Learning Meetup in Brussels, François Garillot of Skymind delivered this lightning talk to a sold-out crowd. Specifically, François offered a tour of the DeepLearning4J architecture intermingled with applications. He went over the main blocks of this deep learning solution for the JVM that includes GPU acceleration, a custom n-dimensional array library, a parallelized data-loading swiss army tool, deep learning and reinforcement learning libraries — all with an easy-access interface. Along the way, he pointed out the strategic points of parallelization of computation across machines and gave insight on where Spark helps — and where it doesn't.

Apache SystemML - Declarative Large-Scale Machine Learning

Romeo Kienzler

Hands on image recognition with scala spark and deep learning4j

Guglielmo Iozzia

Strata Beijing 2017: Jumpy, a python interface for nd4j

Adam Gibson

Anomaly Detection and Automatic Labeling with Deep Learning

Adam Gibson

Anomaly detection in deep learning (Updated) English

Adam Gibson

IBM Middle East Data Science Connect 2016 - Doha, Qatar

Romeo Kienzler

Machine Intelligence at Google Scale: Tensor Flow and Cloud Machine Learning: The biggest challenge of Deep Learning technology is the scalability. As long as using single GPU server, you have to wait for hours or days to get the result of your work. This doesn’t scale for production service, so you need a Distributed Training on the cloud eventually. Google has been building infrastructure for training the large scale neural network on the cloud for years, and now started to share the technology with external developers. In this session, we will introduce new pre-trained ML services such as Cloud Vision API and Speech API that works without any training. Also, we will look how TensorFlow and Cloud Machine Learning will accelerate custom model training for 10x – 40x with Google’s distributed training infrastructure.

Deep learning in production with the best

Adam Gibson

Bringing Deep Learning into production

Paolo Platter

World Artificial Intelligence Conference Shanghai 2018

Adam Gibson

Snorkel: Dark Data and Machine Learning with Christopher Ré

Jen Aman

Building applications that can read and analyze a wide variety of data may change the way we do science and make business decisions. However, building such applications is challenging: real world data is expressed in natural language, images, or other “dark” data formats which are fraught with imprecision and ambiguity and so are difficult for machines to understand. This talk will describe Snorkel, whose goal is to make routine Dark Data and other prediction tasks dramatically easier. At its core, Snorkel focuses on a key bottleneck in the development of machine learning systems: the lack of large training datasets. In Snorkel, a user implicitly creates large training sets by writing simple programs that label data, instead of performing manual feature engineering or tedious hand-labeling of individual data items. We’ll provide a set of tutorials that will allow folks to write Snorkel applications that use Spark. Snorkel is open source on github and available from Snorkel.Stanford.edu.

Amazon Deep Learning

Amanda Mackay (she/her)

Best Practices for On-Demand HPC in Enterprises

geetachauhan

Traditionally HPC has been popular in Scientific domains, but not in most other Enterprises. With the advent of on-demand-HPC in cloud and growing adoption of Deep Learning, HPC should now be a standard platform for any Enterprise leading with AI and Machine Learning. This session will cover the best practices for building your own on-demand HPC cluster for Enterprise workloads along with key use cases where Enterprises will benefit from HPC solution.

Deep learning with Tensorflow in R

mikaelhuss

"Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power ...

Edge AI and Vision Alliance

For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2017-embedded-vision-summit-warden For more information about embedded vision, please visit: http://www.embedded-vision.com Pete Warden, Research Engineer at Google, presents the "Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power DSP" tutorial at the May 2017 Embedded Vision Summit. TensorFlow is Google’s second-generation deep learning software framework. TensorFlow was designed from the ground up to enable efficient implementation of deep learning algorithms at different scales, from high-performance data centers to low-power embedded and mobile devices. In this talk, Warden presents the technical details of how the TensorFlow and Qualcomm teams collaborated to target TensorFlow to Qualcomm’s low-power Hexagon DSP using Hexagon Vector Extensions, which enables deep learning models to run fast and efficiently. Warden explains how the two companies split up the work between them and how they measured progress with specific benchmarks, and he looks at some of the code optimizations they implemented. Since the majority of the resulting code has been open-sourced, he's able to dive deeply into the specifics of the implementation decisions they made.

Hadoop Summit 2014 - San Jose - Introduction to Deep Learning on Hadoop

Josh Patterson

As the data world undergoes its cambrian explosion phase our data tools need to become more advanced to keep pace. Deep Learning has emerged as a key tool in the non-linear arms race of machine learning. In this session we will take a look at how we parallelize Deep Belief Networks in Deep Learning on Hadoop’s next generation YARN framework with Iterative Reduce. We’ll also look at some real world examples of processing data with Deep Learning such as image classification and natural language processing.

Data Programming: Creating Large Datasets, Quickly -- Presented at JPL MLRG

Thamme Gowda

Deep Learning for Java (DL4J)

신동 강

Deep Learning for NLP (without Magic) - Richard Socher and Christopher Manning

BigDataCloud

A tutorial given at NAACL HLT 2013. Richard Socher and Christopher Manning http://nlp.stanford.edu/courses/NAACL2013/ Machine learning is everywhere in today's NLP, but by and large machine learning amounts to numerical optimization of weights for human designed representations and features. The goal of deep learning is to explore how computers can take advantage of data to develop features and representations appropriate for complex interpretation tasks. This tutorial aims to cover the basic motivation, ideas, models and learning algorithms in deep learning for natural language processing. Recently, these methods have been shown to perform very well on various NLP tasks such as language modeling, POS tagging, named entity recognition, sentiment analysis and paraphrase detection, among others. The most attractive quality of these techniques is that they can perform well without any external hand-designed resources or time-intensive feature engineering. Despite these advantages, many researchers in NLP are not familiar with these methods. Our focus is on insight and understanding, using graphical illustrations and simple, intuitive derivations. The goal of the tutorial is to make the inner workings of these techniques transparent, intuitive and their results interpretable, rather than black boxes labeled "magic here". The first part of the tutorial presents the basics of neural networks, neural word vectors, several simple models based on local windows and the math and algorithms of training via backpropagation. In this section applications include language modeling and POS tagging. In the second section we present recursive neural networks which can learn structured tree outputs as well as vector representations for phrases and sentences. We cover both equations as well as applications. We show how training can be achieved by a modified version of the backpropagation algorithm introduced before. These modifications allow the algorithm to work on tree structures. Applications include sentiment analysis and paraphrase detection. We also draw connections to recent work in semantic compositionality in vector spaces. The principle goal, again, is to make these methods appear intuitive and interpretable rather than mathematically confusing. By this point in the tutorial, the audience members should have a clear understanding of how to build a deep learning system for word-, sentence- and document-level tasks. The last part of the tutorial gives a general overview of the different applications of deep learning in NLP, including bag of words models. We will provide a discussion of NLP-oriented issues in modeling, interpretation, representational power, and optimization.

What's hot

Boolan machine learning summit

Adam Gibson

Hadoop summit 2016

Adam Gibson

Big Data Analytics Tokyo

Adam Gibson

Advanced deeplearning4j features

Adam Gibson

Self driving computers active learning workflows with human interpretable ve...

Adam Gibson

Deploying signature verification with deep learning

Adam Gibson

Deep Learning on Qubole Data Platform

Shivaji Dutta

A Primer on FPGAs - Field Programmable Gate Arrays

Taylor Riggan

DeepLearning and Advanced Machine Learning on IoT

Romeo Kienzler

Kaz Sato, Evangelist, Google at MLconf ATL 2016

MLconf

Deep learning in production with the best

Adam Gibson

Bringing Deep Learning into production

Paolo Platter

World Artificial Intelligence Conference Shanghai 2018

Adam Gibson

Snorkel: Dark Data and Machine Learning with Christopher Ré

Jen Aman

Amazon Deep Learning

Amanda Mackay (she/her)

Best Practices for On-Demand HPC in Enterprises

geetachauhan

Deep learning with Tensorflow in R

mikaelhuss

"Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power ...

Edge AI and Vision Alliance

Hadoop Summit 2014 - San Jose - Introduction to Deep Learning on Hadoop

Josh Patterson

Data Programming: Creating Large Datasets, Quickly -- Presented at JPL MLRG

Thamme Gowda

What's hot (20)

Boolan machine learning summit

Hadoop summit 2016

Big Data Analytics Tokyo

Advanced deeplearning4j features

Self driving computers active learning workflows with human interpretable ve...

Deploying signature verification with deep learning

Deep Learning on Qubole Data Platform

A Primer on FPGAs - Field Programmable Gate Arrays

DeepLearning and Advanced Machine Learning on IoT

Kaz Sato, Evangelist, Google at MLconf ATL 2016

Deep learning in production with the best

Bringing Deep Learning into production

World Artificial Intelligence Conference Shanghai 2018

Snorkel: Dark Data and Machine Learning with Christopher Ré

Amazon Deep Learning

Best Practices for On-Demand HPC in Enterprises

Deep learning with Tensorflow in R

"Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power ...

Hadoop Summit 2014 - San Jose - Introduction to Deep Learning on Hadoop

Data Programming: Creating Large Datasets, Quickly -- Presented at JPL MLRG

Viewers also liked

Deep Learning for Java (DL4J)

신동 강

Deep Learning for NLP (without Magic) - Richard Socher and Christopher Manning

BigDataCloud

RBM with DL4J for Deep Learning

신동 강

Deep Learning on HadoopDataWorks Summit

Neural Networks and Deep Learning

Asim Jalis

Distributed Deep Learning on Hadoop Clusters

DataWorks Summit/Hadoop Summit

Deep Learning for Personalized Search and Recommender Systems

Benjamin Le

Viewers also liked (7)

Deep Learning for Java (DL4J)

Deep Learning for NLP (without Magic) - Richard Socher and Christopher Manning

RBM with DL4J for Deep Learning

Deep Learning on Hadoop

Neural Networks and Deep Learning

Distributed Deep Learning on Hadoop Clusters

Deep Learning for Personalized Search and Recommender Systems

Similar to Deep learning on a mixed cluster with deeplearning4j and spark

Deep Learning with DL4J on Apache Spark: Yeah it's Cool, but are You Doing it...

DataWorks Summit

DeepLearning4J (DL4J) is a powerful Open Source distributed framework that brings Deep Learning to the JVM (it can serve as a DIY tool for Java, Scala, Clojure and Kotlin programmers). It can be used on distributed GPUs and CPUs. It is integrated with Hadoop and Apache Spark. ND4J is a Open Source, distributed and GPU-enabled library that brings the intuitive scientific computing tools of the Python community to the JVM. Training neural network models using DL4J, ND4J and Spark is a powerful combination, but the overall cluster configuration can present some unespected issues that can compromise performances and nullify the benefits of well written code and good model design. In this talk I will walk through some of those problems and will present some best practices to prevent them. The presented use cases will refer to DL4J and ND4J on different Spark deployment modes (standalone, YARN, Kubernetes). The reference programming language for any code example would be Scala, but no preliminary Scala knowledge is mandatory in order to better understanding the presented topics.

Machine Learning with Scala

Susan Eraly

BigDL webinar - Deep Learning Library for Spark

DESMOND YUEN

Deep Learning with DL4J on Apache Spark: Yeah it’s Cool, but are You Doing it...

Databricks

DeepLearning4J (DL4J) is a powerful Open Source distributed framework that brings Deep Learning to the JVM (it can serve as a DIY tool for Java, Scala, Clojure and Kotlin programmers). It can be used on distributed GPUs and CPUs. It is integrated with Hadoop and Apache Spark. ND4J is a Open Source, distributed and GPU-enabled library that brings the intuitive scientific computing tools of the Python community to the JVM. Training neural network models using DL4J, ND4J and Spark is a powerful combination, but it presents some unexpected issues that can compromise performance and nullify the benefits of well written code and good model design. In this talk I will walk through some of those problems and will present some best practices to prevent them, coming from lessons learned when putting things in production.

Image Recognition on AWS with Apache Spark and BigDL

Amazon Web Services

by Dave Nielsen, Technical Program Manager, Big Data Technologies, STO, Intel In this workshop we explain how to use Deep Learning to recognize objects in photos using the BigDL framework for Apache Spark. The workshop starts by explaining how Convolution Neural Network models like SSD, VGG and Inception identify edges and then higher level characteristics in multiple layers in order to identify the object. Then we will describe how to build an end-to-end pipeline for image classification including data prep, model building, training, scoring. The workshop includes a Zeppelin Notebook with python code you can modify to train the model and test for accuracy. The code provided in this workshop can be used to run examples on your laptop, on-premises or on Amazon Web Services managed by Qubole. Level 100

Sjug #26 ml is in java but is dl too - ver1.04 - tomasz sikora 2018-03-23

Tomasz Sikora

Vint Cerf, recognised as one of "the fathers of the Internet" as co-inventor of TCP/IP, said "And programming computers was so fascinating. You create your own little universe, and then it does what you tell it to do.". Now when computers are learning these words give different ground to debate on developer context here. So what can we do from old good Software Craftsmanship perspective? I, as a developer who still believes in Java, can I use my beloved platform? How can I tackle problems requiring deeper models? On this session Tomasz tried to answer those, opening 100s more questions ;). He executed and configured a few models trying to explore the field from as practical perspective as possible. To get more visit https://twitter.com/tomaszsikora and http://silesia.jug.pl/

DeepLearning4J and Spark: Successes and Challenges - François Garillot

sparktc

At the recent sold-out Spark & Machine Learning Meetup in Brussels, François Garillot of Skymind delivered a lightning talk called DeepLearning4J and Spark: Successes and Challenges. Specifically, François offered a tour of the DeepLearning4J architecture intermingled with applications. He went over the main blocks of this deep learning solution for the JVM that includes GPU acceleration, a custom n-dimensional array library, a parallelized data-loading swiss army tool, deep learning and reinforcement learning libraries — all with an easy-access interface. Along the way, he pointed out the strategic points of parallelization of computation across machines and gave insight on where Spark helps — and where it doesn't.

DeepLearning4J and Spark: Successes and Challenges - François Garillot

sparktc

APACHE SPARK PER IL MACHINE LEARNING: INTRODUZIONE ED UN CASO DI STUDIO_ Meet...

Deep Learning Italia

Abstract: Apache Spark è una piattaforma di grande successo nell’ambito del Big Data processing, grazie alla sua grande scalabilità, alle elevate prestazioni ed alla semplicità di utilizzo (favorita dalla disponibilità di API in Python ed in R, oltre che in Java e Scala). La disponibilità della libreria nativa MLLIB, con algoritmi ottimizzati per il calcolo distribuito, rende Spark uno strumento di grande interesse nell’ambito del Machine Learning. In questa sessione verrà introdotta la struttura e le principali caratteristiche di Spark. Verrà inoltre illustrato un esempio di Machine Learning supervisionato su un problema di bioinformatica.

Using Deep Learning to do Real-Time Scoring in Practical Applications - 2015-...

Greg Makowski

Started with-apache-spark

Happiest Minds Technologies

Neural Networks, Spark MLlib, Deep Learning

Asim Jalis

Running Emerging AI Applications on Big Data Platforms with Ray On Apache Spark

Databricks

Integrating Deep Learning Libraries with Apache Spark

Databricks

The combination of deep learning with Apache Spark has the potential to make a huge impact. Joseph Bradley and Xiangrui Meng share best practices for integrating popular deep learning libraries with Apache Spark. Rather than comparing deep learning systems or specific optimizations, Joseph and Xiangrui focus on issues that are common to many deep learning frameworks when running on a Spark cluster, such as optimizing cluster setup and data ingest (clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker), configuring the cluster (setting up pipelines for efficient data ingest improves job throughput), and monitoring long-running jobs (interactive monitoring facilitates both the work of configuration and checking the stability of deep learning jobs). Joseph and Xiangrui then demonstrate the techniques using Google’s popular TensorFlow library.

Deep learning for FinTech

geetachauhan

Jeremy Nixon, Machine Learning Engineer, Spark Technology Center at MLconf AT...

MLconf

Convolutional Neural Networks at scale in Spark MLlib: Jeremy Nixon will focus on the engineering and applications of a new algorithm built on top of MLlib. The presentation will focus on the methods the algorithm uses to automatically generate features to capture nonlinear structure in data, as well as the process by which it’s trained. Major aspects of that include compositional transformations over the data, convolution, and distributed backpropagation via SGD with adaptive gradients and an adaptive learning rate. Applications will look into how to use convolutional neural networks to model data in computer vision, natural language and signal processing. Details around optimal preprocessing, the type of structure that can be learned, and managing its ability to generalize will inform developers looking to apply nonlinear modeling tools to problems that they face.

Build a deep learning pipeline on apache spark for ads optimization

Craig Chao

Meetup deeplearningitalia-milano-valerio-morfino

Deep Learning Italia

"Machine Learning with Apache Spark: from lab to scale" Abstract: Apache Spark è una piattaforma di grande successo nell’ambito del Big Data processing, grazie all’elevata scalabilità, alle prestazioni ed alla semplicità di utilizzo. In questo talk verranno introdotte la struttura di Apache Spark, le principali caratteristiche, la libreria MLLIB per il Machine Learning. Le potenzialità di Spark verranno illustrate tramite l’analisi di un caso di Machine Learning supervisionato su un problema di bioinformatica. Infine, verranno illustrate delle possibili architetture per applicare il modello a grandi quantità di dati.

BigDL: Bringing Ease of Use of Deep Learning for Apache Spark with Jason Dai ...

Databricks

BigDL is a distributed deep learning framework for Apache Spark open sourced by Intel. BigDL helps make deep learning more accessible to the Big Data community, by allowing them to continue the use of familiar tools and infrastructure to build deep learning applications. With BigDL, users can write their deep learning applications as standard Spark programs, which can then directly run on top of existing Spark or Hadoop clusters. In this session, we will introduce BigDL, how our customers use BigDL to build End to End ML/DL applications, platforms on which BigDL is deployed and also provide an update on the latest improvements in BigDL v0.1, and talk about further developments and new upcoming features of BigDL v0.2 release (e.g., support for TensorFlow models, 3D convolutions, etc.).

AI and Spark - IBM Community AI Day

Nick Pentreath

Apache Spark’s machine learning library provides a simple, elegant, yet powerful framework for creating scalable machine learning pipelines. It provides out of the box components for feature extraction and transformation, as well as various machine learning algorithms. However, in recent years specialized systems (such as TensorFlow, Caffe, PyTorch and Apache MXNet) have been dominant in the domain of AI and deep learning, as they allow greater performance and flexibility for training complex models. While there are a few deep learning frameworks that are Spark specific, in most cases these frameworks are separate from Spark and the ease of integration and feature set exposed varies considerably. This session will explore the role of Spark within the AI landscape, the current state of deep learning on top of Spark and the most recent developments in the Spark project to better integrate Spark with the deep learning ecosystem.

Similar to Deep learning on a mixed cluster with deeplearning4j and spark (20)

Deep Learning with DL4J on Apache Spark: Yeah it's Cool, but are You Doing it...

Machine Learning with Scala

BigDL webinar - Deep Learning Library for Spark

Deep Learning with DL4J on Apache Spark: Yeah it’s Cool, but are You Doing it...

Image Recognition on AWS with Apache Spark and BigDL

Sjug #26 ml is in java but is dl too - ver1.04 - tomasz sikora 2018-03-23

DeepLearning4J and Spark: Successes and Challenges - François Garillot

APACHE SPARK PER IL MACHINE LEARNING: INTRODUZIONE ED UN CASO DI STUDIO_ Meet...

Using Deep Learning to do Real-Time Scoring in Practical Applications - 2015-...

Started with-apache-spark

Neural Networks, Spark MLlib, Deep Learning

Running Emerging AI Applications on Big Data Platforms with Ray On Apache Spark

Integrating Deep Learning Libraries with Apache Spark

Deep learning for FinTech

Jeremy Nixon, Machine Learning Engineer, Spark Technology Center at MLconf AT...

Build a deep learning pipeline on apache spark for ads optimization

Meetup deeplearningitalia-milano-valerio-morfino

BigDL: Bringing Ease of Use of Deep Learning for Apache Spark with Jason Dai ...

AI and Spark - IBM Community AI Day

More from François Garillot

Growing Your Types Without Growing Your Workload

François Garillot

Mobility insights at Swisscom - Understanding collective mobility in Switzerland

François Garillot

Swisscom is the leading mobile-service provider in Switzerland, with a market share high enough to enable us to model and understand the collective mobility in every area of the country. To accomplish that, we built an urban planning tool that helps cities better manage their infrastructure based on data-based insights, produced with Apache Spark, YARN, Kafka and a good dose of machine learning. In this talk, we will explain how building such a tool involves mining a massive amount of raw data (1.5E9 records/day) to extract fine-grained mobility features from raw network traces. These features are obtained using different machine learning algorithms. For example, we built an algorithm that segments a trajectory into mobile and static periods and trained classifiers that enable us to distinguish between different means of transport. As we sketch the different algorithmic components, we will present our approach to continuously run and test them, which involves complex pipelines managed with Oozie and fuelled with ground truth data. Finally, we will delve into the streaming part of our analytics and see how network events allow Swisscom to understand the characteristics of the flow of people on roads and paths of interest. This requires making a link between network coverage information and geographical positioning in the space of milliseconds and using Spark streaming with libraries that were originally designed for batch processing. We will conclude on the advantages and pitfalls of Spark involved in running this kind of pipeline on a multi-tenant cluster. Audiences should come back from this talk with an overall picture of the use of Apache Spark and related components of its ecosystem in the field of trajectory mining.

Delivering near real time mobility insights at swisscom

François Garillot

Spark Streaming : Dealing with State

François Garillot

A Gentle Introduction to Locality Sensitive Hashing with Apache Spark

François Garillot

Ramping up your Devops Fu for Big Data developers

François Garillot

Diving In The Deep End Of The Big Data Pool

François Garillot

Scala Collections : Java 8 on Steroids

François Garillot

More from François Garillot (8)

Growing Your Types Without Growing Your Workload

Mobility insights at Swisscom - Understanding collective mobility in Switzerland

Delivering near real time mobility insights at swisscom

Spark Streaming : Dealing with State

A Gentle Introduction to Locality Sensitive Hashing with Apache Spark

Ramping up your Devops Fu for Big Data developers

Diving In The Deep End Of The Big Data Pool

Scala Collections : Java 8 on Steroids

Recently uploaded

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...

Globus

The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.

Cracking the code review at SpringIO 2024

Paco van Beckhoven

Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production. Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process? In this session we will cover: - The Art of Effective Code Reviews - Streamlining the Review Process - Elevating Reviews with Automated Tools By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces

Large Language Models and the End of Programming

Matt Welsh

May Marketo Masterclass, London MUG May 22 2024.pdf

Adele Miller

Using IESVE for Room Loads Analysis - Australia & New Zealand

IES VE

Accelerate Enterprise Software Engineering with Platformless

WSO2

Key takeaways: Challenges of building platforms and the benefits of platformless. Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience. How Choreo enables the platformless experience. How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo. Demo of an end-to-end app built and deployed on Choreo.

Enterprise Resource Planning System in Telangana

NYGGS Automation Suite

Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics. To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/

Vitthal Shirke Microservices Resume Montevideo

Vitthal Shirke

How to Position Your Globus Data Portal for Success Ten Good Practices

Globus

Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.

Cyaniclab : Software Development Agency Portfolio.pdf

Cyanic lab

CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.

Globus Compute Introduction - GlobusWorld 2024

Globus

Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf

AMB-Review

Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos https://www.amb-review.com/tubetrivia-ai Exclusive Features: AI-Powered Questions, Wide Range of Categories, Adaptive Difficulty, User-Friendly Interface, Multiplayer Mode, Regular Updates. #TubeTriviaAI #QuizVideoMagic #ViralQuizVideos #AIQuizGenerator #EngageExciteExplode #MarketingRevolution #BoostYourTraffic #SocialMediaSuccess #AIContentCreation #UnlimitedTraffic

RISE with SAP and Journey to the Intelligent Enterprise

Srikant77

Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...

Globus

The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.

Lecture 1 Introduction to games development

abdulrafaychaudhry

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

Mind IT Systems

Orion Context Broker introduction 20240604

Fermin Galan

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

Google

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-pilot-review/ AI Pilot Review: Key Features ✅Deploy AI expert bots in Any Niche With Just A Click ✅With one keyword, generate complete funnels, websites, landing pages, and more. ✅More than 85 AI features are included in the AI pilot. ✅No setup or configuration; use your voice (like Siri) to do whatever you want. ✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It… ✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again. ✅ZERO Limits On Features Or Usages ✅Use Our AI-powered Traffic To Get Hundreds Of Customers ✅No Complicated Setup: Get Up And Running In 2 Minutes ✅99.99% Up-Time Guaranteed ✅30 Days Money-Back Guarantee ✅ZERO Upfront Cost See My Other Reviews Article: (1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review

BoxLang: Review our Visionary Licenses of 2024

Ortus Solutions, Corp

Into the Box 2024 - Keynote Day 2 Slides.pdf

Ortus Solutions, Corp

Recently uploaded (20)

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...

Cracking the code review at SpringIO 2024

Large Language Models and the End of Programming

May Marketo Masterclass, London MUG May 22 2024.pdf

Using IESVE for Room Loads Analysis - Australia & New Zealand

Accelerate Enterprise Software Engineering with Platformless

Enterprise Resource Planning System in Telangana

Vitthal Shirke Microservices Resume Montevideo

How to Position Your Globus Data Portal for Success Ten Good Practices

Cyaniclab : Software Development Agency Portfolio.pdf

Globus Compute Introduction - GlobusWorld 2024

Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf

RISE with SAP and Journey to the Intelligent Enterprise

Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...

Lecture 1 Introduction to games development

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

Orion Context Broker introduction 20240604

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

BoxLang: Review our Visionary Licenses of 2024

Into the Box 2024 - Keynote Day 2 Slides.pdf

Deep learning on a mixed cluster with deeplearning4j and spark

1. Deep learning on a mixed cluster with Deeplearning4j and Spark Barcelona Spark meetup, Dec 9, 2016 (right after NIPS) francois@garillot.net @huitseeker

2. Agenda Intro Why Deep Learning on a Cluster Big Data Architecture Deeplearning4j Spark challenges

3. Introduction : Deep Learning in the trenches today

4. The bad thing about doing a talk right after NIPS you guys are scary.

5. The good thing about doing a talk right after NIPS You guys don't need to be told SkyNet is a fantasy (for now).

6. Paying algorithms Anomaly detection in many forms (bad guys / predictive maintenance / market rally) Fraud detection Network intrusion Fintech secutiries churn prediction Video object detection (security)

7. Models that are being neglected in benchmarks and implementation efforts LSTMs Autoencoders

8. How to deal with this in the Spark world ? experiment with trained model application: Tensorframes, what are the deep learning frameworks that let you train ?

9. Why Deep Learning on a cluster ?

10. Practically ... let's look at benchmarks

11. Practically ... let's look at benchmarks

12. Practically ... let's look at benchmarks

13. Practically ... let's look at benchmarks

14. Training, but how ? New Amazon GPU instances

15. Training, but how ?

16. Training, but how ?

17. Cluster training in the enterprise it's really about multi-tenancy & economies of scale a big bunch of machines shared among everybody shares better if only because you can reuse it for other workloads Minor reasons enterprises may not have GPUs

18. Distributing training basically distributing SGD (R) challenge is AllReduce Communication Sparse updates, async communications

19. Distributing training : good engineering matters

20. Cluster training in your (experimentor) case ? it's a fun problem : AllReduce Ultimately solved for people with a large amount of images that solution is not open-source (but at Facebook, Google, Amazon, Microsoft¹, Baidu) ¹: 1-bit SGD is under non-commercial license in CNTK 2.0

21. Big Data architecture

22. With a parameter server

23. With Spark Spark does the initial ETL Spark ingests the nal result In the middle : parameter server.

24. Spark cluster modes Mesos GPU support merged devices cgroups ! YARN GPU support through tags Spark Standalone : ?

25. Deeplearning4j

26. Deeplearning4j the rst commercial-grade, open-source, distributed deeplearning library written for Java and Scala Skymind its commercial support arm

27. Scienti c computing on the JVM libnd4j : Vectorization, 32-bit addressing, linalg (BLAS!) JavaCPP: generates JNI bindings to your CPP libs ND4J : numpy for the JVM, native superfast arrays Datavec : one-stop interface to an NDArray DeepLearning4J: orchestration, backprop, layer de nition ScalNet: gateway drug, inspired from (and closely following) Keras RL4J : Reinforcement learning for the JVM

28. With Spark JavaSparkContent sc = ...; JavaRDD<DataSet> trainingData = ...; MultiLayerConfiguration networkConfig = ...; //Create the TrainingMaster instance int examplesPerDataSetObject = 1; TrainingMaster trainingMaster = new ParameterAveragingTrainingMaster.Builder(examplesPerDataSetObjec .(other configuration options) .build(); //Create the SparkDl4jMultiLayer instance SparkDl4jMultiLayer sparkNetwork = new SparkDl4jMultiLayer(sc, networkConfig, trainingMaster); //Fit the network using the training data: sparkNetwork.fit(trainingData);

29. Spark Challenges

30. Even if you don't care about Deep learning (from Kazuaki Ishizaki @ IBM Japan) SPARK-6442 : better linear algebra than breeze ND4J will have sparse representations soon

31. Even if you don't care about Deep learning II Meta- RDDs

32. Killing the bottlenecks Spark has already changed its networking backend once. better support for parameters servers and their fault tolerance.

33. A Last Word (from Andrew Y. Ng) get involved ! don't just read papers, reproduce research results Also We're happy to mentor contributions, and there's a book !

34. Questions ?

Deep learning on a mixed cluster with deeplearning4j and spark

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Deep learning on a mixed cluster with deeplearning4j and spark

Similar to Deep learning on a mixed cluster with deeplearning4j and spark (20)

More from François Garillot

More from François Garillot (8)

Recently uploaded

Recently uploaded (20)

Deep learning on a mixed cluster with deeplearning4j and spark