DeepLearning4J and Spark: Successes and Challenges - François Garillot

Advanced deeplearning4j features

Hadoop summit 2016

Big Data Processing Above and Beyond Hadoop: Data-intensive computing represents a new computing paradigm to address Big Data processing requirements using high-performance architectures supporting scalable parallel processing to allow government, commercial organizations, and research environments to process massive amounts of data and implement new applications previously thought to be impractical or infeasible. The fundamental challenges of data-intensive computing are managing and processing exponentially growing data volumes, significantly reducing associated data analysis cycles to support practical, timely applications, and developing new algorithms which can scale to search and process massive amounts of data. The open source HPCC (High-Performance Computing Cluster) Systems platform offers a unified approach to Big Data processing requirements: (1) a scalable, integrated computer systems hardware and software architecture designed for parallel processing of data-intensive computing applications, and (2) a new programming paradigm in the form of a high-level, declarative, data-centric programming language designed specifically for big data processing. This presentation explores the challenges of data-intensive computing from a programming perspective, and describes the ECL programming language and the HPCC architecture designed for data-intensive computing applications. HPCC is an alternative to the Hadoop platform, and ECL is compared to Pig Latin, a high-level language developed for the Hadoop MapReduce architecture.

Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016

Self driving computers active learning workflows with human interpretable ve...

Skymind Open Power Summit ISV Round Table

DeepLearning and Advanced Machine Learning on IoT

Romeo Kienzler

Deploying signature verification with deep learning

A Primer on FPGAs - Field Programmable Gate Arrays

Taylor Riggan

Future of ai on the jvm

Deep Learning on Qubole Data Platform

Shivaji Dutta

Deep learning in production with the best

What we call the public cloud was developed primarily to manage and deploy web servers. The target audience for these products is Dev Ops. While this is a massive and exciting market, the world of Data Science and Deep Learning is very different — and possibly even bigger. Unfortunately, the tools available today are not designed for this new audience and the cloud needs to evolve. This talk would cover what the next 10 years of cloud computing will look like.

Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep...

Databricks

Machine Intelligence at Google Scale: Tensor Flow and Cloud Machine Learning: The biggest challenge of Deep Learning technology is the scalability. As long as using single GPU server, you have to wait for hours or days to get the result of your work. This doesn’t scale for production service, so you need a Distributed Training on the cloud eventually. Google has been building infrastructure for training the large scale neural network on the cloud for years, and now started to share the technology with external developers. In this session, we will introduce new pre-trained ML services such as Cloud Vision API and Speech API that works without any training. Also, we will look how TensorFlow and Cloud Machine Learning will accelerate custom model training for 10x – 40x with Google’s distributed training infrastructure.

Kaz Sato, Evangelist, Google at MLconf ATL 2016

Why Machine Learning Algorithms Fall Short (And What You Can Do About It): Many think that machine learning is all about the algorithms. Want a self-learning system? Get your data, start coding or hire a PhD that will build you a model that will stand the test of time. Of course we know that this is not enough. Models degrade over time, algorithms that work great on yesterday’s data may not be the best option, new data sources and types are made available. In short, your self-learning system may not be learning anything at all. In this session, we will examine how to overcome challenges in creating self-learning systems that perform better and are built to stand the test of time. We will show how to apply mathematical optimization algorithms that often prove superior to local optimization methods favored by typical machine learning applications and discuss why these methods can crate better results. We will also examine the role of smart automation in the context of machine learning and how smart automation can create self-learning systems that are built to last.

Jean-François Puget, Distinguished Engineer, Machine Learning and Optimizatio...

For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/may-2017-embedded-vision-summit-warden For more information about embedded vision, please visit: http://www.embedded-vision.com Pete Warden, Research Engineer at Google, presents the "Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power DSP" tutorial at the May 2017 Embedded Vision Summit. TensorFlow is Google’s second-generation deep learning software framework. TensorFlow was designed from the ground up to enable efficient implementation of deep learning algorithms at different scales, from high-performance data centers to low-power embedded and mobile devices. In this talk, Warden presents the technical details of how the TensorFlow and Qualcomm teams collaborated to target TensorFlow to Qualcomm’s low-power Hexagon DSP using Hexagon Vector Extensions, which enables deep learning models to run fast and efficiently. Warden explains how the two companies split up the work between them and how they measured progress with specific benchmarks, and he looks at some of the code optimizations they implemented. Since the majority of the resulting code has been open-sourced, he's able to dive deeply into the specifics of the implementation decisions they made.

"Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power ...

Edge AI and Vision Alliance

The Potential of GPU-driven High Performance Data Analytics in Spark

Spark Summit

Convolutional Neural Networks at scale in Spark MLlib: Jeremy Nixon will focus on the engineering and applications of a new algorithm built on top of MLlib. The presentation will focus on the methods the algorithm uses to automatically generate features to capture nonlinear structure in data, as well as the process by which it’s trained. Major aspects of that include compositional transformations over the data, convolution, and distributed backpropagation via SGD with adaptive gradients and an adaptive learning rate. Applications will look into how to use convolutional neural networks to model data in computer vision, natural language and signal processing. Details around optimal preprocessing, the type of structure that can be learned, and managing its ability to generalize will inform developers looking to apply nonlinear modeling tools to problems that they face.

Jeremy Nixon, Machine Learning Engineer, Spark Technology Center at MLconf AT...

Building applications that can read and analyze a wide variety of data may change the way we do science and make business decisions. However, building such applications is challenging: real world data is expressed in natural language, images, or other “dark” data formats which are fraught with imprecision and ambiguity and so are difficult for machines to understand. This talk will describe Snorkel, whose goal is to make routine Dark Data and other prediction tasks dramatically easier. At its core, Snorkel focuses on a key bottleneck in the development of machine learning systems: the lack of large training datasets. In Snorkel, a user implicitly creates large training sets by writing simple programs that label data, instead of performing manual feature engineering or tedious hand-labeling of individual data items. We’ll provide a set of tutorials that will allow folks to write Snorkel applications that use Spark. Snorkel is open source on github and available from Snorkel.Stanford.edu.

Snorkel: Dark Data and Machine Learning with Christopher Ré

Jen Aman

Does more data always improve ML models? Is it better to use distributed ML instead of single node ML? In this talk I will show that while more data often improves DL models in high variance problem spaces (with semi or unstructured data) such as NLP, image, video more data does not significantly improve high bias problem spaces where traditional ML is more appropriate. Additionally, even in the deep learning domain, single node models can still outperform distributed models via transfer learning. Data scientists have pain points running many models in parallel automating the experimental set up. Getting others (especially analysts) within an organization to use their models Databricks solves these problems using pandas udfs, ml runtime and MLflow.

Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas

Databricks

What's hot (20)

Big Data Analytics Tokyo

Advanced deeplearning4j features

Hadoop summit 2016

Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016

Self driving computers active learning workflows with human interpretable ve...

Skymind Open Power Summit ISV Round Table

DeepLearning and Advanced Machine Learning on IoT

Deploying signature verification with deep learning

A Primer on FPGAs - Field Programmable Gate Arrays

Future of ai on the jvm

Deep Learning on Qubole Data Platform

Deep learning in production with the best

Cloud Computing Was Built for Web Developers—What Does v2 Look Like for Deep...

Kaz Sato, Evangelist, Google at MLconf ATL 2016

Jean-François Puget, Distinguished Engineer, Machine Learning and Optimizatio...

"Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power ...

The Potential of GPU-driven High Performance Data Analytics in Spark

Jeremy Nixon, Machine Learning Engineer, Spark Technology Center at MLconf AT...

Snorkel: Dark Data and Machine Learning with Christopher Ré

Distributed Models Over Distributed Data with MLflow, Pyspark, and Pandas

Viewers also liked

Introduction to Deeplearning4j

Daehyun Kim

These slides accompanied a demo of Deeplearning4j, while the meetup explored distributed clustering and various deep learning explanations. http://www.meetup.com/SF-Neural-Network-Afficianados-Discussion-Group/events/182645252/ Deep-learning is useful in detecting anomalies like fraud, spam and money laundering; identifying similarities to augment search and text analytics; predicting customer lifetime value and churn; recognizing faces and voices. Deeplearning4j is an infinitely scalable deep-learning architecture suitable for Hadoop and other big-data structures. It includes a distributed deep-learning framework and a normal deep-learning framework; i.e. it runs on a single thread as well. Training takes place in the cluster, which means it can process massive amounts of data. Nets are trained in parallel via iterative reduce, and they are equally compatible with Java, Scala and Clojure. The distributed deep-learning framework is made for data input and neural net training at scale, and its output should be highly accurate predictive models. The framework's neural nets include restricted Boltzmann machines, deep-belief networks, deep autoencoders, convolutional nets and recursive neural tensor networks.

Ersatz meetup - DeepLearning4j Demo

Video replay: http://nvidia.fullviewmedia.com/siggraph2012/ondemand/SS106.html Location: West Hall Meeting Room 503, Los Angeles Convention Center Date: Wednesday, August 8, 2012 Time: 2:40 PM – 3:40 PM The future of GPU-based visual computing integrates the web, resolution-independent 2D graphics, and 3D to maximize interactivity and quality while minimizing consumed power. See what NVIDIA is doing today to accelerate resolution-independent 2D graphics for web content. This presentation explains NVIDIA's unique "stencil, then cover" approach to accelerating path rendering with OpenGL and demonstrates the wide variety of web content that can be accelerated with this approach. More information: http://developer.nvidia.com/nv-path-rendering

A Segmentation of Water Consumption with Apache Spark

Diego García Valverde

SIGGRAPH 2012: GPU-Accelerated 2D and Web Rendering

Mark Kilgard

At StampedeCon 2014, John Tran of NVIDIA presented "GPUs in Big Data." Modern graphics processing units (GPUs) are massively parallel general-purpose processors that are taking Big Data by storm. In terms of power efficiency, compute density, and scalability, it is clear now that commodity GPUs are the future of parallel computing. In this talk, we will cover diverse examples of how GPUs are revolutionizing Big Data in fields such as machine learning, databases, genomics, and other computational sciences.

GPUs in Big Data - StampedeCon 2014

StampedeCon

GPU Ecosystem

Ofer Rosenberg

Presented at the GPU Technology Conference 2012 in San Jose, California. Tuesday, May 15, 2012. Standards such as Scalable Vector Graphics (SVG), PostScript, TrueType outline fonts, and immersive web content such as Flash depend on a resolution-independent 2D rendering paradigm that GPUs have not traditionally accelerated. This tutorial explains a new opportunity to greatly accelerate vector graphics, path rendering, and immersive web standards using the GPU. By attending, you will learn how to write OpenGL applications that accelerate the full range of path rendering functionality. Not only will you learn how to render sophisticated 2D graphics with OpenGL, you will learn to mix such resolution-independent 2D rendering with 3D rendering and do so at dynamic, real-time rates.

GTC 2012: GPU-Accelerated Path Rendering

Mark Kilgard

PG-Strom - GPU Accelerated Asyncr

Kohei KaiGai

Matrix factorization (MF) is widely used in recommendation systems. We present cuMF, a highly-optimized matrix factorization tool with supreme performance on graphics processing units (GPUs) by fully utilizing the GPU compute power and minimizing the overhead of data movement. Firstly, we introduce a memory-optimized alternating least square (ALS) method by reducing discontiguous memory access and aggressively using registers to reduce memory latency. Secondly, we combine data parallelism with model parallelism to scale to multiple GPUs. Results show that with up to four GPUs on one machine, cuMF can be up to ten times as fast as those on sizable clusters on large scale problems, and has impressively good performance when solving the largest matrix factorization problem ever reported.

Accelerating Machine Learning Applications on Spark Using GPUs

IBM

Computational Techniques for the Statistical Analysis of Big Data in R

herbps10

Deep learning on spark

Satyendra Rana

From social networks to protein networks to financial transactions, graphs are everywhere. Graph Analytics represent a key tool for data science to take advance of this type of network information. Many “Bigdata” and NoSQL techniques for analysis and data science that work well for relational and structured data, do not scale effectively when applied to challenges in graph analytics and traversal algorithms. The data locality and graph access patterns challenge existing HW architectures and place a premium on bandwidth to main memory.GPUs currently have 10X advantage over CPUs in this area. The advantage is projected to grow to 100X by 2016. This talk will discuss why GPUs are game-changer by dramatically improving the price-performance ratio for very large graph analytics over existing technologies. It will present results for work in GPU Acceleration of graph analytics within both research and industry applications.

Enabling Graph Analytics at Scale: The Opportunity for GPU-Acceleration of D...

odsc

In this video from SC13, Vinod Tipparaju presents an Heterogeneous System Architecture Overview. "The HSA Foundation seeks to create applications that seamlessly blend scalar processing on the CPU, parallel processing on the GPU, and optimized processing on the DSP via high bandwidth shared memory access enabling greater application performance at low power consumption. The Foundation is defining key interfaces for parallel computation utilizing CPUs, GPUs, DSPs, and other programmable and fixed-function devices, thus supporting a diverse set of high-level programming languages and creating the next generation in general-purpose computing." Learn more: http://hsafoundation.com/ Watch the video presentation: http://wp.me/p3RLHQ-aXk

Heterogeneous System Architecture Overview

inside-BigData.com

PG-Strom - GPGPU meets PostgreSQL, PGcon2015

Kohei KaiGai

Wendell Kuling works as a Data Scientist at ING in the Wholesale Banking Advanced Analytics team. Their projects aim to provide better services to corporate customers of ING, by using innovative techniques from data-science. In this talk, Wendell covers key insights from their experience in matching large datasets based on names. After covering the key algorithms and packages ING uses for name matching, Wendell will share his best-practice approach in applying these algorithms at scale… would you bet on a Cruncher (48-CPU/512 MB RAM machine), a Tesla (Cuda Tesla K80 with 4992 cores, 24GB memory) or a Spark cluster (80 cores/2,5 TB memory)?

PyData Amsterdam - Name Matching at Scale

GoDataDriven

Hadoop + GPU

Vladimir Starostenkov

Deep Learning on Hadoop

DataWorks Summit

From Machine Learning to Learning Machines: Creating an End-to-End Cognitive ...

Spark Summit

Containerizing GPU Applications with Docker for Scaling to the Cloud

Subbu Rama

How to Solve Real-Time Data Problems

IBM Power Systems

Viewers also liked (20)

Introduction to Deeplearning4j

Ersatz meetup - DeepLearning4j Demo

A Segmentation of Water Consumption with Apache Spark

SIGGRAPH 2012: GPU-Accelerated 2D and Web Rendering

GPUs in Big Data - StampedeCon 2014

GPU Ecosystem

GTC 2012: GPU-Accelerated Path Rendering

PG-Strom - GPU Accelerated Asyncr

Accelerating Machine Learning Applications on Spark Using GPUs

Computational Techniques for the Statistical Analysis of Big Data in R

Deep learning on spark

Enabling Graph Analytics at Scale: The Opportunity for GPU-Acceleration of D...

Heterogeneous System Architecture Overview

PG-Strom - GPGPU meets PostgreSQL, PGcon2015

PyData Amsterdam - Name Matching at Scale

Hadoop + GPU

Deep Learning on Hadoop

From Machine Learning to Learning Machines: Creating an End-to-End Cognitive ...

Containerizing GPU Applications with Docker for Scaling to the Cloud

How to Solve Real-Time Data Problems

Similar to DeepLearning4J and Spark: Successes and Challenges - François Garillot

Algorithmia is a startup with a mission to make state of the art machine learning discoverable by everyone&emdash;they offer the largest algorithm marketplace in the world, with over 2500 algorithms supporting tens of thousands of application developers. Algorithma is the first company to make deep learning, one of the most conceptually difficult areas of computing, accessible to any company via microservices. In this session, you learn how this startup has selected and optimized Amazon EC2 instances for various algorithms (including the latest generation of GPU optimized instances), to create a flexible and scalable platform. They also share their architecture and best practices for getting any computationally-intensive application started quickly.

AWS re:Invent 2016: Bringing Deep Learning to the Cloud with Amazon EC2 (CMP314)

Amazon Web Services

Demystifying Machine Learning and Artificial Intelligence

EPCC, University of Edinburgh

David Kale and Ruben Fizsel from Skymind talk about deep learning for the JVM and enterprise using deeplearning4j (DL4J). Deep learning (nouveau neural nets) have sparked a renaissance in empirical machine learning with breakthroughs in computer vision, speech recognition, and natural language processing. However, many popular deep learning frameworks are targeted to researchers and poorly suited to enterprise settings that use Java-centric big data ecosystems. DL4J bridges the gap, bringing high performance numerical linear algebra libraries and state-of-the-art deep learning functionality to the JVM.

DL4J at Workday Meetup

David Kale

The combination of Deep Learning with Apache Spark has the potential for tremendous impact in many sectors of the industry. This webinar, based on the experience gained in assisting customers with the Databricks Virtual Analytics Platform, will present some best practices for building deep learning pipelines with Spark. Rather than comparing deep learning systems or specific optimizations, this webinar will focus on issues that are common to deep learning frameworks when running on a Spark cluster, including: * optimizing cluster setup; * configuring the cluster; * ingesting data; and * monitoring long-running jobs. We will demonstrate the techniques we cover using Google’s popular TensorFlow library. More specifically, we will cover typical issues users encounter when integrating deep learning libraries with Spark clusters. Clusters can be configured to avoid task conflicts on GPUs and to allow using multiple GPUs per worker. Setting up pipelines for efficient data ingest improves job throughput, and monitoring facilitates both the work of configuration and the stability of deep learning jobs.

Deep Learning on Apache® Spark™ : Workflows and Best Practices

Jen Aman

Deep Learning on Apache® Spark™: Workflows and Best Practices

Databricks

Deep Learning on Apache® Spark™: Workflows and Best Practices

Jen Aman

Deep Learning and Recurrent Neural Networks in the Enterprise

Josh Patterson

More information, visit: http://www.godatadriven.com/accelerator.html Data scientists aren’t a nice-to-have anymore, they are a must-have. Businesses of all sizes are scooping up this new breed of engineering professional. But how do you find the right one for your business? The Data Science Accelerator Program is a one year program, delivered in Amsterdam by world-class industry practitioners. It provides your aspiring data scientists with intensive on- and off-site instruction, access to an extensive network of speakers and mentors and coaching. The Data Science Accelerator Program helps you assess and radically develop the skills of your data science staff or recruits. Our goal is to deliver you excellent data scientists that help you become a data driven enterprise. The right tools We teach your organisation the proven data science tools. The right hands We are trusted by many industry leading partners. The right experience We've done big data and data science at many clients, we know what the real world is like. The right experts We have a world class selection of lecturers that you will be working with. Vincent D. Warmerdam Jonathan Samoocha Ivo Everts Rogier van der Geer Ron van Weverwijk Giovanni Lanzani The right curriculum We meet twice a month. Once for a lecture, once for a hackathon. Lectures The RStudio stack. The art of simulation. The iPython stack. Linear modelling. Operations research. Nonlinear modelling. Clustering & ensemble methods. Natural language processing. Time series. Visualisation. Scaling to big data. Advanced topics. Hackathons Scrape and mine the internet. Solving multiarmed bandit problems. Webdev with flask and pandas as a backend. Build an automation script for linear models. Build a heuristic tsp solver. Code review your automation for nonlinear models. Build a method that outperforms random forests. Build a markov chain to generate song lyrics. Predict an optimal portfolio for the stock market. Create an interactive d3 app with backend. Start up a spark cluster with large s3 data. You pick! Interested? Ping us here. signal@godatadriven.com

Data Science Accelerator Program

GoDataDriven

Recent Advances in Machine Learning: Bringing a New Level of Intelligence to ...

Brocade

by Dave Nielsen, Technical Program Manager, Big Data Technologies, STO, Intel In this workshop we explain how to use Deep Learning to recognize objects in photos using the BigDL framework for Apache Spark. The workshop starts by explaining how Convolution Neural Network models like SSD, VGG and Inception identify edges and then higher level characteristics in multiple layers in order to identify the object. Then we will describe how to build an end-to-end pipeline for image classification including data prep, model building, training, scoring. The workshop includes a Zeppelin Notebook with python code you can modify to train the model and test for accuracy. The code provided in this workshop can be used to run examples on your laptop, on-premises or on Amazon Web Services managed by Qubole. Level 100

Image Recognition on AWS with Apache Spark and BigDL

Amazon Web Services

Deep Domain

Zachary S. Brown

Big Data Analytics (ML, DL, AI) hands-on

Dony Riyanto

Volunteers around the world increasingly act as human sensors to collect millions of data points. A team from the World Bank trained deep learning models, using Apache Spark and BigDL, to confirm that photos gathered through a crowdsourced data collection pilot matched the goods for which observations were submitted. In this talk, Maurice Nsabimana, a statistician at the World Bank, and Jiao Wang, a software engineer on the Big Data Technology team at Intel, demonstrate a collaborative project to design and train large-scale deep learning models using crowdsourced images from around the world. BigDL is a distributed deep learning library designed from the ground up to run natively on Apache Spark. It enables data engineers and scientists to write deep learning applications in Scala or Python as standard Spark programs-without having to explicitly manage distributed computations. Attendees of this session will learn how to get started with BigDL, which runs in any Apache Spark environment, whether on-premise or in the Cloud.

Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...

Maurice Nsabimana

Notes from 2016 bay area deep learning school

Niketan Pansare

Bringing Deep Learning into production

Paolo Platter

Deep learning with tensorflow

Charmi Chokshi

Deep learning for dummies dec 23 2017

Ashok Govindarajan

Taming Your Deep Learning Workflow by Determined AI

desmondchanatdet

Introducing TensorFlow: The game changer in building "intelligent" applications

Rokesh Jankie

Amazon Deep Learning

Amanda Mackay (she/her)

Similar to DeepLearning4J and Spark: Successes and Challenges - François Garillot (20)

AWS re:Invent 2016: Bringing Deep Learning to the Cloud with Amazon EC2 (CMP314)

Demystifying Machine Learning and Artificial Intelligence

DL4J at Workday Meetup

Deep Learning on Apache® Spark™ : Workflows and Best Practices

Deep Learning on Apache® Spark™: Workflows and Best Practices

Deep Learning and Recurrent Neural Networks in the Enterprise

Data Science Accelerator Program

Recent Advances in Machine Learning: Bringing a New Level of Intelligence to ...

Image Recognition on AWS with Apache Spark and BigDL

Deep Domain

Big Data Analytics (ML, DL, AI) hands-on

Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...

Notes from 2016 bay area deep learning school

Bringing Deep Learning into production

Deep learning with tensorflow

Deep learning for dummies dec 23 2017

Taming Your Deep Learning Workflow by Determined AI

Introducing TensorFlow: The game changer in building "intelligent" applications

Amazon Deep Learning

More from sparktc

At the sold-out Spark & Machine Learning Meetup in Brussels on October 27, 2016, Pierre Borckmans of Real Impact Analytics delivered a lightning talk called "Writing Spark applications, the easy way". As Pierre explained, even though Apache Spark™ offers intuitive and high-level APIs, writing production-ready Spark data pipelines involves non-trivial challenges for data scientists without expert background in software development and devops matters. In this short talk, he showed how his team tackled these issues at Real Impact Analytics, by developing an intuitive framework for writing dataflows, offering convenient data exploration and testing facilities, while hiding devops-related complexity.

Apache Spark™ Applications the Easy Way - Pierre Borckmans

At the sold-out Spark & Machine Learning Meetup in Brussels on October 27, 2016, Sven Hafeneger of IBM delivered a lightning talk called Hyperparameter Optimization - when scikit-learn meets PySpark. As Sven explained, Apache Spark™ is not only useful when you have big data problems. If you have a relatively small data set you might still have a big computational problem. One problem is the search for optimal parameters for ML algorithms. Normally, a data scientist has a laptop with 4 cores (8 threads), that means it will take some time to perform a grid search. However, using Spark opens the possibility that the grid search can be taken out on a cluster with a higher degree of parallelism.

Hyperparameter Optimization - Sven Hafeneger

Data Science Hub & the Data Science Community - Philippe Van Impe

At the sold-out Spark & Machine Learning Meetup in Brussels on October 27, 2016, Kris Peeters of Data Minded delivered a lightning talk called A data scientist, a BI expert and a big data engineer walk into a bar: how 3 different worlds come together with Spark. As Kris explained, because of its general-purpose nature, Apache Spark™ is being used by a wide variety of data professionals, each with their own backgrounds. The data warehouse / data lake of a large organization is a spot where those 3 worlds collide. We've experience the good, the bad and the ugly of those encounters first hand. In this lightning talk, Kris shared what each group can learn from each other, how they can collaborate, and which are the recipes for disaster.

Data Science and Beer - Kris peeters

At the recent sold-out Spark & Machine Learning Meetup in Brussels, Holden Karau of the Spark Technology Center delivered a lightning talk called A very brief introduction to extending Spark ML for custom models: Talk + Demo. Holden took a look at Apache SparkML™ pipelines. Inspired by sci-kit learn, they have the potential to make machine learning tasks much easier. This talk looked at how to extend Spark ML with custom model types when the built-in options don't meet your needs.

Holden Karau - Spark ML for Custom Models

At the sold-out Spark & Machine Learning Meetup in Brussels on October 27, 2016, Nick Pentreath of the Spark Technology Center teamed up with Jean-François Puget of IBM Analytics to deliver a talk called Creating an end-to-endRecommender System with Apache Spark and Elasticsearch. Jean-François and Nick started with a look at the workflow for recommender systems and machine learning, then moved on to data modeling and using Spark ML for collaborative filtering. They closed with a discussion of deploying and scoring the recommender models, including a demo.

Creating an end-to-end Recommender System with Apache Spark and Elasticsearch...

At the recent sold-out Spark & Machine Learning Meetup in Brussels, François Garillot of Skymind delivered a lightning talk called DeepLearning4J and Spark: Successes and Challenges. Specifically, François offered a tour of the DeepLearning4J architecture intermingled with applications. He went over the main blocks of this deep learning solution for the JVM that includes GPU acceleration, a custom n-dimensional array library, a parallelized data-loading swiss army tool, deep learning and reinforcement learning libraries — all with an easy-access interface. Along the way, he pointed out the strategic points of parallelization of computation across machines and gave insight on where Spark helps — and where it doesn't.

DeepLearning4J and Spark: Successes and Challenges - François Garillot

Building Custom Machine Learning Algorithms with Apache SystemML

Weather is both massive and intensely local. Building a weather service involves capturing lots of data for many, many places over long timeframes, applying intensive analysis to those datasets to predict what will happen next, and then delivering those insights to the millions mobile users and other endpoints who request it, all within a fraction of a second. Here we share the slides from Robbie Strickland's Spark Summit talk. In the slides, Robbie offers the full end-to-end view of how The Weather Company uses Spark to manage this large scale challenge.

The Internet of Everywhere — How The Weather Company Scales

GPU Support in Spark and GPU/CPU Mixed Resource Scheduling at Production Scale

STC Design - Engage

How Spark Enables the Internet of Things: Efficient Integration of Multiple ...

Spark Summit EU: IBM Keynote