Some weeks ago, our ML6 agent Karel Dumon gave a talk at a Nexxworks Bootcamp. During this week-long event, several speakers are invited to take the floor to inspire a heterogenous group of (senior) business people from a wide range of industries. On the third day, Artificial Intelligence was planned. A broad intro to AI and ML was given by prof. dr. Eric Mannens, after which Karel provided the audience with some hands-on insights through use cases.
Some weeks ago, our ML6 agent Karel Dumon gave a talk at a Nexxworks Bootcamp. During this week-long event, several speakers are invited to take the floor to inspire a heterogenous group of (senior) business people from a wide range of industries. On the third day, Artificial Intelligence was planned. A broad intro to AI and ML was given by prof. dr. Eric Mannens, after which Karel provided the audience with some hands-on insights through use cases.
In this paper we propose Regularised Cross-Modal Hashing
(RCMH) a new cross-modal hashing model that projects
annotation and visual feature descriptors into a common
Hamming space. RCMH optimises the hashcode similarity
of related data-points in the annotation modality using an
iterative three-step hashing algorithm: in the first step each
training image is assigned a K-bit hashcode based on hyperplanes learnt at the previous iteration; in the second step the binary bits are smoothed by a formulation of graph regularisation so that similar data-points have similar bits; in the third step a set of binary classifiers are trained to predict the regularised bits with maximum margin. Visual descriptors are projected into the annotation Hamming space by a set of binary classifiers learnt using the bits of the corresponding annotations as labels. RCMH is shown to consistently improve retrieval effectiveness over state-of-the-art baselines.
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FO...Rob Emanuele
Slides from the 2017 FOSS4G Workshop "Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS"
See the repository at https://github.com/lossyrob/foss4g-2017-geopyspark-workshop
Pillow - The python Image Processing Library provides histogram() method in the Image class to get a histogram of colors/bands present in the Image.
histogram() method provides a list of counts of pixels for each color.eg., Red, Blue, Green for an Image of mode "RGB"
Talk given on September 21 to the Bay Area R User Group. The talk walks a stochastic project SVD algrorithm through the steps from initial implementation in R to a proposed implementation using map-reduce that integrates cleanly with R via NFS export of the distributed file system. Not surprisingly, this algorithm is essentially the same as the one used by Mahout.
LocationTech is an Eclipse Foundation industry working group for location aware technologies. This presentation introduces LocationTech, looks at what it means for our industry and the participating projects.
Libraries: JTS Topology Suite is the rocket science of GIS providing an implementation of Geometry. Mobile Map Tools provides a C++ foundation that is translated into Java and Javascript for maps on iOS, Andriod and WebGL. GeoMesa is a distributed key/value store based on Accumulo. Spatial4j integrates with JTS to provide Geometry on curved surface.
Process: GeoTrellis real-time distributed processing used scala, akka and spark. GeoJinni mixes spatial data/indexing with Hadoop.
Applications: GEOFF offers OpenLayers 3 as a SWT component. GeoGit distributed revision control for feature data. GeoScipt brings spatial data to Groovy, JavaScript, Python and Scala. uDig offers an eclipse based desktop GIS solution.
Attend this presentation if want to know what LocationTech is about, are interested in these projects or curious about what projects will be next.
The ArangoML Group had a detailed discussion on the topic "GraphSage Vs PinSage" where they shared their thoughts on the difference between the working principles of two popular Graph ML algorithms. The following slidedeck is an accumulation of their thoughts about the comparison between the two algorithms.
Tensor Processing Units (TPUs) are Google's custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs are designed from the ground up with the benefit of Google's deep experience and leadership in machine learning.
In this paper we propose Regularised Cross-Modal Hashing
(RCMH) a new cross-modal hashing model that projects
annotation and visual feature descriptors into a common
Hamming space. RCMH optimises the hashcode similarity
of related data-points in the annotation modality using an
iterative three-step hashing algorithm: in the first step each
training image is assigned a K-bit hashcode based on hyperplanes learnt at the previous iteration; in the second step the binary bits are smoothed by a formulation of graph regularisation so that similar data-points have similar bits; in the third step a set of binary classifiers are trained to predict the regularised bits with maximum margin. Visual descriptors are projected into the annotation Hamming space by a set of binary classifiers learnt using the bits of the corresponding annotations as labels. RCMH is shown to consistently improve retrieval effectiveness over state-of-the-art baselines.
Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS - FO...Rob Emanuele
Slides from the 2017 FOSS4G Workshop "Analyzing Larger RasterData in a Jupyter Notebook with GeoPySpark on AWS"
See the repository at https://github.com/lossyrob/foss4g-2017-geopyspark-workshop
Pillow - The python Image Processing Library provides histogram() method in the Image class to get a histogram of colors/bands present in the Image.
histogram() method provides a list of counts of pixels for each color.eg., Red, Blue, Green for an Image of mode "RGB"
Talk given on September 21 to the Bay Area R User Group. The talk walks a stochastic project SVD algrorithm through the steps from initial implementation in R to a proposed implementation using map-reduce that integrates cleanly with R via NFS export of the distributed file system. Not surprisingly, this algorithm is essentially the same as the one used by Mahout.
LocationTech is an Eclipse Foundation industry working group for location aware technologies. This presentation introduces LocationTech, looks at what it means for our industry and the participating projects.
Libraries: JTS Topology Suite is the rocket science of GIS providing an implementation of Geometry. Mobile Map Tools provides a C++ foundation that is translated into Java and Javascript for maps on iOS, Andriod and WebGL. GeoMesa is a distributed key/value store based on Accumulo. Spatial4j integrates with JTS to provide Geometry on curved surface.
Process: GeoTrellis real-time distributed processing used scala, akka and spark. GeoJinni mixes spatial data/indexing with Hadoop.
Applications: GEOFF offers OpenLayers 3 as a SWT component. GeoGit distributed revision control for feature data. GeoScipt brings spatial data to Groovy, JavaScript, Python and Scala. uDig offers an eclipse based desktop GIS solution.
Attend this presentation if want to know what LocationTech is about, are interested in these projects or curious about what projects will be next.
The ArangoML Group had a detailed discussion on the topic "GraphSage Vs PinSage" where they shared their thoughts on the difference between the working principles of two popular Graph ML algorithms. The following slidedeck is an accumulation of their thoughts about the comparison between the two algorithms.
Tensor Processing Units (TPUs) are Google's custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning workloads. TPUs are designed from the ground up with the benefit of Google's deep experience and leadership in machine learning.
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017StampedeCon
This talk will go over how to build an end-to-end data processing system in Python, from data ingest, to data analytics, to machine learning, to user presentation. Developments in old and new tools have made this particularly possible today. The talk in particular will talk about Airflow for process workflows, PySpark for data processing, Python data science libraries for machine learning and advanced analytics, and building agile microservices in Python.
System architects, software engineers, data scientists, and business leaders can all benefit from attending the talk. They should learn how to build more agile data processing systems and take away some ideas on how their data systems could be simpler and more powerful.
Machine learning at scale by Amy Unruh from GoogleBill Liu
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
At the technology meeting of the Association of Independent Research Centers (http://airi.org): An overview of recent Scientific Computing activities at Fred Hutch, Seattle
Streaming Random Forest Learning in Spark and StreamDM with Heitor Murilogome...Databricks
We present how to build random forest models from streaming data. This is achieved by training, predicting and adapting the model in real-time with evolving data streams. The implementation is on the open source library StreamDM, built on top of Apache Spark.
Accelerating Random Forests in Scikit-LearnGilles Louppe
Random Forests are without contest one of the most robust, accurate and versatile tools for solving machine learning tasks. Implementing this algorithm properly and efficiently remains however a challenging task involving issues that are easily overlooked if not considered with care. In this talk, we present the Random Forests implementation developed within the Scikit-Learn machine learning library. In particular, we describe the iterative team efforts that led us to gradually improve our codebase and eventually make Scikit-Learn's Random Forests one of the most efficient implementations in the scientific ecosystem, across all libraries and programming languages. Algorithmic and technical optimizations that have made this possible include:
- An efficient formulation of the decision tree algorithm, tailored for Random Forests;
- Cythonization of the tree induction algorithm;
- CPU cache optimizations, through low-level organization of data into contiguous memory blocks;
- Efficient multi-threading through GIL-free routines;
- A dedicated sorting procedure, taking into account the properties of data;
- Shared pre-computations whenever critical.
Overall, we believe that lessons learned from this case study extend to a broad range of scientific applications and may be of interest to anybody doing data analysis in Python.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
The Future of Computing is Distributed
Professor Ion Stoica, UC Berkeley RISELab
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Accelerated Machine Learning with RAPIDS and MLflow, Nvidia/RAPIDSDatabricks
Accelerated Machine Learning with RAPIDS and MLflow, Nvidia/RAPIDS
Abstract: We will introduce RAPIDS, a suite of open source libraries for GPU-accelerated data science, and illustrate how it operates seamlessly with MLflow to enable reproducible training, model storage, and deployment. We will walk through a baseline example that incorporates MLflow locally, with a simple SQLite backend, and briefly introduce how the same workflow can be deployed in the context of GPU enabled Kubernetes clusters.
Automated Intrusion Response - CDIS Spring Conference 2024Kim Hammar
Presentation at CDIS Spring Conference 2024.
The ubiquity and evolving nature of cyber attacks is of growing concern to industry and society. In response, the automation of security processes and functions is the focus of many current research efforts. In this talk we will present a framework for automated network intrusion response, in which we model the interaction between an attacker and a defender as a partially observed Markov game. Within this framework, reinforcement learning enables the controlled evolution of attack and defense strategies towards a Nash equilibrium through the process of self-play. To realize and experiment with the self-play process on a practical IT infrastructure, we have developed a software platform for creating digital twins, which provide two key functions for our framework: (i) a safe and realistic test environment; and (ii) a tool for evaluation that enables closed-loop learning of security strategies.
Intrusion Tolerance for Networked Systems through Two-level Feedback ControlKim Hammar
We formulate intrusion tolerance for a system with service replicas as a two-level optimal control problem. On the local control level, node controllers perform intrusion recoveries and on the global control level, a system controller manages the replication factor.
Learning Near-Optimal Intrusion Responses for IT Infrastructures via Decompos...Kim Hammar
We study automated intrusion response and formulate the interaction between an attacker and a defender on an IT infrastructure as a stochastic game where attack and defense strategies evolve through reinforcement learning and self-play. Direct application of reinforcement learning to any non-trivial instantiation of this game is impractical due to the exponential growth of the state and action spaces with the number of components in the infrastructure. We propose a decompositional approach to deal with this challenge and prove that under assumptions generally met in practice the game decomposes into a) additive subgames on the workflow-level that can be optimized independently; and b) subgames on the component-level that satisfy the optimal substructure property. We further show that the optimal defender strategies on the component-level exhibit threshold structures. To solve the decomposed game we develop Decompositional Fictitious Self-Play (\dfsp), an efficient fictitious self-play algorithm that learns Nash equilibria through stochastic approximation. We show that \dfsp outperforms a state-of-the-art algorithm for our use case. To evaluate the learned strategies, we deploy them in a a virtual IT infrastructure in which we run real network intrusions and real response actions. From our experimental investigation we conclude that our approach can produce effective defender strategies for a practical IT infrastructure.
Learning Near-Optimal Intrusion Responses for IT Infrastructures via Decompos...Kim Hammar
We study automated intrusion response and formulate the interaction between an attacker and a defender on an IT infrastructure as a stochastic game where attack and defense strategies evolve through reinforcement learning and self-play. Direct application of reinforcement learning to any non-trivial instantiation of this game is impractical due to the exponential growth of the state and action spaces with the number of components in the infrastructure. We propose a decompositional approach to deal with this challenge and prove that under assumptions generally met in practice, the game decomposes into a) additive subgames on the workflow-level that can be optimized independently; and b) subgames on the component-level that satisfy the optimal substructure property. We further show that the optimal defender strategies on the component-level exhibit threshold structures. To solve the decomposed game we develop Decompositional Fictitious Self-Play (\dfsp), an efficient fictitious self-play algorithm that learns Nash equilibria through stochastic approximation. We show that \dfsp outperforms a state-of-the-art algorithm for our use case. To evaluate the learned strategies, we deploy them in a a virtual IT infrastructure in which we run real network intrusions and real response actions. From our experimental investigation we conclude that our approach can produce effective defender strategies for a practical IT infrastructure.
Learning Optimal Intrusion Responses via DecompositionKim Hammar
We study automated intrusion response and formulate the interaction between an attacker and a defender on an IT infrastructure as a stochastic game where attack and defense strategies evolve through reinforcement learning and self-play. Direct application of reinforcement learning to any non-trivial instantiation of this game is impractical due to the exponential growth of the state and action spaces with the number of components in the infrastructure. We propose a decompositional approach to deal with this challenge and prove that under assumptions generally met in practice, the game decomposes into a) additive subgames on the workflow-level that can be optimized independently; and b) subgames on the component-level that satisfy the optimal substructure property. We further show that the optimal defender strategies on the component-level exhibit threshold structures. To solve the decomposed game we develop Decompositional Fictitious Self-Play (\dfsp), an efficient fictitious self-play algorithm that learns Nash equilibria through stochastic approximation. We show that \dfsp outperforms a state-of-the-art algorithm for our use case. To evaluate the learned strategies, we deploy them in a a virtual IT infrastructure in which we run real network intrusions and real response actions. From our experimental investigation we conclude that our approach can produce effective defender strategies for a practical IT infrastructure.
—We present a novel emulation system for creating
high-fidelity digital twins of IT infrastructures. The digital twins
replicate key functionality of the corresponding infrastructures
and allow to play out security scenarios in a safe environment.
We show that this capability can be used to automate the process
of finding effective security policies for a target infrastructure. In
our approach, a digital twin of the target infrastructure is used
to run security scenarios and collect data. The collected data is
then used to instantiate simulations of Markov decision processes
and learn effective policies through reinforcement learning, whose
performances are validated in the digital twin. This closed-loop
learning process executes iteratively and provides continuously
evolving and improving security policies. We apply our approach
to an intrusion response scenario. Our results show that the
digital twin provides the necessary evaluative feedback to learn
near-optimal intrusion response policies.
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...sameer shah
"Join us for STATATHON, a dynamic 2-day event dedicated to exploring statistical knowledge and its real-world applications. From theory to practice, participants engage in intensive learning sessions, workshops, and challenges, fostering a deeper understanding of statistical methodologies and their significance in various fields."
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeWalaa Eldin Moustafa
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
Adjusting OpenMP PageRank : SHORT REPORT / NOTESSubhajit Sahu
For massive graphs that fit in RAM, but not in GPU memory, it is possible to take
advantage of a shared memory system with multiple CPUs, each with multiple cores, to
accelerate pagerank computation. If the NUMA architecture of the system is properly taken
into account with good vertex partitioning, the speedup can be significant. To take steps in
this direction, experiments are conducted to implement pagerank in OpenMP using two
different approaches, uniform and hybrid. The uniform approach runs all primitives required
for pagerank in OpenMP mode (with multiple threads). On the other hand, the hybrid
approach runs certain primitives in sequential mode (i.e., sumAt, multiply).
Kim Hammar - Distributed Deep Learning - RISE Learning Machines Meetup
1. Distributed Deep Learning (DDL) with HopsML
RISE Machine Learning Study Group
Kim Hammar
kim@logicalclocks.com
November 29, 2018
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 1 / 20
2. Outline
1 Distributed Deep Learning (DDL) Theory
2 HopsML: Distributed Deep Learning in Practice
3 Use-Case of DDL: Anti-Money-Laundering
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 2 / 20
3. b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy
Distributed Computing Deep Learning
Why Combine the two?
More productive Data Science1
Unreasonable effectiveness of data2
To achieve state-of-the-art results3
1
Alex Sergeev and title = Meet Horovod: Uber’s Open Source Distributed Deep Learning Framework for
TensorFlow howpublished = https://eng.uber.com/horovod/ note = Accessed: 2018-11-24 Mike
Del Balso year=2017.
2
Chen Sun et al. “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era”. In: CoRR
abs/1707.02968 (2017). arXiv: 1707.02968. URL: http://arxiv.org/abs/1707.02968.
3
Jeffrey Dean et al. “Large Scale Distributed Deep Networks”. In: Advances in Neural Information Processing
Systems 25. Ed. by F. Pereira et al. Curran Associates, Inc., 2012, pp. 1223–1231. URL:
http://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks.pdf.
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 3 / 20
4. What is Distributed Deep Learning?
HPC
Deep Learning
Systems
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 4 / 20
6. Data Parallelism
Data Parallel Workers W
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy . . .
b0
x0,1
x0,2
x0,3
b1
x1,1
x1,2
x1,3
ˆy
Data Partitions P p1 p2 . . . pn
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 6 / 20
7. When to use Model Parallel and Data Parallel?
How big is your model parameters θ vs
GPU memory? If size(θ) > size(gpu) you have to use model parallelism
If your model fits on a single GPU =⇒ in 99.999% you want to use
data parallelism to reduce training time
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 7 / 20
8. Parameter Server Architecture
Parameter Server ps d
Data Parallel Workers W e1 e2 . . . en
Data Partitions P p1 p2 . . . pn
Broadcast parameters
Upload gradients
Local training
Read data
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 8 / 20
10. When to use Parameter-Server and when to use
Ring-All-Reduce?
Ring-all-reduce scales better =⇒ generally prefer ring-all-reduce4
4
Alex Sergeev and title = Meet Horovod: Uber’s Open Source Distributed Deep Learning Framework for
TensorFlow howpublished = https://eng.uber.com/horovod/ note = Accessed: 2018-11-24 Mike
Del Balso year=2017.
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 10 / 20
11. How to get started?
ICE (RISE SICS NORTH) provides the hardware that you need
GPU Machines for training
CPU Machines for data prep
Disks for storing large datasets
HopsML provides the ML infrastructure that you need
Fast Distributed File System
Spark-jobs and notebooks for data prep
Framework for reproducible and versioned parallel experiments
Framework for distributed training
Framework for monitoring training
Support for auto-scaling model serving
Feature store (Soon!)
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 11 / 20
12. Hopsworks: UI-driven front-end to the ML infrastructure
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 12 / 20
13. Python-First API-powered workflow
Write your regular tensorflow/python/pytorch/keras code and put it in a
function, for example called collective_all_reduce_mnist, then you
can create a reproducible experiment using many GPUs and
collective-all-reduce as follows:
from hops import experiment
from hops import hdfs
notebook = hdfs.project_path() +
Jupyter/Distributed_Training/collective_allreduce_strategy/mnist.ipynb
experiment.collective_all_reduce(collective_all_reduce_mnist ,
name=’mnist estimator’,
description=’A minimal mnist example with two hidden layers’,
versioned_resources=[notebook], local_logdir=True)
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 13 / 20
14. Single-GPU Training on Hops
HopsFS
Spark Driver
Read Data
Write Results
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 14 / 20
15. Parallel Experiments on Hops
HopsFS
Spark Driver
Distributed Hyperparameter Search
With GPUs
E1 E2 E3 E4 E5 E6
read data
write results
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 15 / 20
16. Multi-GPU Training on Hops
HopsFS
Spark Driver
Read Data
Write Results
send/receiveGPUs aranged in
a logical ring for
ring-all-reduce training
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 16 / 20
17. Distributed GPU Training on Hops
HopsFS
Spark Driver
E1
E2
E3
E4
read data
write results
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 17 / 20
18. Model Serving on Hops
Kim Hammar (Logical Clocks) DDL on Hops November 29, 2018 18 / 20
19. Register at hops.site, email: kim@logicalclocks.com if your
registration is not approved
Try out the deep learning tour on hopsworks
Example code:
https://github.com/logicalclocks/hops-examples
Look at the docs: https://www.hops.io/
If you get stuck, write on gitter:
https://gitter.im/hopshadoop/hopsworks
DEMO