This document discusses Joblib, a Python package for parallel computing and caching. It provides an overview of Joblib's capabilities including: making parallel computing easy using an embarrassingly parallel helper; efficient caching of computations on disk to avoid recomputation; persistence of arbitrary objects; and support for multiple parallel and caching backends including threading, multiprocessing, distributed, and Hadoop. The document also discusses using Joblib for cloud computing by configuring parallel backends to run on multiple machines and extending caching to cloud storage backends like S3 and HDFS. Future work includes in-memory caching, overriding parallel backends, replacing multiprocessing, and expanding cloud provider support.
This document discusses function-as-a-service (FaaS) computing from a Python perspective. It begins with an overview of FaaS and some existing tools. Then it discusses Limbadi, a tool that can decompose Python functions for FaaS deployment. It also discusses Snafu, a Pythonic serverless computing platform that can execute functions across FaaS ecosystems and provides interactive and daemon modes for development and hosting functions. Hands-on examples are provided for Limbadi and Snafu.
This document discusses Joblib, a Python package for parallel computing and caching. It provides an overview of Joblib's capabilities including: making parallel computing easy using an embarrassingly parallel helper; efficient caching of computations on disk to avoid recomputation; persistence of arbitrary objects; and support for multiple parallel and caching backends including threading, multiprocessing, distributed, and Hadoop. The document also discusses using Joblib for cloud computing by configuring parallel backends to run on multiple machines and extending caching to cloud storage backends like S3 and HDFS. Future work includes in-memory caching, overriding parallel backends, replacing multiprocessing, and expanding cloud provider support.
All you need to know about the JavaScript event loopSaša Tatar
The document discusses the JavaScript event loop and call stack. It explains that JavaScript is single-threaded with an event loop that processes tasks in the order they are received. There is a call stack that processes code synchronously, and an event queue that holds asynchronous callbacks from events like timers, promises, etc. The event loop continually checks the call stack and event queue, running tasks from the queue after the stack is empty. This allows asynchronous code to run without blocking synchronous code.
Vector and ListBuffer have similar performance for random reads. Benchmarking showed no significant difference in throughput, average time, or sample times between reading randomly from a Vector versus a ListBuffer. Vectors are generally faster than Lists for random access due to Vectors being implemented as arrays under the hood.
The document discusses Karol Kubicki's experience learning Swift after being asked to give a talk on it. It provides examples of code snippets comparing implementations of common patterns like singletons, error handling, laziness, structs, and curried functions between Swift and Objective-C. Key differences highlighted include Swift being more thread-safe, having cleaner syntax, and providing language support for features like laziness while Objective-C requires more boilerplate code. Real-world use cases are presented to illustrate concepts like currying functions.
Internship final report@Treasure Data Inc.Ryuichi ITO
Ryuchi Ito completed an internship at Treasure Data Inc. where he worked on the open source machine learning library Hivemall. He conducted benchmarks of Hivemall's performance on logistic regression and random forest algorithms. He also added new features to Hivemall including a system testing framework, feature binning, feature selection, and Spark integrations. The benchmarks showed that Hivemall was relatively slow for logistic regression compared to other tools but had good scalability. For random forests, Hivemall performed well on small to medium datasets but struggled on very large datasets.
This document discusses function-as-a-service (FaaS) computing from a Python perspective. It begins with an overview of FaaS and some existing tools. Then it discusses Limbadi, a tool that can decompose Python functions for FaaS deployment. It also discusses Snafu, a Pythonic serverless computing platform that can execute functions across FaaS ecosystems and provides interactive and daemon modes for development and hosting functions. Hands-on examples are provided for Limbadi and Snafu.
This document discusses Joblib, a Python package for parallel computing and caching. It provides an overview of Joblib's capabilities including: making parallel computing easy using an embarrassingly parallel helper; efficient caching of computations on disk to avoid recomputation; persistence of arbitrary objects; and support for multiple parallel and caching backends including threading, multiprocessing, distributed, and Hadoop. The document also discusses using Joblib for cloud computing by configuring parallel backends to run on multiple machines and extending caching to cloud storage backends like S3 and HDFS. Future work includes in-memory caching, overriding parallel backends, replacing multiprocessing, and expanding cloud provider support.
All you need to know about the JavaScript event loopSaša Tatar
The document discusses the JavaScript event loop and call stack. It explains that JavaScript is single-threaded with an event loop that processes tasks in the order they are received. There is a call stack that processes code synchronously, and an event queue that holds asynchronous callbacks from events like timers, promises, etc. The event loop continually checks the call stack and event queue, running tasks from the queue after the stack is empty. This allows asynchronous code to run without blocking synchronous code.
Vector and ListBuffer have similar performance for random reads. Benchmarking showed no significant difference in throughput, average time, or sample times between reading randomly from a Vector versus a ListBuffer. Vectors are generally faster than Lists for random access due to Vectors being implemented as arrays under the hood.
The document discusses Karol Kubicki's experience learning Swift after being asked to give a talk on it. It provides examples of code snippets comparing implementations of common patterns like singletons, error handling, laziness, structs, and curried functions between Swift and Objective-C. Key differences highlighted include Swift being more thread-safe, having cleaner syntax, and providing language support for features like laziness while Objective-C requires more boilerplate code. Real-world use cases are presented to illustrate concepts like currying functions.
Internship final report@Treasure Data Inc.Ryuichi ITO
Ryuchi Ito completed an internship at Treasure Data Inc. where he worked on the open source machine learning library Hivemall. He conducted benchmarks of Hivemall's performance on logistic regression and random forest algorithms. He also added new features to Hivemall including a system testing framework, feature binning, feature selection, and Spark integrations. The benchmarks showed that Hivemall was relatively slow for logistic regression compared to other tools but had good scalability. For random forests, Hivemall performed well on small to medium datasets but struggled on very large datasets.
This document summarizes Yurie Yamane's presentation on using mruby to make robots. It discusses using mruby and TOPPERS on the LEGO EV3 to create an inverted pendulum self-balancing robot. It then covers creating a DIY self-balancing robot using a Raspberry Pi, gyro sensor, DC motors, and a motor driver. Code examples are provided for reading sensor values, controlling motors with PWM, and implementing a balancing algorithm using a balancer class. Source code links are included for further reference.
This document discusses JavaScript engines and event loops. It explains key concepts like the call stack, event queue, microtasks vs macrotasks, and how JavaScript creates the illusion of being asynchronous through the event loop. The event loop constantly checks the call stack and queue, executing functions to give the appearance of multithreading on a single thread. Web APIs also provide asynchronous capabilities that interact with the event loop.
node.js and native code extensions by examplePhilipp Fehre
Over the last years node.js has evolved to be a great language to build web applications. The reason for this is not only that it is based on JavaScript which already is established around "the web" but also that it provides excellent facilities for extensions, not only via JavaScript but also integration of native C libraries. Couchbase makes a lot of use of this fact making the Couchbase node.js SDK (Couchnode) a wrapper around the C library providing a node.js like API, but leveraging the power of a native C library underneat. So how is this done? How does such a package look like? Let me show you how integration of C in node.js works and how to "read" a package like Couchnode.
Presentation to the MIT IAP HTML5 Game Development Class on Debugging and Optimizing Javascript, Local storage, Offline Storage and Server side Javascript with Node.js
Here are the key points about GRASP patterns:
1. GRASP stands for General Responsibility Assignment Software Patterns. It is a set of 9 patterns used to assign responsibilities to classes in object-oriented design.
2. Properly assigning responsibilities and distributing them among classes is a core part of object-oriented design. Developers and designers should be familiar with these patterns.
3. The 9 GRASP patterns are: Information Expert, Creator, Controller, Low Coupling, High Cohesion, Indirection, Polymorphism, Pure Fabrication, and Protected Variations. Each pattern provides guidance on distributing responsibilities in a certain context.
4. When designing a system, analyzing which classes should have which responsibilities based
The document is a presentation by Clément Sauvage about Realm, a new mobile database. It introduces Realm as being amazingly simple to use, requiring only a few lines of code to instantiate and query. Realm uses object-oriented classes to define schemas and supports common operations like queries, sorting, relations between objects, and migrations between schema versions. The presentation provides examples of adding, querying, and relating objects to demonstrate Realm's ease of use compared to other mobile databases.
The document discusses concepts related to event loops and concurrency in programming. It includes code snippets in Java showing the use of NIO selectors and channels for non-blocking I/O. Diagrams are shown illustrating reactor patterns and Vert.x modules deployed across multiple CPUs. The summary provides an overview of the main topics and techniques discussed in the technical document.
The history of asynchronous programming in .NET began with threads and the thread pool to handle concurrency. Later, patterns like the Asynchronous Programming Model (APM) and Event-based Asynchronous Pattern (EAP) simplified asynchronous code but it remained complex. The Task Parallel Library (TPL) and async/await further abstracted asynchronous operations so code more closely resembled synchronous code. Now, asynchronous code no longer requires Wait() or GetAwaiter().GetResult() and avoids potential deadlocks.
The document discusses extending Node.js using C++. It covers wrapping C++ classes for use in JavaScript, including setters, getters, exceptions, and instantiating objects. Examples are provided for wrapping Book and Person classes to be accessible from JavaScript. Building the extension using waf or gyp build systems is also covered. The presenter's background and the agenda are outlined initially.
- Rcpp is a package that facilitates interoperability between R and C++ by providing data structures and functions that make it easy to write C++ code that integrates with R. It has been released 54 times since 2008 with over 170 CRAN packages depending on it.
- Rcpp allows users to source C++ code from R using sourceCpp() and export C++ functions to R using attributes like // [[Rcpp::export]]. This improves performance over pure R code by leveraging fast C++ implementations.
- dplyr is a popular R package for data manipulation that achieves great performance through its use of Rcpp. Functions like arrange(), filter(), and summarise() are much faster when
This document discusses background processes in Android, including threads, services, and notifications. It explains that threads are used to perform long-running tasks without blocking the main UI thread. Services are intended for long-running background tasks rather than threads, and can be started and stopped from an activity. Notifications allow services to display status information in the status bar and notify the user of events even when the app is not visible.
Let'Swift 2019 컨퍼런스에서 RxSwift to Combine 이라고 발표한 자료입니다. 자료 내에 언급되는 코드 링크는 다음과 같습니다.
[BringMyOwnBeer]
• RxSwift/RxCocoa: https://github.com/fimuxd/BringMyOwnBeer-
• Combine/SwiftUI: https://github.com/fimuxd/BringMyOwnBeer-Combine
Let'Swift 2019 컨퍼런스에서 RxSwift to Combine 이라고 발표한 자료입니다. 자료 내에 언급되는 코드 링크는 다음과 같습니다.
[BringMyOwnBeer]
• RxSwift/RxCocoa: https://github.com/fimuxd/BringMyOwnBeer-
• Combine/SwiftUI: https://github.com/fimuxd/BringMyOwnBeer-Combine
- Python uses reference counting and a cyclic garbage collector to manage memory and free objects that are no longer referenced. It allocates memory for objects in blocks and assigns them to size classes based on their size.
- Objects in Python have a type and attributes. They are created via type constructors and can have specific attributes like __dict__, __slots__, and descriptors.
- At the Python virtual machine level, code is represented as code objects that contain details like the bytecode, constants, and variable names. Code objects are compiled from Python source files.
The document provides an overview of key differences between Python and Scala. Some key points summarized:
1. Python is a dynamically typed, interpreted language while Scala is statically typed and compiles to bytecode. Scala supports both object-oriented and functional programming paradigms.
2. Scala has features like case classes, traits, and pattern matching that Python lacks. Scala also has features like type parameters, implicit conversions, and tail call optimization that Python does not support natively.
3. Common data structures like lists and maps are implemented differently between the languages - Scala uses immutable Lists while Python uses mutable lists. Scala also has features like lazy vals.
4. Control
The document discusses asynchronous programming with futures and promises in Scala. It covers key topics like:
1) Futures provide a way to reason about parallel operations efficiently in a non-blocking way. Futures represent asynchronous computations with results that can be accessed later.
2) Promises allow completing a future with a value or exception. Callbacks allow accessing future results asynchronously once completed.
3) Functional composition with futures uses methods like map, flatMap, recover to transform and compose futures. For-comprehensions provide a cleaner way to compose futures.
Streams allow connecting programs like connecting garden hoses. They provide a way to pipe input through multiple steps to an output. There are different types of streams like readable, writable, duplex, and transform streams. Common uses of streams in Node.js include http, fs, child processes, tcp, and zlib. Streams are event emitters that can be used to process data incrementally instead of all at once.
Griffon is a desktop application framework inspired by Grails that leverages Swing and Groovy. It follows conventions like convention over configuration and pervasive MVC. Griffon emphasizes writing modular code, automated testing, and deploying applications easily to applets, webstart, or desktop without code changes. The presentation discusses porting legacy apps to Griffon by embracing its tenets like data binding, simpler layout managers like MigLayout, and using SwingWorker for long background tasks.
Joblib is a Python module that provides tools for parallel computing such as caching results on disk, efficient persistence of large numpy arrays, and support for multiple parallel backends including threading, multiprocessing, distributed, and ipyparallel. Recent updates have improved Joblib's persistence functionality by storing multiple arrays in a single file without memory copies, supporting additional compression formats like gzip and bz2, and allowing custom parallel backends. Future work may include persisting directly to file-like objects and remote storage, as well as using Joblib in cloud computing environments.
Joblib Toward efficient computing : from laptop to cloudPyDataParis
Joblib is a Python module that provides tools for parallel computing such as caching results on disk, efficient persistence of large numpy arrays, and support for multiple parallel backends including threading, multiprocessing, distributed, and ipyparallel. Recent updates have improved Joblib's persistence functionality by allowing multiple arrays to be stored in a single file without memory copies, supporting additional compression formats like gzip and bz2, and enabling custom parallel backends. Future work may include persisting directly to file-like objects and remote storage as well as using Joblib in cloud computing environments.
This document summarizes Yurie Yamane's presentation on using mruby to make robots. It discusses using mruby and TOPPERS on the LEGO EV3 to create an inverted pendulum self-balancing robot. It then covers creating a DIY self-balancing robot using a Raspberry Pi, gyro sensor, DC motors, and a motor driver. Code examples are provided for reading sensor values, controlling motors with PWM, and implementing a balancing algorithm using a balancer class. Source code links are included for further reference.
This document discusses JavaScript engines and event loops. It explains key concepts like the call stack, event queue, microtasks vs macrotasks, and how JavaScript creates the illusion of being asynchronous through the event loop. The event loop constantly checks the call stack and queue, executing functions to give the appearance of multithreading on a single thread. Web APIs also provide asynchronous capabilities that interact with the event loop.
node.js and native code extensions by examplePhilipp Fehre
Over the last years node.js has evolved to be a great language to build web applications. The reason for this is not only that it is based on JavaScript which already is established around "the web" but also that it provides excellent facilities for extensions, not only via JavaScript but also integration of native C libraries. Couchbase makes a lot of use of this fact making the Couchbase node.js SDK (Couchnode) a wrapper around the C library providing a node.js like API, but leveraging the power of a native C library underneat. So how is this done? How does such a package look like? Let me show you how integration of C in node.js works and how to "read" a package like Couchnode.
Presentation to the MIT IAP HTML5 Game Development Class on Debugging and Optimizing Javascript, Local storage, Offline Storage and Server side Javascript with Node.js
Here are the key points about GRASP patterns:
1. GRASP stands for General Responsibility Assignment Software Patterns. It is a set of 9 patterns used to assign responsibilities to classes in object-oriented design.
2. Properly assigning responsibilities and distributing them among classes is a core part of object-oriented design. Developers and designers should be familiar with these patterns.
3. The 9 GRASP patterns are: Information Expert, Creator, Controller, Low Coupling, High Cohesion, Indirection, Polymorphism, Pure Fabrication, and Protected Variations. Each pattern provides guidance on distributing responsibilities in a certain context.
4. When designing a system, analyzing which classes should have which responsibilities based
The document is a presentation by Clément Sauvage about Realm, a new mobile database. It introduces Realm as being amazingly simple to use, requiring only a few lines of code to instantiate and query. Realm uses object-oriented classes to define schemas and supports common operations like queries, sorting, relations between objects, and migrations between schema versions. The presentation provides examples of adding, querying, and relating objects to demonstrate Realm's ease of use compared to other mobile databases.
The document discusses concepts related to event loops and concurrency in programming. It includes code snippets in Java showing the use of NIO selectors and channels for non-blocking I/O. Diagrams are shown illustrating reactor patterns and Vert.x modules deployed across multiple CPUs. The summary provides an overview of the main topics and techniques discussed in the technical document.
The history of asynchronous programming in .NET began with threads and the thread pool to handle concurrency. Later, patterns like the Asynchronous Programming Model (APM) and Event-based Asynchronous Pattern (EAP) simplified asynchronous code but it remained complex. The Task Parallel Library (TPL) and async/await further abstracted asynchronous operations so code more closely resembled synchronous code. Now, asynchronous code no longer requires Wait() or GetAwaiter().GetResult() and avoids potential deadlocks.
The document discusses extending Node.js using C++. It covers wrapping C++ classes for use in JavaScript, including setters, getters, exceptions, and instantiating objects. Examples are provided for wrapping Book and Person classes to be accessible from JavaScript. Building the extension using waf or gyp build systems is also covered. The presenter's background and the agenda are outlined initially.
- Rcpp is a package that facilitates interoperability between R and C++ by providing data structures and functions that make it easy to write C++ code that integrates with R. It has been released 54 times since 2008 with over 170 CRAN packages depending on it.
- Rcpp allows users to source C++ code from R using sourceCpp() and export C++ functions to R using attributes like // [[Rcpp::export]]. This improves performance over pure R code by leveraging fast C++ implementations.
- dplyr is a popular R package for data manipulation that achieves great performance through its use of Rcpp. Functions like arrange(), filter(), and summarise() are much faster when
This document discusses background processes in Android, including threads, services, and notifications. It explains that threads are used to perform long-running tasks without blocking the main UI thread. Services are intended for long-running background tasks rather than threads, and can be started and stopped from an activity. Notifications allow services to display status information in the status bar and notify the user of events even when the app is not visible.
Let'Swift 2019 컨퍼런스에서 RxSwift to Combine 이라고 발표한 자료입니다. 자료 내에 언급되는 코드 링크는 다음과 같습니다.
[BringMyOwnBeer]
• RxSwift/RxCocoa: https://github.com/fimuxd/BringMyOwnBeer-
• Combine/SwiftUI: https://github.com/fimuxd/BringMyOwnBeer-Combine
Let'Swift 2019 컨퍼런스에서 RxSwift to Combine 이라고 발표한 자료입니다. 자료 내에 언급되는 코드 링크는 다음과 같습니다.
[BringMyOwnBeer]
• RxSwift/RxCocoa: https://github.com/fimuxd/BringMyOwnBeer-
• Combine/SwiftUI: https://github.com/fimuxd/BringMyOwnBeer-Combine
- Python uses reference counting and a cyclic garbage collector to manage memory and free objects that are no longer referenced. It allocates memory for objects in blocks and assigns them to size classes based on their size.
- Objects in Python have a type and attributes. They are created via type constructors and can have specific attributes like __dict__, __slots__, and descriptors.
- At the Python virtual machine level, code is represented as code objects that contain details like the bytecode, constants, and variable names. Code objects are compiled from Python source files.
The document provides an overview of key differences between Python and Scala. Some key points summarized:
1. Python is a dynamically typed, interpreted language while Scala is statically typed and compiles to bytecode. Scala supports both object-oriented and functional programming paradigms.
2. Scala has features like case classes, traits, and pattern matching that Python lacks. Scala also has features like type parameters, implicit conversions, and tail call optimization that Python does not support natively.
3. Common data structures like lists and maps are implemented differently between the languages - Scala uses immutable Lists while Python uses mutable lists. Scala also has features like lazy vals.
4. Control
The document discusses asynchronous programming with futures and promises in Scala. It covers key topics like:
1) Futures provide a way to reason about parallel operations efficiently in a non-blocking way. Futures represent asynchronous computations with results that can be accessed later.
2) Promises allow completing a future with a value or exception. Callbacks allow accessing future results asynchronously once completed.
3) Functional composition with futures uses methods like map, flatMap, recover to transform and compose futures. For-comprehensions provide a cleaner way to compose futures.
Streams allow connecting programs like connecting garden hoses. They provide a way to pipe input through multiple steps to an output. There are different types of streams like readable, writable, duplex, and transform streams. Common uses of streams in Node.js include http, fs, child processes, tcp, and zlib. Streams are event emitters that can be used to process data incrementally instead of all at once.
Griffon is a desktop application framework inspired by Grails that leverages Swing and Groovy. It follows conventions like convention over configuration and pervasive MVC. Griffon emphasizes writing modular code, automated testing, and deploying applications easily to applets, webstart, or desktop without code changes. The presentation discusses porting legacy apps to Griffon by embracing its tenets like data binding, simpler layout managers like MigLayout, and using SwingWorker for long background tasks.
Joblib is a Python module that provides tools for parallel computing such as caching results on disk, efficient persistence of large numpy arrays, and support for multiple parallel backends including threading, multiprocessing, distributed, and ipyparallel. Recent updates have improved Joblib's persistence functionality by storing multiple arrays in a single file without memory copies, supporting additional compression formats like gzip and bz2, and allowing custom parallel backends. Future work may include persisting directly to file-like objects and remote storage, as well as using Joblib in cloud computing environments.
Joblib Toward efficient computing : from laptop to cloudPyDataParis
Joblib is a Python module that provides tools for parallel computing such as caching results on disk, efficient persistence of large numpy arrays, and support for multiple parallel backends including threading, multiprocessing, distributed, and ipyparallel. Recent updates have improved Joblib's persistence functionality by allowing multiple arrays to be stored in a single file without memory copies, supporting additional compression formats like gzip and bz2, and enabling custom parallel backends. Future work may include persisting directly to file-like objects and remote storage as well as using Joblib in cloud computing environments.
Euro python2011 High Performance PythonIan Ozsvald
I ran this as a 4 hour tutorial at EuroPython 2011 to teach High Performance Python coding.
Techniques covered include bottleneck analysis by profiling, bytecode analysis, converting to C using Cython and ShedSkin, use of the numerical numpy library and numexpr, multi-core and multi-machine parallelisation and using CUDA GPUs.
Write-up with 49 page PDF report: http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamFlink Forward
http://flink-forward.org/kb_sessions/no-shard-left-behind-dynamic-work-rebalancing-in-apache-beam/
The Apache Beam (incubating) programming model is designed to support several advanced data processing features such as autoscaling and dynamic work rebalancing. In this talk, we will first explain how dynamic work rebalancing not only provides a general and robust solution to the problem of stragglers in traditional data processing pipelines, but also how it allows autoscaling to be truly effective. We will then present how dynamic work rebalancing works as implemented in Google Cloud Dataflow and which path other Apache Beam runners link Apache Flink can follow to benefit from it.
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...Flink Forward
The Apache Beam programming model is designed to support several advanced data processing features such as autoscaling and dynamic work rebalancing. In this talk, we will first explain how dynamic work rebalancing not only provides a general and robust solution to the problem of stragglers in traditional data processing pipelines, but also how it allows autoscaling to be truly effective. We will then present how dynamic work rebalancing works as implemented in the Google Cloud Dataflow runner and which path other Apache Beam runners link Apache Flink can follow to benefit from it.
The document discusses several Java and Android internals topics:
1. How ArrayList and StringBuilder work internally using arrays and memory copying as the size increases. This can lead to inefficient memory usage.
2. How inner classes are implemented by compilers by generating additional accessor methods, increasing method count and affecting optimizations.
3. How the Android zygote process improves startup and memory usage by loading the framework once and sharing it across apps.
4. How the CPU cache works and how optimizing code to improve cache locality can significantly increase performance despite doing less work.
5. Issues like memory fragmentation that can occur if the Android garbage collector and compactor are unable to run due to the app being
The primary focus of this presentation is approaching the migration of a large, legacy data store into a new schema built with Django. Includes discussion of how to structure a migration script so that it will run efficiently and scale. Learn how to recognize and evaluate trouble spots.
Also discusses some general tips and tricks for working with data and establishing a productive workflow.
The document discusses strategies for migrating large amounts of legacy data from an old database into a new Django application. Some key points:
- Migrating data in batches and minimizing database queries per row processed can improve performance for large datasets.
- Tools like SQLAlchemy and Maatkit can help optimize the migration process.
- It's important to profile queries, enable logging/debugging, and design migrations that can resume/restart after failures or pause for maintenance.
- Preserving some legacy metadata like IDs on the new models allows mapping data between the systems. Declarative and modular code helps scale the migration tasks.
Accelerating the Development of Efficient CP Optimizer ModelsPhilippe Laborie
The IBM Constraint Programming optimization system CP Optimizer was designed to provide automatic search and a simple modeling of discrete optimization problems, with a particular focus on scheduling applications. It is used in industry for solving operational planning and scheduling problems. We will give an overview of CP Optimizer and then describe in further detail a set of features such as input/output file format, warm-start or conflict refinement that help accelerate the development of efficient models.
This document discusses Criteo's use of SummingBird, an open source framework that allows writing data processing jobs once to run on multiple platforms like Storm, Scalding, and Spark. It handles ingesting large amounts of event data from servers into Hadoop and processing it in real-time with Storm. SummingBird uses concepts like monoids and injections to merge results and handle data types across systems. This allows Criteo's analysts to query both real-time and batch processed data to gain insights from billions of events per day.
This document provides an overview of the bonobo Python library for simple ETL (extract, transform, load) workflows. It discusses the history and concepts of ETL, demonstrates basic usage of bonobo to define extract, transform, and load functions in a graph, and outlines the library's future plans, which include adding support for scheduling, monitoring, and more execution strategies. The goal of bonobo is to provide a Pythonic way to write ETL jobs using code as configuration in a simple and testable way.
This document provides tips and strategies for optimizing Java application performance through software tuning techniques. It discusses identifying and addressing bottlenecks, avoiding unnecessary object creation, using string pooling and interned strings efficiently, leveraging profilers to analyze performance issues, and optimizing loops and exception handling. The key strategies outlined are to reduce object creation, reuse objects when possible, compare strings effectively, and eliminate unnecessary method calls in loops.
The document discusses asynchronous programming using async and await in C#. It begins by explaining what asynchronous programming is and why it is useful for improving app responsiveness and simplifying asynchronous code. It then describes how async and await works by generating state machines and using continuation tasks. The document covers some gotchas with async code as well as best practices like naming conventions. It provides references for further reading on asynchronous patterns, tasks, and unit testing asynchronous code.
Keynote talk at PyCon Estonia 2019 where I discuss how to extend CPython and how that has led to a robust ecosystem around Python. I then discuss the need to define and build a Python extension language I later propose as EPython on OpenTeams: https://openteams.com/initiatives/2
Matt Franklin - Apache Software (Geekfest)W2O Group
The document discusses the potential benefits of container technologies like Docker. It notes that containers offer significantly higher density than virtual machines by avoiding hypervisor overhead. This density improvement can lead to major cost reductions by reducing infrastructure needs. Containers also improve developer efficiency by making development environments portable and disposable. This allows more rapid experimentation and innovation, potentially translating to increased revenue. Technologies like Amazon Lambda take the on-demand aspects of containers even further by abstracting compute resources. The document promotes StackEngine as a solution for managing containers at scale in production environments.
This document provides tips and tools for improving Angular application runtime performance. It discusses measuring performance using Benchmark.js, Chrome Dev Tools profiler, and @angular/benchpress. Specific optimizations covered include using the Ivy renderer, lazy loading modules, pre-rendering with Angular Universal, caching requests with service workers, immutable data structures, memoization, trackBy in *ngFor, virtual scrolling, RxJS operators, and controlling change detection. The document aims to help developers measure and improve the runtime performance of their Angular applications.
With Anaconda (in particular Numba and Dask) you can scale up your NumPy and Pandas stack to many cpus and GPUs as well as scale-out to run on clusters of machines including Hadoop.
The genesis of clusterlib - An open source library to tame your favourite sup...Arnaud Joly
The presentations tells the story of clusterlib an open source package from the problem statement to a first grade an open source library. Awesome tools are also presented for software projects.
The goal of the clusterlib is to ease the creation, launch and management of embarrassingly parallel jobs on supercomputers with schedulers such as SLURM and SGE.
Voxxed Days Vienna - The Why and How of Reactive Web-Applications on the JVMManuel Bernhardt
The document discusses the need for reactive and functional programming approaches to build scalable applications that can take advantage of many-core processors and distributed systems. It introduces key concepts like immutability, functions, and declarative programming. Specific frameworks like Scala, Play and Akka are presented as tools that support this reactive, functional style for building web applications that can horizontally scale across multiple cores and nodes. The talk promotes adopting these approaches to build systems that can better handle concurrency, distribution and failure.
Similar to PyParis2017 / Cloud computing made easy in Joblib, by Alexandre Abadie (20)
Short-range wireless communication technologies such as Bluetooth or ZigBee represent an important part of the Internet of Things ecosystem.
By design, this category of smart devices has physically limited reachability inside their Wireless Personal Area Network (WPAN) and are not directly compatible with the TCP/IP stack.
However, users may need to access them from anywhere at any moment.
To address this problem, we design a new application-agnostic approach called RCM (Remote Connection Manager) enabling transparent communication between an application and out-of-range devices.
It creates new IoT use cases by seamlessly mixing remote and local devices.
We implemented an open-source prototype for Bluetooth Low Energy (BLE) technology on top of Linux and Android BLE stacks and demonstrated its efficiency through experiments performed on real devices.
SAFC est un nouveau framework d’ordonnancement des conteneurs dans le cloud basé sur un modèle économique. La nouveauté de SAFC est qu’il permet de décider automatiquement quel est le nombre de ressources allouées pour chaque conteneur.
On parle d’observabilité des services lorsque ceux-ci exposent des états et métriques internes pour améliorer la disponibilité globale.
Qu’en est-il de l’observabilité des infrastructures sur lesquelles ils sont déployés, configurés et maintenus ?
Les différents logs (centralisés, agrégés) permettent un bon début d’analyse mais il faut aussi observer les systèmes au fil de l’eau pour tracer chaque changement et les corréler avec le monitoring. Aujourd’hui, ces étapes de configuration IT devraient être prises en charge par les outils de gestion de configuration, qui deviennent la passerelle vers l’observabilité des opérations.
Nous montrerons l'intérêt de cette approche pour la gestion IT moderne avec un retour d’expérience sur les challenges de leur mise en place dans Rudder, notre solution libre d’audit et de gestion de configuration en continu.
My research is in virtualized infrastructure domain. I aim at minimizing electricity consumption while improving application performance. To achieve the first goal, I work both at the entire datacenter level (by providing better VM placement strategies) and at the physical machine level (by providing better power management policies). Concerning the second goal, I work both at the VM monitor level (for minimizing its overhead) and at the VM's operating system (OS) level (for making it aware of the fact that it is virtualized).
In this talk I present two contributions of my research team, one for each objective.
The first contribution presents Drowsy-DC, a novel way to reduce data center power consumption inspired by smartphones.
The second contribution presents XPV (eXtended Para-Virtualization), a new principle for well virtualizing NUMA machines.
L'expérience du développement de CRESON, support pour des objets distants fortement cohérents dans Infinispan, par Etienne Riviere (UCLouvain).
Cet exposé présentera des résultats obtenus dans le cadre du projet européen LEADS que j'ai coordonné et où l'entreprise Red Hat était partenaire. Le code produit a été intégré dans le “staging" de la base de données NoSQL Infinispan, et évalué avec un équivalent open source de Dropbox développé par CloudSpaces, un autre projet européen.
L'approche de virtualisation en micro-services entraine des difficultés natives dans le capacity planning. La consommation de ressources des services déployés étant élastique et fonction du volume de requêtes / appels reçus par ce service.
The conference will describe the main concepts of security for embedded and IoT solutions : security vs safety, IT vs OT, main standards, level of security of available operating systems (Linux, Android, etc.), examples of attacks and secure solutions.
Pointers are a notorious "defect attractor", in particular when dynamic memory management is involved. Ada mitigates these issues by having much less need for pointers overall (thanks to first-class arrays, parameter modes, generics) and stricter rules for pointer manipulations that limit access to dangling memory. Still, dynamic memory management in Ada may lead to use-after-free, double-free and memory leaks, and dangling memory issues may lead to runtime exceptions.
The SPARK subset of Ada is focused on making it possible to guarantee properties of the program statically, in particular the absence of programming language errors, with a mostly automatic analysis. For that reason, and because static analysis of pointers is notoriously hard to automate, pointers have been forbidden in SPARK until now. We are working at AdaCore since 2017 on including pointer support in SPARK by restricting the use of pointers in programs so that they respect "ownership" constraints, like what is found in Rust.
In this talk, I will present the current state of the ownership rules for pointer support in SPARK, and the current state of the implementation in the GNAT compiler and GNATprove prover, as well as our roadmap for the future.
Durant ce talk Laurent Chemla revient sur l'expérience au niveau du projet Open Source Caliopen pour la création d'un commun et la mise en place d'une communauté.
Il abordera plusieurs questions essentielles dans la vie un projet Open Source tel que:
- Qu'est-ce qu'un commun et comment il naît?
- Qui vient en premier le commun ou la communauté qui soutient ce commun?
La virtualisation est une technologie mature dont le surcoût est aujourd’hui marginal sur les machines grand public. Néamoins, ce surcoût augmente radicalement pour les machines reposant sur une architecture Non Uniform Memory Access (NUMA), omniprésentes dans les data centers. Les techniques de virtualisation actuelles exploitent mal cette architecture et causent une dégradation des performances des applications allant jusqu’à 700%. Cette présentation détaille les causes de telles dégradations et propose une méthode qui permet la virtualisation efficace d’architectures NUMA. Une évaluation de cette méthode montre qu’il est possible de multiplier par 2 ou plus la performance de 9 des 29 applications testées.
This document describes DeepTorrent, a distributed storage system based on Bittorrent that aims to provide permanent storage. It discusses issues with cloud storage like high energy consumption, limited security, and legal restrictions. DeepTorrent uses Python and C libraries and Redis for control flow. It allows uploading and downloading files and managing storers. DeepTorrent is being tested for archiving production logs and medical imaging data. Future work includes access control and distributed control flow using DHT.
The document discusses the Software Heritage initiative, which aims to collect, preserve, and share the source code of all publicly available software. It notes that software is culturally and economically important but fragile, as source code is often lost. The initiative builds an archive of software source code and history using free and open source methods. It has already archived over 150 terabytes of open source software from repositories like GitHub, and seeks more support to continue its work of preserving software for current and future use.
Dans cet exposé, on présentera OMicroB, une machine virtuelle OCaml pour microcontrôleurs à faibles ressources, inspirée des travaux précédents sur le projet OCaPIC. Cette machine virtuelle, destinée à être exécutée sur diverses architectures matérielles (AVR, PIC, ARM, ...) permet ainsi de factoriser le développement d’applications, mais aussi de généraliser l’analyse et le débogage du bytecode associé, tout en permettant un usage précautionneux de la mémoire. On cible alors des programmes ludiques ou de domotiques destinés à être exécutés sur des microcontrôleurs à faibles ressources, en insistant sur les particularités inhérentes à la programmation de systèmes embarqués.
La société Farjump propose une solution simple, innovante et bon marché pour la mise au point des systèmes embarqués utilisés dans l'IoT. La solution est basé sur la mise en place d'agents GDB sur la cible.
Le principe est d’appliquer des mécanismes de contrôle dynamiques et fins sur les communications des objets (entre eux ou bien vers le cloud) sous le contrôle des utilisateurs.
The programming language Ada offers unique features to safely program a micro-controller. From the start, Ada was designed to make it difficult to introduce errors, and to make it easy to discover errors that were introduced. For example, language rules enforced at compile time make it possible to have safe concurrency by design. And run-time checking allows immediate detection of what would be "undefined behavior" in C/C++. In the first part of this presentation, we will present the benefits of using Ada for micro-controller programming, including support for debugging on a board. In the second part of this presentation, we will present how the Ada language and its subset SPARK provide a strong foundation for static analyzers, that make it possible to detect errors and provide guarantees on embedded software in Ada/SPARK.
Nous présenterons le système d'exploitation RTEMS, ses applications passées et actuelles ainsi que les travaux en cours pour son utilisation dans l'IoT professionnel.
The document discusses a MOOC on Python programming offered on the FUN platform in France. It provides details on the audience, which is French-speaking students with some exposure to programming. Around 33,000 students have registered across 3 sessions so far. The course materials include videos, quizzes, and Jupyter notebooks hosted on a separate infrastructure that allows notebooks to be viewed within the online course platform. The content covers basic and advanced Python concepts across 7 weeks. Feedback is welcomed to help update the course for an upcoming session that will transition to teaching Python 3.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Speck&Tech
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Infrastructure Challenges in Scaling RAG with Custom AI modelsZilliz
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.
6. The ecosystem
54 different contributors since the beginning in 2008
Contributors per month
Joblib is the computing backend used by Scikit-Learn
7. The ecosystem
54 different contributors since the beginning in 2008
Contributors per month
Joblib is the computing backend used by Scikit-Learn
Stable and mature code base
https://github.com/joblib/joblib
10. Why Joblib?
Because we want to make use of all available computing resources
⇒ And ensure algorithms run as fast as possible
11. Why Joblib?
Because we want to make use of all available computing resources
⇒ And ensure algorithms run as fast as possible
Because we work on large datasets
12. Why Joblib?
Because we want to make use of all available computing resources
⇒ And ensure algorithms run as fast as possible
Because we work on large datasets
⇒ Data that just fits in RAM
13. Why Joblib?
Because we want to make use of all available computing resources
⇒ And ensure algorithms run as fast as possible
Because we work on large datasets
⇒ Data that just fits in RAM
Because we want the internal algorithm logic to remain unchanged
14. Why Joblib?
Because we want to make use of all available computing resources
⇒ And ensure algorithms run as fast as possible
Because we work on large datasets
⇒ Data that just fits in RAM
Because we want the internal algorithm logic to remain unchanged
⇒ Adapted to embarrassingly parallel problems
15. Why Joblib?
Because we want to make use of all available computing resources
⇒ And ensure algorithms run as fast as possible
Because we work on large datasets
⇒ Data that just fits in RAM
Because we want the internal algorithm logic to remain unchanged
⇒ Adapted to embarrassingly parallel problems
Because we love simple APIs
16. Why Joblib?
Because we want to make use of all available computing resources
⇒ And ensure algorithms run as fast as possible
Because we work on large datasets
⇒ Data that just fits in RAM
Because we want the internal algorithm logic to remain unchanged
⇒ Adapted to embarrassingly parallel problems
Because we love simple APIs
⇒ And parallel programming is not user friendly in general
18. How?
Embarrassingly Parallel computing helper
⇒ make parallel computing easy
Efficient disk caching to avoid recomputation
⇒ computation resource friendly
19. How?
Embarrassingly Parallel computing helper
⇒ make parallel computing easy
Efficient disk caching to avoid recomputation
⇒ computation resource friendly
Fast I/O persistence
⇒ limit cache access time
20. How?
Embarrassingly Parallel computing helper
⇒ make parallel computing easy
Efficient disk caching to avoid recomputation
⇒ computation resource friendly
Fast I/O persistence
⇒ limit cache access time
No dependencies, optimized for numpy arrays
⇒ simple installation and integration in other projects
22. Parallel helper
>>> from joblib import Parallel, delayed
>>> from math import sqrt
>>> Parallel(n_jobs=3, verbose=50)(delayed(sqrt)(i**2) for i in range(6))
[Parallel(n_jobs=3)]: Done 1 tasks | elapsed: 0.0s
[...]
[Parallel(n_jobs=3)]: Done 6 out of 6 | elapsed: 0.0s finished
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0]
23. Parallel helper
>>> from joblib import Parallel, delayed
>>> from math import sqrt
>>> Parallel(n_jobs=3, verbose=50)(delayed(sqrt)(i**2) for i in range(6))
[Parallel(n_jobs=3)]: Done 1 tasks | elapsed: 0.0s
[...]
[Parallel(n_jobs=3)]: Done 6 out of 6 | elapsed: 0.0s finished
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0]
⇒ API can be extended with external backends
25. Parallel backends
Single machine backends: works on a Laptop
⇒ threading, multiprocessing and soon Loky
Multi machine backends: available as optional extensions
⇒ distributed, ipyparallel, CMFActivity, Hadoop Yarn
26. Parallel backends
Single machine backends: works on a Laptop
⇒ threading, multiprocessing and soon Loky
Multi machine backends: available as optional extensions
⇒ distributed, ipyparallel, CMFActivity, Hadoop Yarn
>>> from distributed.joblib import DistributedBackend
>>> from joblib import (Parallel, delayed,
>>> register_parallel_backend, parallel_backend)
>>> register_parallel_backend('distributed', DistributedBackend)
>>> with parallel_backend('distributed', scheduler_host='dscheduler:8786'):
>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(6))
[...]
27. Parallel backends
Single machine backends: works on a Laptop
⇒ threading, multiprocessing and soon Loky
Multi machine backends: available as optional extensions
⇒ distributed, ipyparallel, CMFActivity, Hadoop Yarn
>>> from distributed.joblib import DistributedBackend
>>> from joblib import (Parallel, delayed,
>>> register_parallel_backend, parallel_backend)
>>> register_parallel_backend('distributed', DistributedBackend)
>>> with parallel_backend('distributed', scheduler_host='dscheduler:8786'):
>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(6))
[...]
Future: new backends for Celery, Spark
28. Caching on disk
Use a memoize pattern with the Memory object
>>> from joblib import Memory
>>> import numpy as np
>>> a = np.vander(np.arange(3)).astype(np.float)
>>> mem = Memory(cachedir='/tmp/joblib')
>>> square = mem.cache(np.square)
29. Caching on disk
Use a memoize pattern with the Memory object
>>> from joblib import Memory
>>> import numpy as np
>>> a = np.vander(np.arange(3)).astype(np.float)
>>> mem = Memory(cachedir='/tmp/joblib')
>>> square = mem.cache(np.square)
>>> b = square(a)
________________________________________________________________________________
[Memory] Calling square...
square(array([[ 0., 0., 1.],
[ 1., 1., 1.],
[ 4., 2., 1.]]))
___________________________________________________________square - 0...s, 0.0min
>>> c = square(a) # no recomputation
array([[ 0., 0., 1.],
[...]
30. Caching on disk
Use a memoize pattern with the Memory object
>>> from joblib import Memory
>>> import numpy as np
>>> a = np.vander(np.arange(3)).astype(np.float)
>>> mem = Memory(cachedir='/tmp/joblib')
>>> square = mem.cache(np.square)
>>> b = square(a)
________________________________________________________________________________
[Memory] Calling square...
square(array([[ 0., 0., 1.],
[ 1., 1., 1.],
[ 4., 2., 1.]]))
___________________________________________________________square - 0...s, 0.0min
>>> c = square(a) # no recomputation
array([[ 0., 0., 1.],
[...]
Least Recently Used (LRU) cache replacement policy
31. Persistence
Convert/create an arbitrary object into/from a string of bytes
Streamable persistence to/from file or socket objects
>>> import numpy as np
>>> import joblib
>>> obj = [('a', [1, 2, 3]), ('b', np.arange(10))]
>>> joblib.dump(obj, '/tmp/test.pkl')
['/tmp/test.pkl']
>>> with open('/tmp/test.pkl', 'rb') as f:
>>> joblib.load(f)
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
32. Persistence
Convert/create an arbitrary object into/from a string of bytes
Streamable persistence to/from file or socket objects
>>> import numpy as np
>>> import joblib
>>> obj = [('a', [1, 2, 3]), ('b', np.arange(10))]
>>> joblib.dump(obj, '/tmp/test.pkl')
['/tmp/test.pkl']
>>> with open('/tmp/test.pkl', 'rb') as f:
>>> joblib.load(f)
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
Use compression for fast I/O:
support for zlib, gz, bz2, xz and lzma compressors
>>> joblib.dump(obj, '/tmp/test.pkl.gz', compress=True, cache_size=0)
['/tmp/test.pkl.gz']
>>> joblib.load('/tmp/test.pkl.gz')
35. The Cloud trend
Lots of Cloud providers on the market:
Existing solutions for processing Big Data:
36. The Cloud trend
Lots of Cloud providers on the market:
Existing solutions for processing Big Data:
Existing container orchestration solutions: Docker SWARM, Kubernetes
37. The Cloud trend
Lots of Cloud providers on the market:
Existing solutions for processing Big Data:
Existing container orchestration solutions: Docker SWARM, Kubernetes
How can Joblib be used with them?
39. Use pluggable multi-machine parallel backends
Principle: configure your backend and wrap the calls to Parallel
>>> import time
>>> import ipyparallel as ipp
>>> from ipyparallel.joblib import register as register_joblib
>>> from joblib import parallel_backend, Parallel, delayed
# Setup ipyparallel backend
>>> register_joblib()
>>> dview = ipp.Client()[:]
# Start the job
>>> with parallel_backend("ipyparallel", view=dview):
>>> Parallel(n_jobs=20, verbose=50)(delayed(time.sleep)(1) for i in range(10))
40. Use pluggable multi-machine parallel backends
Principle: configure your backend and wrap the calls to Parallel
>>> import time
>>> import ipyparallel as ipp
>>> from ipyparallel.joblib import register as register_joblib
>>> from joblib import parallel_backend, Parallel, delayed
# Setup ipyparallel backend
>>> register_joblib()
>>> dview = ipp.Client()[:]
# Start the job
>>> with parallel_backend("ipyparallel", view=dview):
>>> Parallel(n_jobs=20, verbose=50)(delayed(time.sleep)(1) for i in range(10))
Complete examples exist for:
Dask distributed: https://github.com/ogrisel/docker-distributed
Hadoop Yarn: https://github.com/joblib/joblib-hadoop
41. Use pluggable store backends
Extends Memory API with other store providers
Not available upstream yet:
⇒ PR opened at https://github.com/joblib/joblib/pull/397
42. Use pluggable store backends
Extends Memory API with other store providers
Not available upstream yet:
⇒ PR opened at https://github.com/joblib/joblib/pull/397
>>> import numpy as np
>>> from joblib import Memory
>>> from joblibhadoop.hdfs import register_hdfs_store_backend
# Register HDFS store backend provider
>>> register_hdfs_store_backend()
# Persist data in hdfs://namenode:9000/user/john/cache/joblib
>>> mem = Memory(location='cache', backend='hdfs',
>>> host='namenode', port=9000, user='john', compress=True)
multiply = mem.cache(np.multiply)
43. Use pluggable store backends
Extends Memory API with other store providers
Not available upstream yet:
⇒ PR opened at https://github.com/joblib/joblib/pull/397
>>> import numpy as np
>>> from joblib import Memory
>>> from joblibhadoop.hdfs import register_hdfs_store_backend
# Register HDFS store backend provider
>>> register_hdfs_store_backend()
# Persist data in hdfs://namenode:9000/user/john/cache/joblib
>>> mem = Memory(location='cache', backend='hdfs',
>>> host='namenode', port=9000, user='john', compress=True)
multiply = mem.cache(np.multiply)
Store backends available:
Amazon S3: https://github.com/aabadie/joblib-s3
Hadoop HDFS: https://github.com/joblib/joblib-hadoop
44. Using Hadoop with Joblib
joblib-hadoop package: https://github.com/joblib/joblib-hadoop
45. Using Hadoop with Joblib
joblib-hadoop package: https://github.com/joblib/joblib-hadoop
Provides docker containers helpers for developing and testing
46. Using Hadoop with Joblib
joblib-hadoop package: https://github.com/joblib/joblib-hadoop
Provides docker containers helpers for developing and testing
⇒ no need for a production Hadoop cluster
⇒ make developer life easier: CI on Travis is possible
⇒ local repository on host is shared with Joblib-hadoop-node container
47. Using Hadoop with Joblib
joblib-hadoop package: https://github.com/joblib/joblib-hadoop
Provides docker containers helpers for developing and testing
⇒ no need for a production Hadoop cluster
⇒ make developer life easier: CI on Travis is possible
⇒ local repository on host is shared with Joblib-hadoop-node container
48. Outline
Joblib in a word
Joblib for cloud computing
⇒Future work and conclusion
50. Future work
In-memory object caching
⇒ Should save RAM during a parallel job
Allow overriding of parallel backends
⇒ See PR: https://github.com/joblib/joblib/pull/524
⇒ Seamless distributed computing in scikit-learn
51. Future work
In-memory object caching
⇒ Should save RAM during a parallel job
Allow overriding of parallel backends
⇒ See PR: https://github.com/joblib/joblib/pull/524
⇒ Seamless distributed computing in scikit-learn
Replace multiprocessing parallel backend with Loky
⇒ See PR: https://github.com/joblib/joblib/pull/516
52. Future work
In-memory object caching
⇒ Should save RAM during a parallel job
Allow overriding of parallel backends
⇒ See PR: https://github.com/joblib/joblib/pull/524
⇒ Seamless distributed computing in scikit-learn
Replace multiprocessing parallel backend with Loky
⇒ See PR: https://github.com/joblib/joblib/pull/516
Extend Cloud providers support
⇒ Using Apache libcloud: give access to a lot more Cloud providers
55. Conclusion
Parallel helper is adapted to embarassingly parallel problems
Already a lot of parallel backends available
⇒ threading, multiprocessing, loky, CMFActivity distributed, ipyparallel, Yarn
56. Conclusion
Parallel helper is adapted to embarassingly parallel problems
Already a lot of parallel backends available
⇒ threading, multiprocessing, loky, CMFActivity distributed, ipyparallel, Yarn
Use caching techniques to avoid recomputation
57. Conclusion
Parallel helper is adapted to embarassingly parallel problems
Already a lot of parallel backends available
⇒ threading, multiprocessing, loky, CMFActivity distributed, ipyparallel, Yarn
Use caching techniques to avoid recomputation
Extra Store backends available ⇒ HDFS (Hadoop) and AWS S3
58. Conclusion
Parallel helper is adapted to embarassingly parallel problems
Already a lot of parallel backends available
⇒ threading, multiprocessing, loky, CMFActivity distributed, ipyparallel, Yarn
Use caching techniques to avoid recomputation
Extra Store backends available ⇒ HDFS (Hadoop) and AWS S3
Use Joblib either on your laptop or in a Cloud with very few code changes