This document summarizes the CoCoA algorithm for distributed optimization. CoCoA uses a primal-dual framework to solve machine learning problems efficiently when data is distributed across multiple machines. It allows local machines to immediately apply updates to their local dual variables, while averaging the local primal updates over a small number of machines. CoCoA guarantees convergence, requires low communication, and can be implemented in just a few lines of code in systems like Spark. It improves upon mini-batch approaches by handling methods beyond stochastic gradient descent and avoiding issues with stale updates.
Graph theory could potentially make a big impact on how we conduct businesses. Imagine the case where you wish to maximize the reach of your promotion via leveraging your customers' influence, to advocate your products and bring their friends on board. The same logic of harnessing one's networks can be applied to purchase recommendation, customer behavior, and fraud detection.
Running analyses on large graphs was not trivial for many companies - until recently. The field has made significant steps in the last five years and scalable graph computations are now the norm. You can now run graph computations out-of-core (no memory constraints) and in parallel (multiple machines), especially in Spark which is spreading like wildfire.
A lot of people are familiar with graphX, a pretty solid implementation of scalable graphs in Spark. GraphX is pretty interesting but the project seems to be orphaned. The good news is, there is now an alternative: Graphframes. They are a new data structure that takes the best parts of dataframes and graphs
In this talk, I will be explaining how to use Graphframes from Python, a new data structure in Spark 2.0 that takes the best parts of dataframes and graphs, with an example using personalized pagerank for recommendations.
Best Practices for Hyperparameter Tuning with MLflowDatabricks
Hyperparameter tuning and optimization is a powerful tool in the area of AutoML, for both traditional statistical learning models as well as for deep learning. There are many existing tools to help drive this process, including both blackbox and whitebox tuning. In this talk, we'll start with a brief survey of the most popular techniques for hyperparameter tuning (e.g., grid search, random search, Bayesian optimization, and parzen estimators) and then discuss the open source tools which implement each of these techniques. Finally, we will discuss how we can leverage MLflow with these tools and techniques to analyze how our search is performing and to productionize the best models.
Speaker: Joseph Bradley
Graph theory could potentially make a big impact on how we conduct businesses. Imagine the case where you wish to maximize the reach of your promotion via leveraging your customers' influence, to advocate your products and bring their friends on board. The same logic of harnessing one's networks can be applied to purchase recommendation, customer behavior, and fraud detection.
Running analyses on large graphs was not trivial for many companies - until recently. The field has made significant steps in the last five years and scalable graph computations are now the norm. You can now run graph computations out-of-core (no memory constraints) and in parallel (multiple machines), especially in Spark which is spreading like wildfire.
A lot of people are familiar with graphX, a pretty solid implementation of scalable graphs in Spark. GraphX is pretty interesting but the project seems to be orphaned. The good news is, there is now an alternative: Graphframes. They are a new data structure that takes the best parts of dataframes and graphs
In this talk, I will be explaining how to use Graphframes from Python, a new data structure in Spark 2.0 that takes the best parts of dataframes and graphs, with an example using personalized pagerank for recommendations.
Best Practices for Hyperparameter Tuning with MLflowDatabricks
Hyperparameter tuning and optimization is a powerful tool in the area of AutoML, for both traditional statistical learning models as well as for deep learning. There are many existing tools to help drive this process, including both blackbox and whitebox tuning. In this talk, we'll start with a brief survey of the most popular techniques for hyperparameter tuning (e.g., grid search, random search, Bayesian optimization, and parzen estimators) and then discuss the open source tools which implement each of these techniques. Finally, we will discuss how we can leverage MLflow with these tools and techniques to analyze how our search is performing and to productionize the best models.
Speaker: Joseph Bradley
Balancing Automation and Explanation in Machine LearningDatabricks
For a machine learning application to be successful, it is not enough to give highly accurate predictions: Customers also want to know why the model has made that prediction, so they can compare it against their intuition and (hopefully) gain trust in the model. However, there is a trade-off between model accuracy and explainability - for example, the more complex your feature transformations become, the harder it is to explain what the resulting features mean to the end customer. However, with the right system design this doesn't mean it has to be a binary choice between these two goals. It is possible to combine complex, even automatic, feature engineering with highly accurate models and explanations. We will describe how we are using lineage tracing to solve this issue at Salesforce Einstein, allowing good model explanations to coexist with automatic feature engineering and model selection. By building this into an open source AutoML library TransmogrifAI, an extension to SparkMlLib, it is easy to ensure a consistent level of transparency in all of our ML applications. As model explanations are provided out of the box, data scientists don't need to re-invent the wheel when model explanations need to be surfaced.
Practical Machine Learning Pipelines with MLlibDatabricks
This talk from 2015 Spark Summit East discusses Pipelines and related concepts introduced in Spark 1.2 which provide a simple API for users to set up complex ML workflows.
Online learning with structured streaming, spark summit brussels 2016Ram Sriharsha
Structured Streaming is a new API in Spark 2.0 that simplifies the end to end development of continuous applications. One such continuous application is online model updates: Online models are incrementally updated with new data and can be continuously queried while being updated. As a result, they can be fast to train and leverage new data faster than offline algorithms. In this talk, we give a brief introduction the area of online learning and describe how online model updates can be built using structured streaming APIs. The end result is a robust pipeline for updating models that is scalable, fast and fault tolerant.
Advanced Hyperparameter Optimization for Deep Learning with MLflowDatabricks
Building on the "Best Practices for Hyperparameter Tuning with MLflow" talk, we will present advanced topics in HPO for deep learning, including early stopping, multi-metric optimization, and robust optimization. We will then discuss implementations using open source tools. Finally, we will discuss how we can leverage MLflow with these tools and techniques to analyze the performance of our models.
The Challenges of Bringing Machine Learning to the MassesAlice Zheng
Why is it hard to build ML software, and why it is like designing a database. Jointly created with Sethu Raman (Dato/GraphLab). Talk at NIPS 2014 workshop on Software Engineering for Machine Learning (https://sites.google.com/site/software4ml/).
Machine learning techniques are powerful, but building and deploying such models for production use require a lot of care and expertise.
A lot of books, articles, and best practices have been written and discussed on machine learning techniques and feature engineering, but putting those techniques into use on a production environment is usually forgotten and under- estimated , the aim of this talk is to shed some lights on current machine learning deployment practices, and go into details on how to deploy sustainable machine learning pipelines.
Overview of Apache SystemML by Berthold Reinwald and Nakul JindalArvind Surve
This deck will provide SystemML architecture, how to get documentation for usage, algorithms etc. It will explain usage of it through command line or through notebook.
Virginia Smith, Researcher, UC Berkeley at MLconf SF 2016MLconf
A General Framework for Communication-Efficient Distributed Optimization: Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning. In light of this, we propose a general framework, CoCoA, that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary communication. Our framework enjoys strong convergence guarantees and exhibits state-of-the-art empirical performance in the distributed setting. We demonstrate this performance with extensive experiments in Apache Spark, achieving speedups of up to 50x compared to leading distributed methods on common machine learning objectives.
https://telecombcn-dl.github.io/2018-dlai/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks or Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles of deep learning from both an algorithmic and computational perspectives.
Balancing Automation and Explanation in Machine LearningDatabricks
For a machine learning application to be successful, it is not enough to give highly accurate predictions: Customers also want to know why the model has made that prediction, so they can compare it against their intuition and (hopefully) gain trust in the model. However, there is a trade-off between model accuracy and explainability - for example, the more complex your feature transformations become, the harder it is to explain what the resulting features mean to the end customer. However, with the right system design this doesn't mean it has to be a binary choice between these two goals. It is possible to combine complex, even automatic, feature engineering with highly accurate models and explanations. We will describe how we are using lineage tracing to solve this issue at Salesforce Einstein, allowing good model explanations to coexist with automatic feature engineering and model selection. By building this into an open source AutoML library TransmogrifAI, an extension to SparkMlLib, it is easy to ensure a consistent level of transparency in all of our ML applications. As model explanations are provided out of the box, data scientists don't need to re-invent the wheel when model explanations need to be surfaced.
Practical Machine Learning Pipelines with MLlibDatabricks
This talk from 2015 Spark Summit East discusses Pipelines and related concepts introduced in Spark 1.2 which provide a simple API for users to set up complex ML workflows.
Online learning with structured streaming, spark summit brussels 2016Ram Sriharsha
Structured Streaming is a new API in Spark 2.0 that simplifies the end to end development of continuous applications. One such continuous application is online model updates: Online models are incrementally updated with new data and can be continuously queried while being updated. As a result, they can be fast to train and leverage new data faster than offline algorithms. In this talk, we give a brief introduction the area of online learning and describe how online model updates can be built using structured streaming APIs. The end result is a robust pipeline for updating models that is scalable, fast and fault tolerant.
Advanced Hyperparameter Optimization for Deep Learning with MLflowDatabricks
Building on the "Best Practices for Hyperparameter Tuning with MLflow" talk, we will present advanced topics in HPO for deep learning, including early stopping, multi-metric optimization, and robust optimization. We will then discuss implementations using open source tools. Finally, we will discuss how we can leverage MLflow with these tools and techniques to analyze the performance of our models.
The Challenges of Bringing Machine Learning to the MassesAlice Zheng
Why is it hard to build ML software, and why it is like designing a database. Jointly created with Sethu Raman (Dato/GraphLab). Talk at NIPS 2014 workshop on Software Engineering for Machine Learning (https://sites.google.com/site/software4ml/).
Machine learning techniques are powerful, but building and deploying such models for production use require a lot of care and expertise.
A lot of books, articles, and best practices have been written and discussed on machine learning techniques and feature engineering, but putting those techniques into use on a production environment is usually forgotten and under- estimated , the aim of this talk is to shed some lights on current machine learning deployment practices, and go into details on how to deploy sustainable machine learning pipelines.
Overview of Apache SystemML by Berthold Reinwald and Nakul JindalArvind Surve
This deck will provide SystemML architecture, how to get documentation for usage, algorithms etc. It will explain usage of it through command line or through notebook.
Virginia Smith, Researcher, UC Berkeley at MLconf SF 2016MLconf
A General Framework for Communication-Efficient Distributed Optimization: Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning. In light of this, we propose a general framework, CoCoA, that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary communication. Our framework enjoys strong convergence guarantees and exhibits state-of-the-art empirical performance in the distributed setting. We demonstrate this performance with extensive experiments in Apache Spark, achieving speedups of up to 50x compared to leading distributed methods on common machine learning objectives.
https://telecombcn-dl.github.io/2018-dlai/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks or Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles of deep learning from both an algorithmic and computational perspectives.
Title: "Understanding PyTorch: PyTorch in Image Processing". Github: https://github.com/azarnyx/PyData_Meetup. The Dataset: https://goo.gl/CWmLWD.
The talk was given in PyData Meetup which took place in Munich on 06.03.2019 in Data Reply office. The talk was given by Dmitrii Azarnykh, data scientist in Data Reply.
Sat4j: from the lab to desktop computers. OW2con'15, November 17, Paris. OW2
The aim of the Sat4j library is to solve Boolean satisfaction and optimization problems. Those problems have received considerable attention in the last two decades, mainly due to its use in hardware verification.Sat4j started as a research project to experiment ideas about while providing an efficient Boolean reasoning engine to the Java community.
Le Song, Assistant Professor, College of Computing, Georgia Institute of Tech...MLconf
Understanding Deep Learning for Big Data: The complexity and scale of big data impose tremendous challenges for their analysis. Yet, big data also offer us great opportunities. Some nonlinear phenomena, features or relations, which are not clear or cannot be inferred reliably from small and medium data, now become clear and can be learned robustly from big data. Typically, the form of the nonlinearity is unknown to us, and needs to be learned from data as well. Being able to harness the nonlinear structures from big data could allow us to tackle problems which are impossible before or obtain results which are far better than previous state-of-the-arts.
Nowadays, deep neural networks are the methods of choice when it comes to large scale nonlinear learning problems. What makes deep neural networks work? Is there any general principle for tackling high dimensional nonlinear problems which we can learn from deep neural works? Can we design competitive or better alternatives based on such knowledge? To make progress in these questions, my machine learning group performed both theoretical and experimental analysis on existing and new deep learning architectures, and investigate three crucial aspects on the usefulness of the fully connected layers, the advantage of the feature learning process, and the importance of the compositional structures. Our results point to some promising directions for future research, and provide guideline for building new deep learning models.
The main challenge of concurrent software verification has always been in achieving modularity, i.e., the ability to divide and conquer the correctness proofs with the goal of scaling the verification effort. Types are a formal method well-known for its ability to modularize programs, and in the case of dependent types, the ability to modularize and scale complex mathematical proofs.
In this talk I will present our recent work towards reconciling dependent types with shared memory concurrency, with the goal of achieving modular proofs for the latter. Applying the type-theoretic paradigm to concurrency has lead us to view separation logic as a type theory of state, and has motivated novel abstractions for expressing concurrency proofs based on the algebraic structure of a resource and on structure-preserving functions (i.e., morphisms) between resources.
Practical and Worst-Case Efficient ApportionmentRaphael Reitzig
Proportional apportionment is the problem of assigning seats to parties according to their relative share of votes. Divisor methods are the de-facto standard solution, used in many countries.
In recent literature, there are two algorithms that implement divisor methods: one by Cheng and Eppstein (ISAAC, 2014) has worst-case optimal running time but is complex, while the other (Pukelsheim, 2014) is relatively simple and fast in practice but does not offer worst-case guarantees.
This talk presents the ideas behind a novel algorithm that avoids the shortcomings of both. We investigate the three contenders in order to determine which is most useful in practice.
Read more over here: http://reitzig.github.io/publications/RW2015b
Distributed Coordinate Descent for Logistic Regression with RegularizationИлья Трофимов
Logistic regression with L1 and L2 regularization is a widely used technique for solving
classi cation and class probability estimation problems. With the numbers of both featurescand examples growing rapidly in the fi elds like text mining and clickstream data analysis parallelization and the use of cluster architectures becomes important. We present a novel algorithm for tting regularized logistic regression in the distributed environment. The algorithm splits data between nodes by features, uses coordinate descent on each node and line search to merge results globally. Convergence proof is provided. A modi cations of the algorithm addresses slow node problem. We empirically compare our program with several state-of-the art approaches that rely on diff erent algorithmic and data spitting methods. Experiments demonstrate that our approach is scalable and superior when training on large and sparse datasets.
----------------------------------------------------------
Machine Learning: Prospects and Applications
58 October 2015, Berlin, Germany
Cloud Native Night, December 2020, talk by Jörg Viechtbauer (Senior Software Architect, QAware)
== Please download slides if blurred! ==
Abstract:
Neural networks like BERT have revolutionized the processing of natural language and achieve state-of-the-art performance in many NLP tasks. One of them is semantic search where documents are found by query intent and not only by exact match.
This talk takes us through the history of information retrieval and shows how keyword search has evolved into the term vector model. The desire for a better search led to the development of the first semantic models like SLI or PLSA. We will see how this culminates today in the use of sophisticated deep neural networks that perform nonlinear dimensional reductions and master long-range dependencies.
Semantic search has never been as good and easy to implement as it is today.
About Jörg:
Jörg is a search expert at QAware and uses neural networks for semantic search and text comprehension. He has spent almost 20 years developing search engines based on both proprietary and open source software for enterprise search, eDiscovery and local search - always hunting for the perfect ranking formula.
Speeding Up Distributed Machine Learning Using CodesNAVER Engineering
발표자: 이강욱 (KAIST 박사 후 연구원)
발표일: 2017.5.
Kangwook Lee is a postdoctoral scholar in the School of EE at KAIST, working with Prof. Changho Suh. He received his PhD degree in 2016 from the EECS department at UC Berkeley under the supervision of Prof. Kannan Ramchandran. He also obtained his MS degree in EECS from UC Berkeley in 2012, and BS degree in EE from KAIST in 2010.
목차:
1. Coded Computation
2. Coded Shuffling
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
1. COCOA
Communication-Efficient
Coordinate Ascent
Virginia Smith
!
Martin Jaggi, Martin Takáč, Jonathan Terhorst,
Sanjay Krishnan, Thomas Hofmann, & Michael I. Jordan
21. Example: SVM Classification
wx-b=1
wx-b=0 w
2 / ||w||
wx-b=-1
min
w2Rd
!
2 ||w||2 +
1
n
Xn
i=1
`hinge(yiwT xi)
22. Example: SVM Classification
wx-b=1
wx-b=0 w
2 / ||w||
wx-b=-1
min
w2Rd
!
2 ||w||2 +
1
n
Xn
i=1
`hinge(yiwT xi)
Descent algorithms and line search methods
Acceleration, momentum, and conjugate gradients
Newton and Quasi-Newton methods
Coordinate descent
Stochastic and incremental gradient methods
SMO
SVMlight
LIBLINEAR
23. Machine Learning Workflow
DATA & PROBLEM
classification, regression,
collaborative filtering, …
MACHINE LEARNING MODEL
logistic regression, lasso,
support vector machines, …
OPTIMIZATION ALGORITHM
gradient descent, coordinate
descent, Newton’s method, …
24. Machine Learning Workflow
DATA & PROBLEM
classification, regression,
collaborative filtering, …
MACHINE LEARNING MODEL
logistic regression, lasso,
support vector machines, …
OPTIMIZATION ALGORITHM
gradient descent, coordinate
descent, Newton’s method, …
SYSTEMS SETTING
multi-core, cluster, cloud,
supercomputer, …
25. Machine Learning Workflow
DATA & PROBLEM
classification, regression,
collaborative filtering, …
Open Problem:
MACHINE LEARNING MODEL
logistic regression, lasso,
support vector machines, …
efficiently solving objective
when data is distributed
OPTIMIZATION ALGORITHM
gradient descent, coordinate
descent, Newton’s method, …
SYSTEMS SETTING
multi-core, cluster, cloud,
supercomputer, …
32. Distributed Optimization
✔ convergence
guarantees
✗ high communication
“always communicate”
reduce: w = w ↵
P
k w
33. Distributed Optimization
✔ convergence
guarantees
✗ high communication
“always communicate”
reduce: w = w ↵
P
k w
34. Distributed Optimization
✔ convergence
guarantees
✗ high communication
average: w := 1K
P
k wk
“always communicate”
reduce: w = w ↵
P
k w
35. Distributed Optimization
✔ convergence
guarantees
✗ high communication
average: w := 1K
P
k wk
“always communicate”
“never communicate”
reduce: w = w ↵
P
k w
36. Distributed Optimization
✔ convergence
guarantees
✗ high communication
✔ low communication
average: w := 1K
P
k wk
“always communicate”
“never communicate”
reduce: w = w ↵
P
k w
37. Distributed Optimization
✔ convergence
guarantees
✗ high communication
✔ low communication
average: w := 1K
P
k wk
“always communicate”
“never communicate”
ZDWJ, 2012
reduce: w = w ↵
P
k w
38. Distributed Optimization
✔ convergence
guarantees
✗ high communication
✔ low communication
✗ convergence
not guaranteed
average: w := 1K
P
k wk
“always communicate”
“never communicate”
ZDWJ, 2012
reduce: w = w ↵
P
k w
55. Mini-batch Limitations
1. METHODS BEYOND SGD
Use Primal-Dual Framework
2. STALE UPDATES
Immediately apply local updates
3. AVERAGE OVER BATCH SIZE
Average over K batch size
56. Communication-Efficient
Distributed Dual
Coordinate Ascent (CoCoA)
1. METHODS BEYOND SGD
Use Primal-Dual Framework
2. STALE UPDATES
Immediately apply local updates
3. AVERAGE OVER BATCH SIZE
Average over K batch size
59. 1. Primal-Dual Framework
PRIMAL ≥ DUAL
min
w2Rd
P(w) :=
!
2 ||w||2 +
1
n
Xn
i=1
`i(wT xi)
#
max
↵2Rn
D(↵) := ||A↵||2
1
n
Xn
i=1
`⇤i
(↵i)
#
Ai =
1
n
xi
60. 1. Primal-Dual Framework
≥
PRIMAL DUAL
Stopping criteria given by duality gap
min
w2Rd
P(w) :=
!
2 ||w||2 +
1
n
Xn
i=1
`i(wT xi)
#
max
↵2Rn
D(↵) := ||A↵||2
1
n
Xn
i=1
`⇤i
(↵i)
#
Ai =
1
n
xi
61. 1. Primal-Dual Framework
≥
PRIMAL DUAL
Stopping criteria given by duality gap
Good performance in practice
min
w2Rd
P(w) :=
!
2 ||w||2 +
1
n
Xn
i=1
`i(wT xi)
#
max
↵2Rn
D(↵) := ||A↵||2
1
n
Xn
i=1
`⇤i
(↵i)
#
Ai =
1
n
xi
62. 1. Primal-Dual Framework
≥
PRIMAL DUAL
Stopping criteria given by duality gap
Good performance in practice
Default in software packages e.g. liblinear
min
w2Rd
P(w) :=
!
2 ||w||2 +
1
n
Xn
i=1
`i(wT xi)
#
max
↵2Rn
D(↵) := ||A↵||2
1
n
Xn
i=1
`⇤i
(↵i)
#
Ai =
1
n
xi
68. CoCoA
ni
Algorithm 1: CoCoA
Input: T ! 1, scaling parameter 1 K K (default: K := 1).
Data: {(xi, yi)}=1 distributed over K machines
Initialize: ↵(0)
[k] 0 for all machines k, and w(0) 0
for t = 1, 2, . . . ,T
for all machines k = 1, 2, . . . ,K in parallel
(↵[k],wk) LocalDualMethod(↵(t1)
[k] ,w(t1))
[k] ↵(t1)
[k] + K
↵(t)
K ↵[k]
end
reduce w(t) w(t1) + K
K
PK
k=1 wk
end
Procedure A: LocalDualMethod: Dual algorithm on machine k
Input: Local ↵[k] 2 Rnk , and w 2 Rd consistent with other coordinate
blocks of ↵ s.t. w = A↵
Data: Local {(xi, yi)}nk
i=1
Output: ↵[k] and w := A[k]↵[k]
69. CoCoA
Algorithm 1: CoCoA
Input: T ! 1, scaling parameter 1 K K (default: K := 1).
Data: {(xi, yi)}ni
✔ 10 lines of code in Spark
=1 distributed over K machines
Initialize: ↵(0)
[k] 0 for all machines k, and w(0) 0
for t = 1, 2, . . . ,T
for all machines k = 1, 2, . . . ,K in parallel
(↵[k],wk) LocalDualMethod(↵(t1)
[k] ,w(t1))
[k] ↵(t1)
[k] + K
↵(t)
K ↵[k]
end
reduce w(t) w(t1) + K
K
PK
k=1 wk
end
Procedure A: LocalDualMethod: Dual algorithm on machine k
Input: Local ↵[k] 2 Rnk , and w 2 Rd consistent with other coordinate
blocks of ↵ s.t. w = A↵
Data: Local {(xi, yi)}nk
i=1
Output: ↵[k] and w := A[k]↵[k]
70. CoCoA
Algorithm 1: CoCoA
Input: T ! 1, scaling parameter 1 K K (default: K := 1).
Data: {(xi, yi)}ni
✔ 10 lines of code in Spark
=1 distributed over K machines
Initialize: ↵(0)
[k] 0 for all machines k, and w(0) 0
✔ primal-dual framework allows for
any internal optimization method
for t = 1, 2, . . . ,T
for all machines k = 1, 2, . . . ,K in parallel
(↵[k],wk) LocalDualMethod(↵(t1)
[k] ,w(t1))
[k] ↵(t1)
[k] + K
↵(t)
K ↵[k]
end
reduce w(t) w(t1) + K
K
PK
k=1 wk
end
Procedure A: LocalDualMethod: Dual algorithm on machine k
Input: Local ↵[k] 2 Rnk , and w 2 Rd consistent with other coordinate
blocks of ↵ s.t. w = A↵
Data: Local {(xi, yi)}nk
i=1
Output: ↵[k] and w := A[k]↵[k]
71. CoCoA
Algorithm 1: CoCoA
Input: T ! 1, scaling parameter 1 K K (default: K := 1).
Data: {(xi, yi)}ni
✔ 10 lines of code in Spark
=1 distributed over K machines
Initialize: ↵(0)
[k] 0 for all machines k, and w(0) 0
✔ primal-dual framework allows for
any internal optimization method
for t = 1, 2, . . . ,T
for all machines k = 1, 2, . . . ,K in parallel
(↵[k],wk) LocalDualMethod(↵(t1)
[k] ,w(t1))
[k] ↵(t1)
[k] + K
↵(t)
K ↵[k]
end
reduce w(t) w(t1) + K
✔ local updates K
applied immediately
PK
k=1 wk
end
Procedure A: LocalDualMethod: Dual algorithm on machine k
Input: Local ↵[k] 2 Rnk , and w 2 Rd consistent with other coordinate
blocks of ↵ s.t. w = A↵
Data: Local {(xi, yi)}nk
i=1
Output: ↵[k] and w := A[k]↵[k]
72. CoCoA
Algorithm 1: CoCoA
Input: T ! 1, scaling parameter 1 K K (default: K := 1).
Data: {(xi, yi)}ni
✔ 10 lines of code in Spark
=1 distributed over K machines
Initialize: ↵(0)
[k] 0 for all machines k, and w(0) 0
✔ primal-dual framework allows for
any internal optimization method
for t = 1, 2, . . . ,T
for all machines k = 1, 2, . . . ,K in parallel
(↵[k],wk) LocalDualMethod(↵(t1)
[k] ,w(t1))
[k] ↵(t1)
[k] + K
↵(t)
K ↵[k]
end
reduce w(t) w(t1) + K
✔ local updates K
applied immediately
PK
k=1 wk
end
Procedure A: LocalDualMethod: Dual algorithm on machine k
Input: Local ↵[✔ k] 2 Rnk average , and w 2 Rd consistent over with K
other coordinate
blocks of ↵ s.t. w = A↵
Data: Local {(xi, yi)}nk
i=1
Output: ↵[k] and w := A[k]↵[k]
73. Convergence
E[D(↵⇤) − D(↵(T))]
✓
1 − (1 − ⇥)
1
K
!n
# + !n
◆T ⇣
D(↵⇤) − D(↵(0))
⌘
Assumptions:
are -smooth
LocalDualMethod makes improvement ⇥
per step
e.g. for SDCA
⇥ =
✓
1
!n
1 + !n
1
˜n
◆H
`i 1/
applies also to duality gap
0 n/!
K
measure of difficulty of data partition
74. Convergence
✔ it converges!
E[D(↵⇤) − D(↵(T))]
✓
1 − (1 − ⇥)
1
K
!n
# + !n
◆T ⇣
D(↵⇤) − D(↵(0))
⌘
Assumptions:
are -smooth
LocalDualMethod makes improvement ⇥
per step
e.g. for SDCA
⇥ =
✓
1
!n
1 + !n
1
˜n
◆H
`i 1/
applies also to duality gap
0 n/!
K
measure of difficulty of data partition
75. Convergence
✔ it converges!
✔ convergence rate matches
current state-of-the-art mini-batch
E[D(↵⇤) − D(↵(T))]
✓
1 − (1 − ⇥)
1
K
!n
# + !n
◆T ⇣
D(↵⇤) − D(↵(0))
⌘
Assumptions:
are -smooth
LocalDualMethod makes improvement ⇥
per step
e.g. for SDCA
⇥ =
✓
1
!n
1 + !n
1
˜n
◆H
`i 1/
methods
applies also to duality gap
0 n/!
K
measure of difficulty of data partition
76. Convergence
✔ it converges!
✔ convergence rate matches
current state-of-the-art mini-batch
E[D(↵⇤) − D(↵(T))]
✓
1 − (1 − ⇥)
1
K
!n
# + !n
◆T ⇣
D(↵⇤) − D(↵(0))
⌘
Assumptions:
are -smooth
LocalDualMethod makes improvement ⇥
per step
e.g. for SDCA
⇥ =
✓
1
!n
1 + !n
1
˜n
◆H
`i 1/
methods
applies also to duality gap
0 n/!
K
measure of difficulty of data partition
