This document summarizes a presentation given by Michael McCourt of SigOpt at the 2nd Annual Workshop on Offline and Online Evaluation of Interactive Systems at KDD 2019. The presentation discusses using Bayesian optimization to efficiently explore the tradeoffs between competing offline metrics. It proposes posing the problem as a constrained multi-objective optimization to explore the Pareto efficient frontier. It describes allowing users to interactively update the constraints as the search progresses to account for changes in their goals. Various strategies for exploring the efficient frontier are discussed, including randomly or systematically varying the constraints. Applications of Bayesian optimization are highlighted, including for materials and model design. Future work directions are proposed, such as better handling black-box constraints.
Agile analytics : An exploratory study of technical complexity managementAgnirudra Sikdar
The thesis involved the reviewing of various case studies to determine the types of modelling, choice of algorithm, types of analytical approaches and trying to determine the various complexities arising from these cases. From these reviews, procedures have been proposed to improve the efficiency and manage the various types of complexities from using agile methodological perspective. Focus was mostly done on Customer Segmentation and Clustering , with the sole purpose to bridge Big Data and Business Intelligence together using Analytic.
Advanced Optimization for the Enterprise WebinarSigOpt
Building on the TWIML eBook, TWIMLcon event and TWIML podcast series that explore Machine Learning Platforms in great detail, this webinar examines the machine learning platforms that power enterprise leaders in AI. SigOpt CEO Scott Clark will provide an overview of critical technical capabilities that our customers have prioritized in their ML platforms.
Review these slides to learn about:
- Critical capabilities for data, experiment and model management
- Tradeoffs between building and buying these capabilities
- Lessons from the implementation of these platforms by AI leaders
Why focus on these platforms and the capabilities that power them? Nearly every company is investing in machine learning that differentiates products or generates revenue. These so-called "differentiated models" represent the biggest opportunity for AI to transform the business. Most of these teams find success hiring expert data scientists and machine learning engineers who can build these models. But most of these teams also struggle to create a more sustainable, scalable and reproducible process for model development, and have begun building ML platforms to tackle this challenge.
Slides Mike Claiborne recently used in his discussion w/ mentees of The Product Mentor.
The Product Mentor is a program designed to pair Product Mentors and Mentees from around the World, across all industries, from start-up to enterprise, guided by the fundamental goals…Better Decisions. Better Products. Better Product People.
Throughout the program, each mentor leads a conversation in an area of their expertise that is live streamed and available to both mentee and the broader product community.
http://TheProductMentor.com
Agile analytics : An exploratory study of technical complexity managementAgnirudra Sikdar
The thesis involved the reviewing of various case studies to determine the types of modelling, choice of algorithm, types of analytical approaches and trying to determine the various complexities arising from these cases. From these reviews, procedures have been proposed to improve the efficiency and manage the various types of complexities from using agile methodological perspective. Focus was mostly done on Customer Segmentation and Clustering , with the sole purpose to bridge Big Data and Business Intelligence together using Analytic.
Advanced Optimization for the Enterprise WebinarSigOpt
Building on the TWIML eBook, TWIMLcon event and TWIML podcast series that explore Machine Learning Platforms in great detail, this webinar examines the machine learning platforms that power enterprise leaders in AI. SigOpt CEO Scott Clark will provide an overview of critical technical capabilities that our customers have prioritized in their ML platforms.
Review these slides to learn about:
- Critical capabilities for data, experiment and model management
- Tradeoffs between building and buying these capabilities
- Lessons from the implementation of these platforms by AI leaders
Why focus on these platforms and the capabilities that power them? Nearly every company is investing in machine learning that differentiates products or generates revenue. These so-called "differentiated models" represent the biggest opportunity for AI to transform the business. Most of these teams find success hiring expert data scientists and machine learning engineers who can build these models. But most of these teams also struggle to create a more sustainable, scalable and reproducible process for model development, and have begun building ML platforms to tackle this challenge.
Slides Mike Claiborne recently used in his discussion w/ mentees of The Product Mentor.
The Product Mentor is a program designed to pair Product Mentors and Mentees from around the World, across all industries, from start-up to enterprise, guided by the fundamental goals…Better Decisions. Better Products. Better Product People.
Throughout the program, each mentor leads a conversation in an area of their expertise that is live streamed and available to both mentee and the broader product community.
http://TheProductMentor.com
BA is used to gain insights that inform business decisions and can be used to automate and optimize business processes. Data-driven companies treat their data as a corporate asset and leverage it for a competitive advantage. Successful business analytics depends on data quality, skilled analysts who understand the technologies and the business, and an organizational commitment to data-driven decision-making.
Business analytics examples
Business analytics techniques break down into two main areas. The first is basic business intelligence. This involves examining historical data to get a sense of how a business department, team or staff member performed over a particular time. This is a mature practice that most enterprises are fairly accomplished at using.
Better Living Through Analytics - Strategies for Data DecisionsProduct School
Data is king! Get ready to understand how a successful analytics team can empower managers from product, marketing, and other areas to make effective, data-driven decisions.
Louis Cialdella, a data scientist at ZipRecruiter, shared some case studies and successful strategies that he has used at ZipRecruiter as well as previous experiences. The purpose of this data talk was to enlighten people on how to make sure that analysts can successfully partner with other departments and get them the information they need to do great things.
Slides for a workshop on agile technical leadership.
Agile teams are complex adaptive systems. In order to obtain a certain level of consistency, required when you want effective teams, teams have to set constraints on themselves and to make strategic decisions. This workshop explores some of the constraints and the difficulties of making strategic technical decisions.
The main target of this project is that it enables the telemarketing team to prioritize targeting for term loan marketing program by adopting a data-driven approach in Machine learning.
Supply Chain Strategy with an Assessment done for a Consumer Products Company with a presence in Europe, North America and Asia. Listing this to show format details for Supply Chain Strategy, and will also put a shorter version on as a Case Study for CI in Supply Chain. We put this on in so much detail to provide people with a guide and base for others to do similar projects (and if they need help, we'd love to speak with them). Good luck and feel free to reach out to me if you have questions on some of the formats or how to do the analysis being suggested.
The complimentary deck to my public speech on "People vs Process" in Minsk at Jan-2020. It's contains the intermediate results of research I'm working on.
Inside is description of the approach to non-invasive measurements of software developers productivity.
For more detailed info please contact alexander_nemtsov at gmail.com
Pareto-Optimal Search-Based Software Engineering (POSBSE): A Literature SurveyAbdel Salam Sayyad
Paper presented at the 2nd International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE’13), San Francisco, USA. May 2013.
This guide is developed to provide a structured approach for conducting a high quality competitive analysis.
It provides a detailed approach and methodology for competitor assessment in five key topic areas:
- business/portolio overview
- management and people
- product/service and technology
- cost structure and operations
- marketing and sales
The competitor and information analysis is divided into several steps:
- develop internal cost model and capability baseline
- conduct competitor data collection and synthesis
- evaluate size and nature of competitive differences
- assess potential competitor strategies and implications
Each steps contain detail description of activities, examples, and tools used.
The document also includes an interview guide for the user to jump-start the process.
gazhoo.com
Using BAPO to apply structure follows strategyJason Yip
BAPO stands for Business Architecture Process Organisation. It is Jan Bosch's more fleshed out expression of "structure should follow strategy". I recently experimented with applying this framework and would like to share what worked and what didn't. Concepts expanded beyond BAPO to include product capabilities versus architecture services; overlapping product lifecycle s-curves; Simon Wardley's Pioneers, Settlers, Town Planners; and a reframing of the teaching people how to fish metaphor.
An annotated slide deck from a webinar hosted by Stilo International and conducted on June 24, 2014.
The talk introduces tactics for moving a content solution project forward quickly while also attending to essential details.
Rapid Optimization Application Development Using Excel and SolverMichael Mina
Marketing optimization is the process of determining how to allocate marketing dollars in order to achieve specific goals (e.g., maximize profit), subject to certain constraints (e.g., a fixed marketing budget). This often takes the form of using mathematical techniques to determine who to target, through which channel, and with what message or offer.
A number of optimization applications are commercially available. However, many of them require changes to data and computational infrastructure that are labor-intensive and cost-prohibitive. This presentation will demonstrate how optimization applications can be developed easily and quickly using Excel combined with Excel Solver, even for large marketing campaigns.
This presentation will discuss how segmentation can be used to reduce the complexity of large optimization problems, and how to quickly develop a simple but effective optimization application using Excel combined with Excel Solver.
This presentation will be of interest to those seeking to optimize marketing campaigns of any size while managing operational and computational complexity.
An electronic copy of the Excel worksheet used for optimization is this presentation is available at tinyurl.com/mina2018artforum.
Optimizing BERT and Natural Language Models with SigOpt Experiment ManagementSigOpt
SigOpt Machine Learning Engineer Meghana Ravikumar explains how she reduced the size of a BERT natural language model trained on the SQUAD 2.0 question-answer database, to reduce its size while maintaining performance using a "distillation" process optimized with SigOpt's Experiment Management functionality.
SigOpt's Fay Kallel, Head of Product, and Jim Blomo, Head of Engineering, describe the latest updates to SigOpt, a suite of features that help you manage your modeling process.
More Related Content
Similar to Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimization
BA is used to gain insights that inform business decisions and can be used to automate and optimize business processes. Data-driven companies treat their data as a corporate asset and leverage it for a competitive advantage. Successful business analytics depends on data quality, skilled analysts who understand the technologies and the business, and an organizational commitment to data-driven decision-making.
Business analytics examples
Business analytics techniques break down into two main areas. The first is basic business intelligence. This involves examining historical data to get a sense of how a business department, team or staff member performed over a particular time. This is a mature practice that most enterprises are fairly accomplished at using.
Better Living Through Analytics - Strategies for Data DecisionsProduct School
Data is king! Get ready to understand how a successful analytics team can empower managers from product, marketing, and other areas to make effective, data-driven decisions.
Louis Cialdella, a data scientist at ZipRecruiter, shared some case studies and successful strategies that he has used at ZipRecruiter as well as previous experiences. The purpose of this data talk was to enlighten people on how to make sure that analysts can successfully partner with other departments and get them the information they need to do great things.
Slides for a workshop on agile technical leadership.
Agile teams are complex adaptive systems. In order to obtain a certain level of consistency, required when you want effective teams, teams have to set constraints on themselves and to make strategic decisions. This workshop explores some of the constraints and the difficulties of making strategic technical decisions.
The main target of this project is that it enables the telemarketing team to prioritize targeting for term loan marketing program by adopting a data-driven approach in Machine learning.
Supply Chain Strategy with an Assessment done for a Consumer Products Company with a presence in Europe, North America and Asia. Listing this to show format details for Supply Chain Strategy, and will also put a shorter version on as a Case Study for CI in Supply Chain. We put this on in so much detail to provide people with a guide and base for others to do similar projects (and if they need help, we'd love to speak with them). Good luck and feel free to reach out to me if you have questions on some of the formats or how to do the analysis being suggested.
The complimentary deck to my public speech on "People vs Process" in Minsk at Jan-2020. It's contains the intermediate results of research I'm working on.
Inside is description of the approach to non-invasive measurements of software developers productivity.
For more detailed info please contact alexander_nemtsov at gmail.com
Pareto-Optimal Search-Based Software Engineering (POSBSE): A Literature SurveyAbdel Salam Sayyad
Paper presented at the 2nd International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE’13), San Francisco, USA. May 2013.
This guide is developed to provide a structured approach for conducting a high quality competitive analysis.
It provides a detailed approach and methodology for competitor assessment in five key topic areas:
- business/portolio overview
- management and people
- product/service and technology
- cost structure and operations
- marketing and sales
The competitor and information analysis is divided into several steps:
- develop internal cost model and capability baseline
- conduct competitor data collection and synthesis
- evaluate size and nature of competitive differences
- assess potential competitor strategies and implications
Each steps contain detail description of activities, examples, and tools used.
The document also includes an interview guide for the user to jump-start the process.
gazhoo.com
Using BAPO to apply structure follows strategyJason Yip
BAPO stands for Business Architecture Process Organisation. It is Jan Bosch's more fleshed out expression of "structure should follow strategy". I recently experimented with applying this framework and would like to share what worked and what didn't. Concepts expanded beyond BAPO to include product capabilities versus architecture services; overlapping product lifecycle s-curves; Simon Wardley's Pioneers, Settlers, Town Planners; and a reframing of the teaching people how to fish metaphor.
An annotated slide deck from a webinar hosted by Stilo International and conducted on June 24, 2014.
The talk introduces tactics for moving a content solution project forward quickly while also attending to essential details.
Rapid Optimization Application Development Using Excel and SolverMichael Mina
Marketing optimization is the process of determining how to allocate marketing dollars in order to achieve specific goals (e.g., maximize profit), subject to certain constraints (e.g., a fixed marketing budget). This often takes the form of using mathematical techniques to determine who to target, through which channel, and with what message or offer.
A number of optimization applications are commercially available. However, many of them require changes to data and computational infrastructure that are labor-intensive and cost-prohibitive. This presentation will demonstrate how optimization applications can be developed easily and quickly using Excel combined with Excel Solver, even for large marketing campaigns.
This presentation will discuss how segmentation can be used to reduce the complexity of large optimization problems, and how to quickly develop a simple but effective optimization application using Excel combined with Excel Solver.
This presentation will be of interest to those seeking to optimize marketing campaigns of any size while managing operational and computational complexity.
An electronic copy of the Excel worksheet used for optimization is this presentation is available at tinyurl.com/mina2018artforum.
Similar to Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimization (20)
Optimizing BERT and Natural Language Models with SigOpt Experiment ManagementSigOpt
SigOpt Machine Learning Engineer Meghana Ravikumar explains how she reduced the size of a BERT natural language model trained on the SQUAD 2.0 question-answer database, to reduce its size while maintaining performance using a "distillation" process optimized with SigOpt's Experiment Management functionality.
SigOpt's Fay Kallel, Head of Product, and Jim Blomo, Head of Engineering, describe the latest updates to SigOpt, a suite of features that help you manage your modeling process.
Efficient NLP by Distilling BERT and Multimetric OptimizationSigOpt
SigOpt ML Engineer Meghana Ravikumar explains how to use multimetric optimization to achieve a more efficient, compact BERT model to perform on a question-answering task.
SigOpt Research Engineer Michael McCourt and DarwinAI CTO Alexander Wong explain how they used SigOpt and hyperparameter optimization to successfully improve accuracy of detecting COVID-19 cases from chest X-Rays, using the COVID-Net model and the COVIDx open dataset.
Metric Management: a SigOpt Applied Use CaseSigOpt
These slides correspond to a recording of a live webcast of a demo of Metric Management functionality in SigOpt, keeping model size down while increasing validation accuracy for a road sign image classification problem.
Tuning for Systematic Trading: Talk 3: Training, Tuning, and Metric StrategySigOpt
This talk explains how you can train and tune efficiently using metric strategy to assign, store, and optimize a variety of metrics, even changing them over time. Tobias Andreassen, who supports a number of our systematic trading customers, explained how he helps customers tune more efficiently with these SigOpt features in real-world scenarios.
Tuning for Systematic Trading: Talk 2: Deep LearningSigOpt
This talk explains how to train deep learning and other expensive models with parallelism and multitask optimization to reduce wall clock time. Tobias Andreassen, who supports a number of our systematic trading customers, presented the intuition behind Bayesian optimization for model optimization with a single or multiple (often competing) metrics. Many times it makes sense to analyze a second metric to avoid myopic training runs that overfit on your data, or otherwise don’t represent or impede performance in real-world scenarios.
This talk discusses the intuition behind Bayesian optimization with and without multiple metrics. Tobias Andreassen, who supports a number of our systematic trading customers, presented the intuition behind Bayesian optimization for model optimization with a single or multiple (often competing) metrics. Many times it makes sense to analyze a second metric to avoid myopic training runs that overfit on your data, or otherwise don’t represent or impede performance in real-world scenarios.
Tuning Data Augmentation to Boost Model PerformanceSigOpt
In this webinar, SigOpt ML Engineer Meghana Ravikumar presents on and builds an image classifier trained on the Stanford Cars dataset to evaluate two approaches to transfer learning—fine tuning and feature extraction—and the impact of Multitask optimization, a more efficient form of Bayesian optimization, on these techniques. Once we define the most performant transfer learning technique for Stanford Cars, we will use image augmentation to double the size of the dataset to boost the classifier’s performance. Instead of manually tuning the hyperparameters associated with image augmentation, we will use Multitask Optimization to learn these hyperparameters using the downstream image classifier’s performance as the guide. In conjunction with model performance, we will also explore the features of these augmented images and the downstream implications for our image classifier.
SigOpt founder and CEO, Scott Clark, PhD, explains the tradeoffs you'll want to consider when designing your modeling platform and integrating hyperparameter optimization to enhance data scientist productivity.
This webinar, hosted by SigOpt co-founder and CEO Scott Clark, explains how advanced features can help you achieve your modeling goals. These features include metric definition and multimetric optimization, conditional parameters, and multitask optimization for long training cycles.
SigOpt helps your algorithmic traders and data scientists build better models faster. Learn how to integrate SigOpt into your modeling platform for quick ROI for your data science team.
As data science workloads grow, so does their need for infrastructure. But, is it fair to ask data scientists to also become infrastructure experts? If not the data scientists, then, who is responsible for spinning up and managing data science infrastructure? This talk will address the context in which ML infrastructure is emerging, walk through two examples of ML infrastructure tools for launching hyperparameter optimization jobs, and end with some thoughts for building better tools in the future.
Originally given as a talk at the PyData Ann Arbor meetup (https://www.meetup.com/PyData-Ann-Arbor/events/260380989/)
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...SigOpt
At the inaugural Uber science symposium, SigOpt research engineer Bolong (Harvey) Cheng shares insights on black-box optimization from his experience working with both leading academics and innovative enterprises.
SigOpt at O'Reilly - Best Practices for Scaling Modeling PlatformsSigOpt
Companies are increasingly building modeling platforms to empower their researchers to efficiently scale the development and productionalization of their models. Scott Clark and Matt Greenwood share a case study from a leading algorithmic trading firm to illustrate best practices for building these types of platforms in any industry. Join in to learn how Two Sigma, a leading quantitative investment and technology firm, solved its model optimization problem.
Training and tuning models with lengthy training cycles like those in deep learning can be extremely expensive and may sometimes involve techniques that degrade performance. We'll explore recent research on optimization strategies to efficiently tune these types of deep learning models. We will provide benchmarks and comparisons to other popular methods for optimizing the models, and we'll recommend valuable areas for further applied research.
SigOpt at GTC - Reducing operational barriers to optimizationSigOpt
Advanced hardware like NVIDIA technology lowers technical barriers to model size and scope, but issues remain in areas like model performance and training infrastructure management. We'll discuss operational challenges to training models at scale with a particular focus on how training management and hyperparameter tuning can inform each other to accomplish specific goals. We'll also explore techniques like parallelism and scheduling, discuss their impact on model optimization, and compare various techniques. We'll also evaluate results of this approach. In particular, we'll focus on how new tools that automate training orchestration accelerate model development and increase the volume and quality of models in production.
SigOpt CEO Scott Clark provides insights for modeling at scale in systematic trading. SigOpt works with algorithmic trading firms that collectively represent $300 billion in assets under management (AUM). In this presentation, Scott draws on this experience to provide a few critical insights to how these companies effectively model at scale. Alongside these insights, Scott shares a more specific case study from working with Two Sigma, a leading systematic investment manager.
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt
In this talk at MLconf NYC, Alexandra Johnson, platform engineering lead at SigOpt, discusses common operational challenges with scaling model training and how solutions are designed to
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Modern design is crucial in today's digital environment, and this is especially true for SharePoint intranets. The design of these digital hubs is critical to user engagement and productivity enhancement. They are the cornerstone of internal collaboration and interaction within enterprises.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Prosigns: Transforming Business with Tailored Technology Solutions
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimization
1. SigOpt. Confidential.
Interactive Tradeoffs Between
Competing Offline Metrics with
Bayesian Optimization
KDD 2019
2nd Annual Workshop
Online and Offline Evaluation of Interactive Systems
Michael McCourt, Research Engineer, SigOpt
2. SigOpt. Confidential.
About me
● Research engineering at SigOpt
● Focus on applied Bayesian optimization
● PhD from Cornell
● Avid Cleveland Cavaliers fan
About SigOpt
● Leading software solution for parameter
optimization and model experimentation
● Customers in finance, trading, media,
technology, consulting, energy, industry
● Free version of our solution for academia
available at sigopt.com/edu
3. SigOpt. Confidential.
Abstract for KDD 2019
2nd Annual Workshop on Offline and Online Evaluation of Interactive Systems
Many real world applications (ML models, simulators, etc.) have multiple competing
metrics that define performance; these require practitioners to carefully consider
potential tradeoffs. However, assessing and ranking this tradeoff is nontrivial,
especially when the number of metrics is more than two. Often times, practitioners
scalarize the metrics into a single objective, e.g., using a weighted sum.
In this talk, we pose this problem as a constrained multi-objective optimization
problem. By setting and updating the constraints, we can efficiently explore only the
region of the Pareto efficient frontier of the model/system of most interest. We
motivate this problem with the application of an experimental design setting, where
we are trying to fabricate high performance glass substrate for solar cell panels.
3
4. SigOpt. Confidential.
Most Metrics are Impacted by Free Parameters
How can these free parameters be chosen?
Generally, these are chosen to yield good future performance.
• This discussion only covers offline metrics.
• Some of the elements apply in an online setting as well.
Given a computable metric defining future performance, a search can be conducted for the free parameters
yielding acceptable/optimal performance.
• In many circumstances, evaluating this performance metric is costly.
• Example: Train a classification model and evaluate a validation accuracy.
• Example: Use financial data from the past year for a trading strategy and evaluate its profit on last
month’s data.
4
5. SigOpt. Confidential.
Searching for Free Parameters Requires Efficiency
Intelligently searching a fixed domain
Many searches benefit from efficiently (actively) learning about the circumstances of the search.
• Active learning -- “Active learning is closely related to experimental design … is most often adaptive …
employs an oracle for data labelling … is usually used to learn a model for classification.” -- [Brochu et al
2010]
Two adjacent fields of research have evolved.
• Bayesian optimization -- “Bayesian optimization is a sequential model-based approach to [optimizing a
function].” -- [Shahriari et al, 2016]
• Active search -- “Active search is an active learning setting with the goal of identifying as many
members of a given class as possible under a labeling budget.” -- [Jiang et al, 2017]
How we conduct this active learning will greatly impact efficiency of the search.
5
6. SigOpt. Confidential.
Bayesian Optimization
A graphical depiction of the iterative process
6
Build a statistical model Build a statistical model
Choose a next point Choose a next point
7. SigOpt. Confidential.
Bayesian Optimization
Efficiently Optimize a Scalar Function
To quote [Frazier 2018]: Bayesian optimization (BayesOpt/BO) is a class of machine-learning-based
optimization methods focused on [maximizing/minimizing a function with] the following properties ...
• Typically the dimension d is less than 20.
• The objective function f is continuous, as is the domain (which is likely a d-dimensional rectangle).
• f is expensive to evaluate: e.g., time, money, access
• f is black-box: if lacks known special structure like concavity or linearity.
• When we evaluate f, we observe only f(x); that is, the optimization is gradient-free.
• f is often observed in the presence of noise.
• Our focus is on finding a global rather than local optimum.
7
8. SigOpt. Confidential.
Many Metrics may Contribute to Success
How do we execute under these ambiguous circumstances?
Defining/measuring future performance is imprecise.
• Generally, several metrics will contribute to a sense of future performance.
• Not all metrics are equally important -- some may only need to reach a threshold.
• The feasible performance and preferred interaction between metrics may not be known a priori.
Popular multiobjective optimization strategies are often population-based (and not often sample-efficient).
8
9. SigOpt. Confidential.
Multiobjective Bayesian Optimization
Balancing competing metrics to find the Pareto frontier
Adaptations to BO to search for the efficient frontier:
• Change the problem to an active search problem [Jiang et al, 2018].
• Search for diverse points near the efficient frontier.
• Scalarize the problem with linear combinations of the metrics [Knowles, 2006].
• Define a hypervolume based acquisition function [Hernandez-Lobato et al, 2016, Emmerich et al, 2016].
• Scalarization through prior beliefs [Astudillo, 2017].
Guiding points
• Users wanted to interactively update the search process.
• Users felt uncomfortable stating a priori preferences.
9
10. SigOpt. Confidential.
Multiobjective Bayesian Optimization
Our strategy
We apply a strategy similar to what was discussed in [Letham et al, 2019].
1. Model all metrics independently.
• Requires no prior beliefs on how metrics interact.
• Missing data removed on a per metric basis if unrecorded.
2. Expose the efficient frontier through constrained scalar optimization.
• Enforce user constraints when given.
• Iterate through sub constraints to better resolve efficient frontier, if desired.
• Consider different regions of the frontier when parallelism is possible.
3. Allow users to change constraints as the search progresses.
• Allow the problems/goals to evolve as the user’s understanding evolves.
10
Variation on
Expected
Improvement
11. SigOpt. Confidential.
One strategy can be to randomly apply constraints.
Multiobjective Bayesian Optimization
Our strategy
11
Note: There are GIFs that do not
show up in this version of the
presentation. For a copy that
includes them, please email
contact@sigopt.com
12. SigOpt. Confidential.
Another strategy can be to “walk” up and down the constraint domain.
Multiobjective Bayesian Optimization
Our strategy
12
Note: There are GIFs that do not
show up in this version of the
presentation. For a copy that
includes them, please email
contact@sigopt.com
13. SigOpt. Confidential.
It can help to alternate which metric the constraint is imposed on.
Multiobjective Bayesian Optimization
Our strategy
13
Note: There are GIFs that do not
show up in this version of the
presentation. For a copy that
includes them, please email
contact@sigopt.com
14. SigOpt. Confidential.
Users can enforce their own bounds to focus on the desired outcome.
Multiobjective Bayesian Optimization
Our strategy
14
Note: There are GIFs that do not
show up in this version of the
presentation. For a copy that
includes them, please email
contact@sigopt.com
15. SigOpt. Confidential.
Users can also update their own bounds as the experiment goes on.
Multiobjective Bayesian Optimization
Our strategy
15
Note: There are GIFs that do not
show up in this version of the
presentation. For a copy that
includes them, please email
contact@sigopt.com
16. SigOpt. Confidential.
Awesome Applications of Bayesian Optimization
Who is using, and can use, BO?
16
● ML/DL hyperparameter tuning [Snoek et al, 2012; Feurer et al, 2015; Kandasamy et al, 2018]
● Engineering system design [Mockus, 1989; Jones et al, 1998; Forrester et al, 2008]
● Drug design [Negoescu et al, 2011; Frazier and Wang, 2016]
● Material design [Packwood, 2017; Haghanifar et al, 2019]
● Model calibration [Shoemaker et al, 2007; Shi et al, 2013; Letham et al. 2019]
● Reinforcement learning [Lizotte, 2008; Brochu et al, 2010; Martinez-Cantin et al, 2018]
There are so many others!
17. SigOpt. Confidential.17
A Joint Collaboration with University of Pittsburgh
[Haghanifar et al, 2019]
Metrics
• Light transmission
• Clarity (low haze)
• Water resistance
Constraints updated on all
metrics during the search.
Note: There is a video that
does not show up in this
version of the presentation.
For a copy that includes this,
please email
contact@sigopt.com
18. SigOpt. Confidential.
Future Work
How can we improve this process?
When black-box constraints exist, how can we encourage our search to respect them?
• Hallucinate bad function values at points which violate the constraints.
• Attenuate the expected improvement by the probability of failure [Gelbart, 2015].
• Model the constraints and average out the noisy behavior [Letham et al, 2019].
• Model the Lagrangian [Picheny et al, 2016].
Question: Exactly how black-box/expensive are these constraints (or the objective)?
• We can adapt to expensive constraints but a cheap objective [Gramacy et al, 2106].
Question: Can we help focus on the important region using preferences?
• Joint work extending [Astudillo, 2017] with Raul and Peter.
18