Alexandra johnson reducing operational barriers to model training

Reducing Operational Barriers to Model Training

Technology

SigOpt. Confidential.
Alexandra Johnson
Software Engineer

SigOpt. Confidential.3
Operational Barriers
Machine learning experts specialize in:
• Gathering data
• Building models
• Extracting insights
Infrastructure engineers specialize in:
• Building shared tools
• Application scalability and performance
• Keeping track of interactions between large
distributed systems
The Challenge:
• Machine learning experts want to maximize
the performance of their models
• SigOpt provides an API for hyperparameter
optimization (HPO)
• SigOpt HPO helps ML experts maximize the
performance of their models!
• ML experts need to use clusters to properly
perform HPO

SigOpt. Confidential.4
Machine Learning Infrastructure
Model building
workflow problems
Infrastructure /
devops solutions+ = Machine learning
infrastructure

SigOpt. Confidential.5
Case Study: Building SigOpt Orchestrate
• Project started in 2018 to bridge ML and infrastructure
• What problems did our customers ask us to solve?
• How did a challenge for the user turn into a technical problem?
• Which tools / technologies did we use?

SigOpt. Confidential.6
Challenge #1: Can't Train Model on Laptop
Problem: Setup each remote machine
Initial Solution:
• Write a setup script to install dependencies
• SCP data, code, and setup script to every
remote machine

SigOpt. Confidential.7
Solution #1: Containerize!
Problem: Setup each remote machine
New Solution:
• Containerize code and dependencies
on the user's local environment
• Push the image to a registry
• Each machine pulls the image from a
registry
Registry

SigOpt. Confidential.8
Challenge #2: Start Training in Parallel
Problem: Kick off the hyperparameter
optimization job on six machines at once
Initial Solution:
• Open a tmux window on every
remote instance
• SSH over command to run setup
script into each tmux window
• SSH over command to train model
into each tmux window

SigOpt. Confidential.9
Solution #2: Kubernetes!
Problem: Kick off the hyperparameter
optimization job on six machines at once
New Solution:
• Spin up AWS EKS (Kubernetes) cluster
• Create a job spec
• "run 6 copies of this container at the
same time"
• Submit job spec to Kubernetes API
• Kubernetes starts the job on the cluster

SigOpt. Confidential.10
Challenge #3: View Progress and Debug
Problem: View the status of a hyperparameter
optimization job at a glance
Initial Solution:
• Save hostname and error information as
metadata in calls to external API
• SSH into machines and view the logs
directly (pre-Kubernetes)
• Use Kubernetes CLI to view logs

SigOpt. Confidential.11
Solution #3: Build a CLI!
Problem: View the status of a hyperparameter
optimization job at a glance
New Solution:
• Write an interface for the data scientist to
interact with the infrastructure tool
• We chose a command line interface
• Serves as an abstraction on top of
Kubernetes APIs + externals APIs
• Screenshots (top and bottom)
○ sigopt logs <experiment_id>
○ sigopt status <experiment_id>

SigOpt. Confidential.12
Final Thoughts...
• We're hiring!
• Connect with us
• Paper: Orchestrate: Infrastructure for Enabling Parallelism
during Hyperparameter Optimization, Alexandra Johnson
and Michael McCourt

SigOpt. Confidential.
Thank you!
Any questions?
Alexandra Johnson
alexandra@sigopt.com @alexandraj777

Open-source technologies allow developers to build microservices framework to build myriad real-time applications. One such application is building the real-time model scoring. In this session, we will showcase how to architect a microservice framework, in particular how to use it to build a low-latency, real-time model scoring system. At the core of the architecture lies Apache Spark’s Structured Streaming capability to deliver low-latency predictions coupled with Docker and Flask as additional open source tools for model service. In this session, you will walk away with: * Knowledge of enterprise-grade model as a service * Streaming architecture design principles enabling real-time machine learning * Key concepts and building blocks for real-time model scoring * Real-time and production use cases across industries, such as IIOT, predictive maintenance, fraud detection, sepsis etc.

Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...

This talk describes migrating a large random forest classifier from scikit-learn to Spark's MLlib. We cut training time from 2 days to 2 hours, reduced failed runs, and track experiments better with MLflow. Kount provides certainty in digital interactions like online credit card transactions. One of our scores uses a random forest classifier with 250 trees and 100,000 nodes per tree. We used scikit-learn to train using 60 million samples that each contained over 150 features. The in-memory requirements exceeded 750 GB, took 2 days, and were not robust to disruption in our database or training execution. To migrate workflow to Spark, we built a 6-node cluster with HDFS. This provides 1.35 TB of RAM and 484 cores. Using MLlib and parallelization, the training time for our random forests are now less than 2 hours. Training data stays in our production environment, which used to require a deploy cycle to move locally-developed code onto our training server. The new implementation uses Jupyter notebooks for remote development with server-side execution. MLflow tracks all input parameters, code, and git revision number, while the performance and model itself are retained as experiment artifacts. The new workflow is robust to service disruption. Our training pipeline begins by pulling from a Vertica database. Originally, this single connection took over 8 hours to complete with any problem causing a restart. Using sqoop and multiple connections, we pull the data in 45 minutes. The old technique used volatile storage and required the data for each experiment. Now, we pull the data from Vertica one time and then reload much faster from HDFS. While a significant undertaking, moving to the Spark ecosystem converted an ad hoc and hands-on training process into a fully repeatable pipeline that meets regulatory and business goals for traceability and speed. Speaker: Josh Johnston

Scaling up Deep Learning by Scaling Down

Best Practices for Engineering Production-Ready Software with Apache Spark

SparkML: Easy ML Productization for Real-Time Bidding

dataxu bids on ads in real-time on behalf of its customers at the rate of 3 million requests a second and trains on past bids to optimize for future bids. Our system trains thousands of advertiser-specific models and runs multi-terabyte datasets. In this presentation we will share the lessons learned from our transition towards a fully automated Spark-based machine learning system and how this has drastically reduced the time to get a research idea into production. We'll also share how we: - continually ship models to production - train models in an unattended fashion with auto-tuning capabilities - tune and overbooked cluster resources for maximum performance - ported our previous ML solution into Spark - evaluate the performance of high-rate bidding models Speakers: Maximo Gurmendez, Javier Buquet

Originally given at MLConf NYC 2017. All large machine learning pipelines have tunable parameters, commonly referred to as hyperparameters. Hyperparameter optimization is the process by which we find the values for these parameters that cause our system to perform the best. SigOpt provides a Bayesian optimization platform that is commonly used for hyperparameter optimization, and I’m going to share some of the common problems we’ve seen when integrating into machine learning pipelines.

Automate your Machine Learning

Ajit Ananthram

“Houston, we have a model...” Introduction to MLOps

Rui Quintino

Looking into the Future: Using Google's Prediction API

Justin Grammens

Bootstrapping of PySpark Models for Factorial A/B Tests

A/B testing, i.e., measuring the impact of proposed variants of e.g. e-commerce websites, is fundamental for increasing conversion rates and other key business metrics. We have developed a solution that makes it possible to run dozens of simultaneous A/B tests, obtain conclusive results sooner, and get more interpretable results than just statistical significance, but rather probabilities of the change having a positive effect, how much revenue is risked, etc. To compute those metrics, we need to estimate the posterior distributions of the metrics, which are computed using Generalized Linear Models (GLMs). Since we process gigabytes of data, we use a PySpark implementation, which however does not provide standard errors of coefficients. We, therefore, use bootstrapping to estimate the distributions. In this talk, I’ll describe how we’ve implemented parallelization of an already parallelized GLM computation to be able to scale this computation horizontally over a large cluster in Databricks and describe various tweaks and how they’ve improved the performance.

Production ready big ml workflows from zero to hero daniel marcous @ waze

Ido Shilon

How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform

In large enterprises, large solutions are sometimes required to tackle even the smallest tasks and ML is no different. At Comcast we are building a comprehensive, configuration based, continuously integrated and deployed platform for data pipeline transformations, model development and deployment. This is accomplished using a range of tools and frameworks such as Databricks, MLflow, Apache Spark and others. With a Databricks environment used by hundreds of researchers and petabytes of data, scale is critical to Comcast, so making it all work together in a frictionless experience is a high priority. The platform consists of a number of components: an abstraction for data pipelines and transformation to allow our data scientists the freedom to combine the most appropriate algorithms from different frameworks , experiment tracking, project and model packaging using MLflow and model serving via the Kubeflow environment on Kubernetes. The architecture, progress and current state of the platform will be discussed as well as the challenges we had to overcome to make this platform work at Comcast scale. As a machine learning practitioner, you will gain knowledge in: an example of data pipeline abstraction; ways to package and track your ML project and experiments at scale; and how Comcast uses Kubeflow on Kubernetes to bring everything together.

GDG PDX - An Intro to Google Cloud AutoML Vision

jerryhargrove

AWS IoT Edge Management

Using MLOps to Bring ML to Production/The Promise of MLOps

Weaveworks

In this final Weave Online User Group of 2019, David Aronchick asks: have you ever struggled with having different environments to build, train and serve ML models, and how to orchestrate between them? While DevOps and GitOps have made huge traction in recent years, many customers struggle to apply these practices to ML workloads. This talk will focus on the ways MLOps has helped to effectively infuse AI into production-grade applications through establishing practices around model reproducibility, validation, versioning/tracking, and safe/compliant deployment. We will also talk about the direction for MLOps as an industry, and how we can use it to move faster, with more stability, than ever before. The recording of this session is on our YouTube Channel here: https://youtu.be/twsxcwgB0ZQ Speaker: David Aronchick, Head of Open Source ML Strategy, Microsoft Bio: David leads Open Source Machine Learning Strategy at Azure. This means he spends most of his time helping humans to convince machines to be smarter. He is only moderately successful at this. Previously, David led product management for Kubernetes at Google, launched GKE, and co-founded the Kubeflow project. David has also worked at Microsoft, Amazon and Chef and co-founded three startups. Sign up for a free Machine Learning Ops Workshop: http://bit.ly/MLOps_Workshop_List Weaveworks will cover concepts such as GitOps (operations by pull request), Progressive Delivery (canary, A/B, blue-green), and how to apply those approaches to your machine learning operations to mitigate risk.

Geo Python16 keynote

Romeo Kienzler

The A-Z of Data: Introduction to MLOps

DataPhoenix

Команда Data Phoenix Events приглашает всех, 17 августа в 19:00, на первый вебинар из серии "The A-Z of Data", который будет посвящен MLOps. В рамках вводного вебинара, мы рассмотрим, что такое MLOps, основные принципы и практики, лучшие инструменты и возможные архитектуры. Мы начнем с простого жизненного цикла разработки ML решений и закончим сложным, максимально автоматизированным, циклом, который нам позволяет реализовать MLOps. https://dataphoenix.info/the-a-z-of-data/ https://dataphoenix.info/the-a-z-of-data-introduction-to-mlops/

Ml ops intro session

Kubeflow and Data Science in Kubernetes

John Liu

Deploying machine learning pipelines robustly at scale is one of the biggest challenges within an organization. Kubeflow is an open-source platform for distributed training, tuning, and serving models on Kubenetes. As a comprehensive solution for deploying and managing end-to-end data science and machine learning pipelines, Kubeflow is rapidly accelerating analytics innovation and adoption. John provides an overview of Kubeflow and how he has been using it in the wild.

Scott Clark, CEO, SigOpt, at The AI Conference 2017

Machine learning model to production

Georg Heiler

ML Platform Q1 Meetup: Airbnb's End-to-End Machine Learning Infrastructure

Fei Chen

Seamless MLOps with Seldon and MLflow

Deploying and managing machine learning models at scale introduces new complexities. Fortunately, there are tools that simplify this process. In this talk we walk you through an end-to-end hands on example showing how you can go from research to production without much complexity by leveraging the Seldon Core and MLflow frameworks. We will train a set of ML models, and we will showcase a simple way to deploy them to a Kubernetes cluster through sophisticated deployment methods, including canary deployments, shadow deployments and we’ll touch upon richer ML graphs such as explainer deployments.

Automated Hyperparameter Tuning, Scaling and Tracking

Automated Machine Learning (AutoML) has received significant interest recently. We believe that the right automation would bring significant value and dramatically shorten time-to-value for data science teams. Databricks is automating the Data Science and Machine Learning process through a combination of product offerings, partnerships, and custom solutions. This talk will focus on how Databricks can help automate hyperparameter tuning. For both traditional Machine Learning and modern Deep Learning, tuning hyperparameters can dramatically increase model performance and improve training times. However, tuning can be a complex and expensive process. In this talk, we'll start with a brief survey of the most popular techniques for hyperparameter tuning (e.g., grid search, random search, and Bayesian optimization). We will then discuss open source tools that implement each of these techniques, helping to automate the search over hyperparameters. Finally, we will discuss and demo improvements we built for these tools in Databricks, including integration with MLflow: Apache PySpark MLlib integration with MLflow for automatically tracking tuning Hyperopt integration with Apache Spark to distribute tuning and with MLflow for automatic tracking Recording and notebooks will be provided after the webinar so that you can practice at your own pace. Presenters Joseph Bradley, Software Engineer, Databricks Joseph Bradley is a Software Engineer and Apache Spark PMC member working on Machine Learning at Databricks. Previously, he was a postdoc at UC Berkeley after receiving his Ph.D. in Machine Learning from Carnegie Mellon in 2013. Yifan Cao, Senior Product Manager, Databricks Yifan Cao is a Senior Product Manager at Databricks. His product area spans ML/DL algorithms and Databricks Runtime for Machine Learning. Prior to Databricks, Yifan worked on two Machine Learning products, applying NLP to find metadata and applying machine learning to predict equipment failures. He helped build the products from ground up to multi-million dollars in ARR. Yifan started his career as a researcher in quantum computing. Yifan received his B.S in UC Berkeley and Master from MIT.

MLOps with serverless architectures (October 2018)

Julien SIMON

Machine Learning Infrastructure

Machine learning infrastructure solve data scientists' problems using infrastructure tools. This talk shows the case study of building SigOpt Orchestrate, an ML infrastructure tool. The talk highlights how data scientists' concerns as user mapped to solutions with some of today's most popular infrastructure tools. To learn more about SigOpt Orchestrate: https://sigopt.com/orchestrate Originally given as a talk for UC Berkeley's Women in Electrical Engineering and Computer Science group on January 24, 2019.

Machine Learning Infrastructure

As data science workloads grow, so does their need for infrastructure. But, is it fair to ask data scientists to also become infrastructure experts? If not the data scientists, then, who is responsible for spinning up and managing data science infrastructure? This talk will address the context in which ML infrastructure is emerging, walk through two examples of ML infrastructure tools for launching hyperparameter optimization jobs, and end with some thoughts for building better tools in the future. Originally given as a talk at the PyData Ann Arbor meetup (https://www.meetup.com/PyData-Ann-Arbor/events/260380989/)

What's hot

Common Problems in Hyperparameter Optimization

Automate your Machine Learning

Ajit Ananthram

“Houston, we have a model...” Introduction to MLOps

Rui Quintino

Looking into the Future: Using Google's Prediction API

Justin Grammens

Bootstrapping of PySpark Models for Factorial A/B Tests

Production ready big ml workflows from zero to hero daniel marcous @ waze

Ido Shilon

How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform

GDG PDX - An Intro to Google Cloud AutoML Vision

jerryhargrove

AWS IoT Edge Management

Using MLOps to Bring ML to Production/The Promise of MLOps

Weaveworks

Geo Python16 keynote

Romeo Kienzler

The A-Z of Data: Introduction to MLOps

DataPhoenix

Ml ops intro session

Kubeflow and Data Science in Kubernetes

John Liu

Scott Clark, CEO, SigOpt, at The AI Conference 2017

Machine learning model to production

Georg Heiler

ML Platform Q1 Meetup: Airbnb's End-to-End Machine Learning Infrastructure

Fei Chen

Seamless MLOps with Seldon and MLflow

Automated Hyperparameter Tuning, Scaling and Tracking

MLOps with serverless architectures (October 2018)

Julien SIMON

What's hot (20)

Common Problems in Hyperparameter Optimization

Automate your Machine Learning

“Houston, we have a model...” Introduction to MLOps

Looking into the Future: Using Google's Prediction API

Bootstrapping of PySpark Models for Factorial A/B Tests

Production ready big ml workflows from zero to hero daniel marcous @ waze

How to Utilize MLflow and Kubernetes to Build an Enterprise ML Platform

GDG PDX - An Intro to Google Cloud AutoML Vision

AWS IoT Edge Management

Using MLOps to Bring ML to Production/The Promise of MLOps

Geo Python16 keynote

The A-Z of Data: Introduction to MLOps

Ml ops intro session

Kubeflow and Data Science in Kubernetes

Scott Clark, CEO, SigOpt, at The AI Conference 2017

Machine learning model to production

ML Platform Q1 Meetup: Airbnb's End-to-End Machine Learning Infrastructure

Seamless MLOps with Seldon and MLflow

Automated Hyperparameter Tuning, Scaling and Tracking

MLOps with serverless architectures (October 2018)

Similar to Alexandra johnson reducing operational barriers to model training

Machine Learning Infrastructure

Machine Learning Infrastructure

Effective .NET Core Unit Testing with SQLite and Dapper

Mike Melusky

Webinar: Começando seus trabalhos com Machine Learning utilizando ferramentas...

Embarcados

Nesse webinar será apresentado o passo a passo de como criar projetos com Machine Learning utilizando ferramentas de terceiros como Sensi ML e Edge Impulse. Tópicos que serão apresentados: Kits de desenvolvimento para Machine Learning: EV18H79A: SAMD21 ML Evaluation Kit with TDK 6-axis MEMS EV45Y33A: SAMD21 ML Evaluation Kit with BOSCH IMU SAMC21 xPlained Pro evaluation kit (ATSAMC21-XPRO) plus its QT8 xPlained Pro Extension Kit (AC164161) Ferramentas de desenvolvimento: MPLAB X Data Visualizer Ambiente de terceiros: Sensi ML e Edge Impulse Coleta de dados Como desenvolver um projeto usando Machine Learning sem conhecimentos específicos sobre o assunto e com conhecimentos sobre Machine Learning.

Effective .NET Core Unit Testing with SQLite and Dapper

Mike Melusky

Developing Digital Twins

Elizabeth Steiner

This webinar is going to cover what is a digital twin and how all stakeholders can benefit from their functionality. You will learn how model-based systems engineering enables digital engineering. Your host will discuss use cases, a realistic look at digital engineering and digital twins, and how you can use Innoslate to get started. The Agenda Here's what we're covering. What is a Digital Twin Benefits of Digital Twin The Digital Engineering Path Enabled by MBSE AR + MBSE Software A More Realistic Digital Twin Getting You Started with Digital Twins Question Answer Session

Pitfalls of machine learning in production

Antoine Sauray

Py data scikit-productionTuri, Inc.

Five Ways to Fix Your SQL Server Dev-Test Problems

Catalogic Software

Rapidly provisioning fresh copies of SQL databases is required for an effective Dev-Test environment. However, it can be challenging and most organizations take weeks to deliver usable data. Catalogic’s copy data management platform allows for timely, space-efficient, masked SQL DB copies. By doing so, Catalogic satisfies both infrastructure DBAs, who need automated copy provisioning, and application DBAs, who continually need fresh, secure data sets. This webinar will describe five ways Catalogic can help fix SQL Server Dev-Test problems.

Lessons Learned Replatforming A Large Machine Learning Application To Apache ...

Morningstar’s Risk Model project is created by stitching together statistical and machine learning models to produce risk and performance metrics for millions of financial securities. Previously, we were running a single version of this application, but needed to expand it to allow for customizations based on client demand. With the goal of running hundreds of custom Risk Model runs at once at an output size of around 1TB of data each, we had a challenging technical problem on our hands! In this presentation, we’ll talk about the challenges we faced replatforming this application to Spark, how we solved them, and the benefits we saw. Some things we’ll touch on include how we created customized models, the architecture of our machine learning application, how we maintain an audit trail of data transformations (for rigorous third party audits), and how we validate the input data our model takes in and output data our model produces. We want the attendees to walk away with some key ideas of what worked for us when productizing a large scale machine learning platform.

JS Fest 2018. Никита Галкин. Микросервисная архитектура с переиспользуемыми к...

JSFestUA

Нарушение DRY принципа особенно часто возникает в микросервисах. Чтобы избежать этой проблемы, вы можете использовать повторно используемые компоненты, например, приватные пакеты npm. Лучшие практики, которые помогут вам достичь этого включают в себя паттерн ECB для организации кода, манифест 12-ти факторного приложения, использование генерации кода. В нашем проекте мы используем технический стек на основе Node.js, Docker, RabbitMQ, но идеи из этого доклада могут быть использованы для любого технического стека микросервисов

Why Upgrade to v8.6?

BillCavaUs

Ship code like a keptn

Rob Jahn

Consolidating MLOps at One of Europe’s Biggest Airports

At Schiphol airport we run a lot of mission critical machine learning models in production, ranging from models that predict passenger flow to computer vision models that analyze what is happening around the aircraft. Especially now in times of Covid it is paramount for us to be able to quickly iterate on these models by implementing new features, retraining them to match the new dynamics and above all to monitor them actively to see if they still fit the current state of affairs. To achieve those needs we rely on MLFlow but have also integrated that with many of our other systems. So have we written Airflow operators for MLFlow to ease the retraining of our models, have we integrated MLFlow deeply with our CI pipelines and have we integrated it with our model monitoring tooling. In this talk we will take you through the way we rely on MLFlow and how that enables us to release (sometimes) multiple versions of a model per week in a controlled fashion. With this set-up we are achieving the same benefits and speed as you have with a traditional software CI pipeline.

Serverless machine learning architectures at Helixa

Data Science Milan

Using Puppet, Ansible, and MongoDB Ops Manager Together to Create Your Own On...

MongoDB

MongoDB World 2018: Using Puppet, Ansible and Ops Manager to Create Your Own ...

MongoDB

MLOps and Reproducible ML on AWS with Kubeflow and SageMaker

Provectus

Looking to implement MLOps using AWS services and Kubeflow? Come and learn about machine learning from the experts of Provectus and Amazon Web Services (AWS)! Businesses recognize that machine learning projects are important but go beyond just building and deploying models, which is mostly done by organizations. Successful ML projects entail a complete lifecycle involving ML, DevOps, and data engineering and are built on top of ML infrastructure. AWS and Amazon SageMaker provide a foundation for building infrastructure for machine learning while Kubeflow is a great open source project, which is not given enough credit in the AWS community. In this webinar, we show how to design and build an end-to-end ML infrastructure on AWS. Agenda - Introductions - Case Study: GoCheck Kids - Overview of AWS Infrastructure for Machine Learning - Provectus ML Infrastructure on AWS - Experimentation - MLOps - Feature Store Intended Audience Technology executives & decision makers, manager-level tech roles, data engineers & data scientists, ML practitioners & ML engineers, and developers Presenters - Stepan Pushkarev, Chief Technology Officer, Provectus - Qingwei Li, ML Specialist Solutions Architect, AWS Feel free to share this presentation with your colleagues and don't hesitate to reach out to us at info@provectus.com if you have any questions! REQUEST WEBINAR: https://provectus.com/webinar-mlops-and-reproducible-ml-on-aws-with-kubeflow-and-sagemaker-aug-2020/

DevOps Days Boston 2017: Real-world Kubernetes for DevOps

Ambassador Labs

DevOps Days Boston 2017 Microservices is an increasingly popular approach to building cloud-native applications. Dozens of new technologies that streamline adopting microservices development such as Docker, Kubernetes, and Envoy have been released over the past few years. But how do you actually use these technologies together to develop, deploy, and run microservices? In this presentation, we’ll cover the nuances of deploying containerized applications on Kubernetes, including creating a Kubernetes manifest, debugging and logging, and how to build an automated continuous deployment pipeline. Then, we’ll do a brief tour of some of the advanced concepts related to microservices, including service mesh, canary deployments, resilience, and security.

How to Build Single Page HTML5 Apps that Scale

Phil Leggetter

Developing large apps is difficult. Ensuring that code is consistent, well structured, tested, maintainable and has an architecture that encourages enhancement is essential. When it comes to large server-focused apps, solutions to this problem have been tried and tested. But, how do you achieve this when building HTML5 single page apps? In this talk we’ll cover the signs to watch out for as your HTML5 SPA grows and provide examples of some of the tooling types that can contribute-to as well as ease the growing pains. Finally, we’ll demonstrate how tooling can be used to support a set of conventions, practices and principles that enable a productive developer workflow where the first line of code is feature code, features can be developed in isolation, code conflicts are avoided by grouping assets by feature and features are composed into apps. The demonstrations will use BladeRunnerJS, an open source developer toolkit written in Java, but the concepts are widely applicable.

Similar to Alexandra johnson reducing operational barriers to model training (20)

Machine Learning Infrastructure

Effective .NET Core Unit Testing with SQLite and Dapper

Webinar: Começando seus trabalhos com Machine Learning utilizando ferramentas...

Effective .NET Core Unit Testing with SQLite and Dapper

Developing Digital Twins

Pitfalls of machine learning in production

Py data scikit-production

Five Ways to Fix Your SQL Server Dev-Test Problems

Lessons Learned Replatforming A Large Machine Learning Application To Apache ...

JS Fest 2018. Никита Галкин. Микросервисная архитектура с переиспользуемыми к...

Why Upgrade to v8.6?

Ship code like a keptn

Consolidating MLOps at One of Europe’s Biggest Airports

Serverless machine learning architectures at Helixa

Using Puppet, Ansible, and MongoDB Ops Manager Together to Create Your Own On...

MongoDB World 2018: Using Puppet, Ansible and Ops Manager to Create Your Own ...

MLOps and Reproducible ML on AWS with Kubeflow and SageMaker

DevOps Days Boston 2017: Real-world Kubernetes for DevOps

How to Build Single Page HTML5 Apps that Scale

More from MLconf

Jamila Smith-Loud - Understanding Human Impact: Social and Equity Assessments...

Understanding Human Impact: Social and Equity Assessments for AI Technologies Social and Equity Impact Assessments have broad applications but can be a useful tool to explore and mitigate for Machine Learning fairness issues and can be applied to product specific questions as a way to generate insights and learnings about users, as well as impacts on society broadly as a result of the deployment of new and emerging technologies. In this presentation, my goal is to advocate for and highlight the need to consult community and external stakeholder engagement to develop a new knowledge base and understanding of the human and social consequences of algorithmic decision making and to introduce principles, methods and process for these types of impact assessments.

Ted Willke - The Brain’s Guide to Dealing with Context in Language Understanding

The Brain’s Guide to Dealing with Context in Language Understanding Like the visual cortex, the regions of the brain involved in understanding language represent information hierarchically. But whereas the visual cortex organizes things into a spatial hierarchy, the language regions encode information into a hierarchy of timescale. This organization is key to our uniquely human ability to integrate semantic information across narratives. More and more, deep learning-based approaches to natural language understanding embrace models that incorporate contextual information at varying timescales. This has not only led to state-of-the art performance on many difficult natural language tasks, but also to breakthroughs in our understanding of brain activity. In this talk, we will discuss the important connection between language understanding and context at different timescales. We will explore how different deep learning architectures capture timescales in language and how closely their encodings mimic the brain. Along the way, we will uncover some surprising discoveries about what depth does and doesn’t buy you in deep recurrent neural networks. And we’ll describe a new, more flexible way to think about these architectures and ease design space exploration. Finally, we’ll discuss some of the exciting applications made possible by these breakthroughs.

Justin Armstrong - Applying Computer Vision to Reduce Contamination in the Re...

Applying Computer Vision to Reduce Contamination in the Recycling Stream With China’s recent refusal of most foreign recyclables, North American waste haulers are scrambling to figure out how to make on-shore recycling cost-effective in order to continue providing recycling services. Recyclables that were once being shipped to China for manual sorting are now primarily being redirected to landfills or incinerators. Without a solution, a nearly $5 billion annual recycling market could come to a halt. Purity in the recycling stream is key to this effort as contaminants in the stream can increase the cost of operations, damage equipment and reduce the ability to create pure commodities suitable for creating recycled goods. This market disruption as a result of China’s new regulations, however, provides us the chance to re-examine and improve our current disposal & collection habits with modern monitoring & artificial intelligence technology. Using images from our in-dumpster cameras, Compology has developed an ML-based process that helps identify, measure and alert for contaminants in recycling containers before they are picked-up, helping keep the recycling stream clean. Our convolutional neural network flags potential instances of contamination inside a dumpster, enabling garbage haulers to know which containers have the wrong type of material inside. This allows them to provide targeted, timely education, and when appropriate, assess fines, to improve recycling compliance at the businesses and residences they serve, helping keep recycling services financially viable. In this presentation, we will walk through our ML-based contamination measurement and scoring process by showing how Waste Management, a national waste hauler, has experienced 57% contamination reduction in nearly 2,000 containers over six months, This progress shows significant strides towards financially viable recycling services.

Igor Markov - Quantum Computing: a Treasure Hunt, not a Gold Rush

Quantum Computing: a Treasure Hunt, not a Gold Rush Quantum computers promise a significant step up in computational power over conventional computers, but also suffer a number of counterintuitive limitations --- both in their computational model and in leading lab implementations. In this talk, we review how quantum computers compete with conventional computers and how conventional computers try to hold their ground. Then we outline what stands in the way of successful quantum ML applications.

Josh Wills - Data Labeling as Religious Experience

Data Labeling as Religious Experience One of the most common places to deploy a production machine learning systems is as a replacement for a legacy rules-based system that is having a hard time keeping up with new edge cases and requirements. I'll be walking through the process and tooling we used to help us design, train, and deploy a model to replace a set of static rules we had for handling invite spam at Slack, talk about what we learned, and discuss some problems to solve in order to make these migrations easier for everyone.

Vinay Prabhu - Project GaitNet: Ushering in the ImageNet moment for human Gai...

Project GaitNet: Ushering in the ImageNet moment for human Gait kinematics The emergence of the upright human bipedal gait can be traced back 4 to 2.8 million years ago, to the now extinct hominin Australopithecus afarensis. Fine grained analysis of gait using the modern MEMS sensors found on all smartphones not just reveals a lot about the person’s orthopedic and neuromuscular health status, but also has enough idiosyncratic clues that it can be harnessed as a passive biometric. While there were many siloed attempts made by the machine learning community to model Bipedal Gait sensor data, these were done with small datasets oft collected in restricted academic environs. In this talk, we will introduce the ImageNet moment for human gait analysis by presenting 'Project GaitNet', the largest ever planet-sized motion sensor based human bipedal gait dataset ever curated. We’ll also present the associated state-of-the-art results in classifying humans harnessing novel deep neural architectures and the related success stories we have enjoyed in transfer-learning into disparate domains of human kinematics analysis.

Jekaterina Novikova - Machine Learning Methods in Detecting Alzheimer’s Disea...

Machine Learning Methods in Detecting Alzheimer’s Disease from Speech and Language Alzheimer's disease affects millions of people worldwide, and it is important to predict the disease as early and as accurate as possible. In this talk, I will discuss development of novel ML models that help classifying healthy people from those who develop Alzheimer's, using short samples of human speech. As an input to the model, features of different modalities are extracted from speech audio samples and transcriptions: (1) syntactic measures, such as e.g. production rules extracted from syntactic parse trees, (2) lexical measures, such as e.g. features of lexical richness and complexity and lexical norms, and (3) acoustic measures, such as e.g. standard Mel-frequency cepstral coefficients. I will present the ML model that detects cognitive impairment by reaching agreement among modalities. The resulting model is able to achieve state of the art performance in both supervised and semi-supervised manner, using manual transcripts of human speech. Additionally, I will discuss potential limitations of any fully-automated speech-based Alzheimer's disease detection model, focusing mostly on the analysis of the impact of a not-so-accurate automatic speech recognition (ASR) on the classification performance. To illustrate this, I will present the experiments with controlled amounts of artificially generated ASR errors and explain how the deletion errors affect Alzheimer's detection performance the most, due to their impact on the features of syntactic and lexical complexity.

Meghana Ravikumar - Optimized Image Classification on the Cheap

Optimized Image Classification on the Cheap In this talk, we anchor on building an image classifier trained on the Stanford Cars dataset to evaluate two approaches to transfer learning -fine tuning and feature extraction- and the impact of hyperparameter optimization on these techniques. Once we define the most performant transfer learning technique for Stanford Cars, we will double the size of the dataset through image augmentation to boost the classifier’s performance. We will use Bayesian optimization to learn the hyperparameters associated with image transformations using the downstream image classifier’s performance as the guide. In conjunction with model performance, we will also focus on the features of these augmented images and the downstream implications for our image classifier. To both maximize model performance on a budget and explore the impact of optimization on these methods, we apply a particularly efficient implementation of Bayesian optimization to each of these architectures in this comparison. Our goal is to draw on a rigorous set of experimental results that can help us answer the question: how can resource-constrained teams make trade-offs between efficiency and effectiveness using pre-trained models?

Noam Finkelstein - The Importance of Modeling Data Collection

The Importance of Modeling Data Collection Data sets used in machine learning are often collected in a systematically biased way - certain data points are more likely to be collected than others. We call this "observation bias". For example, in health care, we are more likely to see lab tests when the patient is feeling unwell than otherwise. Failing to account for observation bias can, of course, result in poor predictions on new data. By contrast, properly accounting for this bias allows us to make better use of the data we do have. In this presentation, we discuss practical and theoretical approaches to dealing with observation bias. When the nature of the bias is known, there are simple adjustments we can make to nonparametric function estimation techniques, such as Gaussian Process models. We also discuss the scenario where the data collection model is unknown. In this case, there are steps we can take to estimate it from observed data. Finally, we demonstrate that having a small subset of data points that are known to be collected at random - that is, in an unbiased way - can vastly improve our ability to account for observation bias in the rest of the data set. My hope is that attendees of this presentation will be aware of the perils of observation bias in their own work, and be equipped with tools to address it.

June Andrews - The Uncanny Valley of ML

The Uncanny Valley of ML Every so often, the conundrum of the Uncanny Valley re-emerges as advanced technologies evolve from clearly experimental products to refined accepted technologies. We have seen its effects in robotics, computer graphics, and page load times. The debate of how to handle the new technology detracts from its benefits. When machine learning is added to human decision systems a similar effect can be measured in increased response time and decreased accuracy. These systems include radiology, judicial assignments, bus schedules, housing prices, power grids and a growing variety of applications. Unfortunately, the Uncanny Valley of ML can be hard to detect in these systems and can lead to degraded system performance when ML is introduced, at great expense. Here, we'll introduce key design principles for introducing ML into human decision systems to navigate around the Uncanny Valley and avoid its pitfalls.

Sneha Rajana - Deep Learning Architectures for Semantic Relation Detection Tasks

Deep Learning Architectures for Semantic Relation Detection Tasks Recognizing and distinguishing specific semantic relations from other types of semantic relations is an essential part of language understanding systems. Identifying expressions with similar and contrasting meanings is valuable for NLP systems which go beyond recognizing semantic relatedness and require to identify specific semantic relations. In this talk, I will first present novel techniques for creating labelled datasets required for training deep learning models for classifying semantic relations between phrases. I will further present various neural network architectures that integrate morphological features into integrated path-based and distributional relation detection algorithms and demonstrate that this model outperforms state-of-the-art models in distinguishing semantic relations and is capable of efficiently handling multi-word expressions.

Anoop Deoras - Building an Incrementally Trained, Local Taste Aware, Global D...

Building an Incrementally Trained, Local Taste Aware, Global Deep Learned Recommender System Model At Netflix, our main goal is to maximize our members’ enjoyment of the selected show by minimizing the amount of time it takes for them to find it. We try to achieve this goal by personalizing almost all the aspects of our product -- from what shows to recommend, to how to present these shows and construct their home-pages to what images to select per show, among many other things. Everything is recommendations for us and as an applied Machine Learning group, we spend our time building models for personalization that will eventually increase the joy and satisfaction of our members. In this talk we will primarily focus our attention on a) making a global deep learned recommender model that is regional tastes and popularity aware and b) adapting this model to changing taste preferences as well as dynamic catalog availability. We will first go through some standard recommender system models that use Matrix Factorization and Topic Models and then compare and contrast them with more powerful and higher capacity deep learning based models such as sequence models that use recurrent neural networks. We will show what it entails to build a global model that is aware of regional taste preferences and catalog availability. We will show how models that are built on simple Maximum Likelihood principle fail to do that. We will then describe one solution that we have employed in order to enable the global deep learned models to focus their attention on capturing regional taste preferences and changing catalog.In the latter half of the talk, we will discuss how we do incremental learning of deep learned recommender system models. Why do we need to do that ? Everything changes with time. Users’ tastes change with time. What’s available on Netflix and what’s popular also change over time. Therefore, updating or improving recommendation systems over time is necessary to bring more joy to users. In addition to how we apply incremental learning, we will discuss some of the challenges we face involving large-scale data preparation, infrastructure setup for incremental model training as well as pipeline scheduling. The incremental training enables us to serve fresher models trained on fresher and larger amounts of data. This helps our recommender system to nicely and quickly adapt to catalog and users’ taste changes, and improve overall performance.

Vito Ostuni - The Voice: New Challenges in a Zero UI World

Vito Ostuni - The Voice: New Challenges in a Zero UI World The adoption of voice-enabled devices has seen an explosive growth in the last few years and music consumption is among the most popular use cases. Music personalization and recommendation plays a major role at Pandora in providing a daily delightful listening experience for millions of users. In turn, providing the same perfectly tailored listening experience through these novel voice interfaces brings new interesting challenges and exciting opportunities. In this talk we will describe how we apply personalization and recommendation techniques in three common voice scenarios which can be defined in terms of request types: known-item, thematic, and broad open-ended. We will describe how we use deep learning slot filling techniques and query classification to interpret the user intent and identify the main concepts in the query. We will also present the differences and challenges regarding evaluation of voice powered recommendation systems. Since pure voice interfaces do not contain visual UI elements, relevance labels need to be inferred through implicit actions such as play time, query reformulations or other types of session level information. Another difference is that while the typical recommendation task corresponds to recommending a ranked list of items, a voice play request translates into a single item play action. Thus, some considerations about closed feedback loops need to be made. In summary, improving the quality of voice interactions in music services is a relatively new challenge and many exciting opportunities for breakthroughs still remain. There are many new aspects of recommendation system interfaces to address to bring a delightful and effortless experience for voice users. We will share a few open challenges to solve for the future.

Anna choromanska - Data-driven Challenges in AI: Scale, Information Selection...

Janani Kalyanam - Machine Learning to Detect Illegal Online Sales of Prescrip...

Esperanza Lopez Aguilera - Using a Bayesian Neural Network in the Detection o...

Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...

Soumith Chintala - Increasing the Impact of AI Through Better Software

Roy Lowrance - Predicting Bond Prices: Regime Changes

Madalina Fiterau - Hybrid Machine Learning Methods for the Interpretation and...

More from MLconf (20)

Jamila Smith-Loud - Understanding Human Impact: Social and Equity Assessments...

Ted Willke - The Brain’s Guide to Dealing with Context in Language Understanding

Justin Armstrong - Applying Computer Vision to Reduce Contamination in the Re...

Igor Markov - Quantum Computing: a Treasure Hunt, not a Gold Rush

Josh Wills - Data Labeling as Religious Experience

Vinay Prabhu - Project GaitNet: Ushering in the ImageNet moment for human Gai...

Jekaterina Novikova - Machine Learning Methods in Detecting Alzheimer’s Disea...

Meghana Ravikumar - Optimized Image Classification on the Cheap

Noam Finkelstein - The Importance of Modeling Data Collection

June Andrews - The Uncanny Valley of ML

Sneha Rajana - Deep Learning Architectures for Semantic Relation Detection Tasks

Anoop Deoras - Building an Incrementally Trained, Local Taste Aware, Global D...

Vito Ostuni - The Voice: New Challenges in a Zero UI World

Anna choromanska - Data-driven Challenges in AI: Scale, Information Selection...

Janani Kalyanam - Machine Learning to Detect Illegal Online Sales of Prescrip...

Esperanza Lopez Aguilera - Using a Bayesian Neural Network in the Detection o...

Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...

Soumith Chintala - Increasing the Impact of AI Through Better Software

Roy Lowrance - Predicting Bond Prices: Regime Changes

Madalina Fiterau - Hybrid Machine Learning Methods for the Interpretation and...

Recently uploaded

JMeter webinar - integration with InfluxDB and Grafana

RTTS

Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application. In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics. Length: 30 minutes Session Overview ------------------------------------------- During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana: - What out-of-the-box solutions are available for real-time monitoring JMeter tests? - What are the benefits of integrating InfluxDB and Grafana into the load testing stack? - Which features are provided by Grafana? - Demonstration of InfluxDB and Grafana using a practice web application To view the webinar recording, go to: https://www.rttsweb.com/jmeter-integration-webinar

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

"Impact of front-end architecture on development cost", Viktor Turskyi

Fwdays

I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

Bits & Pixels using AI for Good.........

Alison B. Lowndes

IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx

Abida Shariff

UiPath Test Automation using UiPath Test Suite series, part 4

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap. The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies. Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques What will you get from this session? 1. Insights into SAP testing best practices 2. Heatmap utilization for testing 3. Optimization of testing processes 4. Demo Topics covered: Execution from the test manager Orchestrator execution result Defect reporting SAP heatmap example with demo Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf