This document discusses model serving using MLflow. It covers:
1. MLflow supports both offline and online model scoring. Offline scoring uses Spark for batch predictions, while online scoring uses the MLflow scoring server for real-time predictions via REST API.
2. The MLflow scoring server can deploy models locally or to cloud providers like SageMaker. It supports various model formats and input/output data formats.
3. MLflow also has deployment plugins to serve models using other tools like TorchServe. It supports popular frameworks like PyTorch, TensorFlow, and Keras.
Productionalizing Models through CI/CD Design with MLflowDatabricks
Often times model deployment and integration consists of several moving parts that require intricate steps woven together. Automating this pipeline and feedback loop can be incredibly challenging, especially in lieu of varying model development techniques.
Continuous Delivery of ML-Enabled Pipelines on Databricks using MLflowDatabricks
ML development brings many new complexities beyond the traditional software development lifecycle. ML projects, unlike software projects, after they were successfully delivered and deployed, cannot be abandoned but must be continuously monitored if model performance still satisfies all requirements.
Developing ML-enabled Data Pipelines on Databricks using IDE & CI/CD at Runta...Databricks
Data & ML projects bring many new complexities beyond the traditional software development lifecycle. Unlike software projects, after they were successfully delivered and deployed, they cannot be abandoned but must be continuously monitored if model performance still satisfies all requirements. We can always get new data with new statistical characteristics that can break our pipelines or influence model performance.
How to Utilize MLflow and Kubernetes to Build an Enterprise ML PlatformDatabricks
In large enterprises, large solutions are sometimes required to tackle even the smallest tasks and ML is no different. At Comcast we are building a comprehensive, configuration based, continuously integrated and deployed platform for data pipeline transformations, model development and deployment. This is accomplished using a range of tools and frameworks such as Databricks, MLflow, Apache Spark and others. With a Databricks environment used by hundreds of researchers and petabytes of data, scale is critical to Comcast, so making it all work together in a frictionless experience is a high priority. The platform consists of a number of components: an abstraction for data pipelines and transformation to allow our data scientists the freedom to combine the most appropriate algorithms from different frameworks , experiment tracking, project and model packaging using MLflow and model serving via the Kubeflow environment on Kubernetes. The architecture, progress and current state of the platform will be discussed as well as the challenges we had to overcome to make this platform work at Comcast scale. As a machine learning practitioner, you will gain knowledge in: an example of data pipeline abstraction; ways to package and track your ML project and experiments at scale; and how Comcast uses Kubeflow on Kubernetes to bring everything together.
50k runs, millions of metrics, parameters or tags, some bursts at 20k QPS. That’s the volume of data managed by our MLflow tracking servers this year at Criteo. In this talk, you will learn how we set up a shared instance of MLflow at company scale. We will present our contributions to the SQLAlchemyStore to make it responsive at this scale. We will present you how we turned MLflow to a production-ready system. How we scaled horizontally a shared instance on a mesos cluster ? Our monitoring system based on prometheus. Integration to the company Single Sign-On (SSO) authentication. And how our data scientists register their runs from the largest hadoop cluster in Europe.
How to revamp machine learning pipelines with MLOps
mlflow demo video 1: https://youtu.be/3q6JXcW_lOI
airflow demo video 2: https://youtu.be/bzMn6kN-yWg
tensowflow js model serving demo video 3: https://youtu.be/M_U99Pmfaf4
I am an instructor of the MLOps workshop for some anonymous startup incubation program where the objectives are (1) to orchestrate and deploy updates to the application and the deep learning model in a unified way. (2) To design a DevOps pipeline to coordinate retrieving the latest best model from the model registry, packaging the web application, deploying the web application and inferencing web service.
Productionalizing Models through CI/CD Design with MLflowDatabricks
Often times model deployment and integration consists of several moving parts that require intricate steps woven together. Automating this pipeline and feedback loop can be incredibly challenging, especially in lieu of varying model development techniques.
Continuous Delivery of ML-Enabled Pipelines on Databricks using MLflowDatabricks
ML development brings many new complexities beyond the traditional software development lifecycle. ML projects, unlike software projects, after they were successfully delivered and deployed, cannot be abandoned but must be continuously monitored if model performance still satisfies all requirements.
Developing ML-enabled Data Pipelines on Databricks using IDE & CI/CD at Runta...Databricks
Data & ML projects bring many new complexities beyond the traditional software development lifecycle. Unlike software projects, after they were successfully delivered and deployed, they cannot be abandoned but must be continuously monitored if model performance still satisfies all requirements. We can always get new data with new statistical characteristics that can break our pipelines or influence model performance.
How to Utilize MLflow and Kubernetes to Build an Enterprise ML PlatformDatabricks
In large enterprises, large solutions are sometimes required to tackle even the smallest tasks and ML is no different. At Comcast we are building a comprehensive, configuration based, continuously integrated and deployed platform for data pipeline transformations, model development and deployment. This is accomplished using a range of tools and frameworks such as Databricks, MLflow, Apache Spark and others. With a Databricks environment used by hundreds of researchers and petabytes of data, scale is critical to Comcast, so making it all work together in a frictionless experience is a high priority. The platform consists of a number of components: an abstraction for data pipelines and transformation to allow our data scientists the freedom to combine the most appropriate algorithms from different frameworks , experiment tracking, project and model packaging using MLflow and model serving via the Kubeflow environment on Kubernetes. The architecture, progress and current state of the platform will be discussed as well as the challenges we had to overcome to make this platform work at Comcast scale. As a machine learning practitioner, you will gain knowledge in: an example of data pipeline abstraction; ways to package and track your ML project and experiments at scale; and how Comcast uses Kubeflow on Kubernetes to bring everything together.
50k runs, millions of metrics, parameters or tags, some bursts at 20k QPS. That’s the volume of data managed by our MLflow tracking servers this year at Criteo. In this talk, you will learn how we set up a shared instance of MLflow at company scale. We will present our contributions to the SQLAlchemyStore to make it responsive at this scale. We will present you how we turned MLflow to a production-ready system. How we scaled horizontally a shared instance on a mesos cluster ? Our monitoring system based on prometheus. Integration to the company Single Sign-On (SSO) authentication. And how our data scientists register their runs from the largest hadoop cluster in Europe.
How to revamp machine learning pipelines with MLOps
mlflow demo video 1: https://youtu.be/3q6JXcW_lOI
airflow demo video 2: https://youtu.be/bzMn6kN-yWg
tensowflow js model serving demo video 3: https://youtu.be/M_U99Pmfaf4
I am an instructor of the MLOps workshop for some anonymous startup incubation program where the objectives are (1) to orchestrate and deploy updates to the application and the deep learning model in a unified way. (2) To design a DevOps pipeline to coordinate retrieving the latest best model from the model registry, packaging the web application, deploying the web application and inferencing web service.
How to choose correct framework and define your manifesto for technology practices around Machine Learning Journey.
Kubernetes being successor in this space, Seldom Core and Kubeflow is truly winner in this Segment.
DevOps for Applications in Azure Databricks: Creating Continuous Integration ...Databricks
Working with our customers, developers and partners around the world, it's clear DevOps has become increasingly critical to a team's success. Continuous integration (CI) and continuous delivery (CD) which is part of DevOps, embody a culture, set of operating principles, and collection of practices that enable application development teams to deliver code changes more frequently and reliably. In this session, we will cover how you can automate your entire process from code commit to production using CI/CD pipelines in Azure DevOps for Azure Databricks applications. Using CI/CD practices, you can simplify, speed and improve your cloud development to deliver features to your customers as soon as they're ready.
Continuous Delivery of Deep Transformer-Based NLP Models Using MLflow and AWS...Databricks
Transformer-based pretrained language models such as BERT, XLNet, Roberta and Albert significantly advance the state-of-the-art of NLP and open doors for solving practical business problems with high performance transfer learning. However, operationalizing these models with production-quality continuous integration/ delivery (CI/CD) end-to-end pipelines that cover the full machine learning life cycle stages of train, test, deploy and serve while managing associated data and code repositories is still a challenging task.
Hamburg Data Science Meetup - MLOps with a Feature StoreMoritz Meister
MLOps is a trend in machine learning (ML) engineering that unifies ML system development (Dev) and ML system operation (Ops). Some ML lifecycle frameworks, such as TensorFlow Extended, are based around end-to-end pipelines that start with raw data and end in production models. During this talk we will introduce the concept of a feature store as the missing piece of ML infrastructure that enables faster lower cost deployment of models. We will show how the Hopsworks Feature Store - factors monolithic end-to-end ML pipelines into feature and model training pipelines that can each run at different cadences. We will show examples of ingestion and training pipelines including hyperparameter optimization and model deployment.
Building machine learning applications locally with Spark — Joel Pinho Lucas ...PAPIs.io
In times of huge amounts of heterogeneous data available, processing and extracting knowledge requires more and more efforts on building complex software architectures. In this context, Apache Spark provides a powerful and efficient approach for large-scale data processing. This talk will briefly introduce a powerful machine learning library (MLlib) along with a general overview of the Spark framework, describing how to launch applications within a cluster. In this way, a demo will show how to simulate a Spark cluster in a local machine using images available on a Docker Hub public repository. In the end, another demo will show how to save time using unit tests for validating jobs before running them in a cluster.
Deploying and Monitoring Heterogeneous Machine Learning Applications with Cli...Databricks
Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy serving loads. However, most machine learning frameworks and systems only address model training and not deployment.
Clipper is an open-source, general-purpose model-serving system that addresses these challenges. Interposing between applications that consume predictions and the machine-learning models that produce predictions, Clipper simplifies the model deployment process by adopting a modular serving architecture and isolating models in their own containers, allowing them to be evaluated using the same runtime environment as that used during training. Clipper’s modular architecture provides simple mechanisms for scaling out models to meet increased throughput demands and performing fine-grained physical resource allocation for each model. Further, by abstracting models behind a uniform serving interface, Clipper allows developers to compose many machine-learning models within a single application to support increasingly common techniques such as ensemble methods, multi-armed bandit algorithms, and prediction cascades.
In this talk I will provide an overview of the Clipper serving system and discuss how to get started using Clipper to serve Apache Spark and TensorFlow models on Kubernetes. I will then discuss some recent work on statistical performance monitoring for machine learning models.
MLflow 1.0 is coming soon as the first stable release of MLflow. It also packs many cleanups and improvements, such as simpler metadata management, search APIs and HDFS support. In this talk, we’ll present these new features in detail, and then discuss additional MLflow components that Databricks and other companies are working on for the rest of 2019. These new tools include a model registry to share and track models, as well as a multi-step workflow abstraction, both of which were announced at Spark + AI Summit 2019.
Команда Data Phoenix Events приглашает всех, 17 августа в 19:00, на первый вебинар из серии "The A-Z of Data", который будет посвящен MLOps. В рамках вводного вебинара, мы рассмотрим, что такое MLOps, основные принципы и практики, лучшие инструменты и возможные архитектуры. Мы начнем с простого жизненного цикла разработки ML решений и закончим сложным, максимально автоматизированным, циклом, который нам позволяет реализовать MLOps.
https://dataphoenix.info/the-a-z-of-data/
https://dataphoenix.info/the-a-z-of-data-introduction-to-mlops/
Simplifying AI integration on Apache SparkDatabricks
Spark is an ETL and Data Processing engine especially suited for big data. Most of the time an organization has different teams working on different languages, frameworks and libraries, which needs to be integrated in the ETL Pipelines or for general data processing. For example, a Spark ETL job may be written in Scala by data engineering team, but there is a need to integrate a machine learning solution written in python/R developed by Data Science team. These kinds of solutions are not very straightforward to integrate with spark engine, and it required great amount of collaboration between different teams, hence increasing overall project time and cost. Furthermore, these solutions will keep on changing/upgrading with time using latest versions of the technologies and with improved design and implementation, especially in Machine Learning domain where ML models/algorithms keep on improving with new data and new approaches. And so there is significant downtime involved in integrating the these upgraded version.
Productionzing ML Model Using MLflow Model ServingDatabricks
Productionzing ML Models are needs to ensure model integrity while it efficiently replicate runtime environments across servers besides it keep track of how each of our models were created. It helps us better trace the root cause of changes and issues over time as we acquire new data and update our model. We have greater accountability over our models and the results they generate.
MLflow Model Serving delivers cost-effective and on-click deployment of model for real-time inferences. Also the Model Version deployed in the Model Serving can also be conveniently managed with MLflow Model Registry. We will going to cover following topics Deployment, Consumption and Monitoring. For deployment, we will demo the different version deployment and validate the deployment. For consumption, we demo connecting power bi and generate prediction report using ML Model deployed in MLflow serving. Lastly will wrap up with managing the MLflow serving like, access rights and monitoring capabilities.
Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and PrometheusManasi Vartak
These are slides from Manasi Vartak's Strata Talk in March 2020 on Robust MLOps with Open-Source.
* Introduction to talk
* What is MLOps?
* Building an MLOps Pipeline
* Real-world Simulations
* Let’s fix the pipeline
* Wrap-up
A Microservices Framework for Real-Time Model Scoring Using Structured Stream...Databricks
Open-source technologies allow developers to build microservices framework to build myriad real-time applications. One such application is building the real-time model scoring. In this session,
we will showcase how to architect a microservice framework, in particular how to use it to build a low-latency, real-time model scoring system. At the core of the architecture lies Apache Spark’s Structured
Streaming capability to deliver low-latency predictions coupled with Docker and Flask as additional open source tools for model service. In this session, you will walk away with:
* Knowledge of enterprise-grade model as a service
* Streaming architecture design principles enabling real-time machine learning
* Key concepts and building blocks for real-time model scoring
* Real-time and production use cases across industries, such as IIOT, predictive maintenance, fraud detection, sepsis etc.
MLflow model serving
How to score models with MLflow
Offline scoring with Spark
Online scoring with MLflow model server
Custom model deployment and scoring
Databricks model server
Discuss the different ways model can be served with MLflow. We will cover both the open source MLflow and Databricks managed MLflow ways to serve models. Will cover the basic differences between batch scoring and real-time scoring. Special emphasis on the new upcoming Databricks production-ready model serving.
How to choose correct framework and define your manifesto for technology practices around Machine Learning Journey.
Kubernetes being successor in this space, Seldom Core and Kubeflow is truly winner in this Segment.
DevOps for Applications in Azure Databricks: Creating Continuous Integration ...Databricks
Working with our customers, developers and partners around the world, it's clear DevOps has become increasingly critical to a team's success. Continuous integration (CI) and continuous delivery (CD) which is part of DevOps, embody a culture, set of operating principles, and collection of practices that enable application development teams to deliver code changes more frequently and reliably. In this session, we will cover how you can automate your entire process from code commit to production using CI/CD pipelines in Azure DevOps for Azure Databricks applications. Using CI/CD practices, you can simplify, speed and improve your cloud development to deliver features to your customers as soon as they're ready.
Continuous Delivery of Deep Transformer-Based NLP Models Using MLflow and AWS...Databricks
Transformer-based pretrained language models such as BERT, XLNet, Roberta and Albert significantly advance the state-of-the-art of NLP and open doors for solving practical business problems with high performance transfer learning. However, operationalizing these models with production-quality continuous integration/ delivery (CI/CD) end-to-end pipelines that cover the full machine learning life cycle stages of train, test, deploy and serve while managing associated data and code repositories is still a challenging task.
Hamburg Data Science Meetup - MLOps with a Feature StoreMoritz Meister
MLOps is a trend in machine learning (ML) engineering that unifies ML system development (Dev) and ML system operation (Ops). Some ML lifecycle frameworks, such as TensorFlow Extended, are based around end-to-end pipelines that start with raw data and end in production models. During this talk we will introduce the concept of a feature store as the missing piece of ML infrastructure that enables faster lower cost deployment of models. We will show how the Hopsworks Feature Store - factors monolithic end-to-end ML pipelines into feature and model training pipelines that can each run at different cadences. We will show examples of ingestion and training pipelines including hyperparameter optimization and model deployment.
Building machine learning applications locally with Spark — Joel Pinho Lucas ...PAPIs.io
In times of huge amounts of heterogeneous data available, processing and extracting knowledge requires more and more efforts on building complex software architectures. In this context, Apache Spark provides a powerful and efficient approach for large-scale data processing. This talk will briefly introduce a powerful machine learning library (MLlib) along with a general overview of the Spark framework, describing how to launch applications within a cluster. In this way, a demo will show how to simulate a Spark cluster in a local machine using images available on a Docker Hub public repository. In the end, another demo will show how to save time using unit tests for validating jobs before running them in a cluster.
Deploying and Monitoring Heterogeneous Machine Learning Applications with Cli...Databricks
Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy serving loads. However, most machine learning frameworks and systems only address model training and not deployment.
Clipper is an open-source, general-purpose model-serving system that addresses these challenges. Interposing between applications that consume predictions and the machine-learning models that produce predictions, Clipper simplifies the model deployment process by adopting a modular serving architecture and isolating models in their own containers, allowing them to be evaluated using the same runtime environment as that used during training. Clipper’s modular architecture provides simple mechanisms for scaling out models to meet increased throughput demands and performing fine-grained physical resource allocation for each model. Further, by abstracting models behind a uniform serving interface, Clipper allows developers to compose many machine-learning models within a single application to support increasingly common techniques such as ensemble methods, multi-armed bandit algorithms, and prediction cascades.
In this talk I will provide an overview of the Clipper serving system and discuss how to get started using Clipper to serve Apache Spark and TensorFlow models on Kubernetes. I will then discuss some recent work on statistical performance monitoring for machine learning models.
MLflow 1.0 is coming soon as the first stable release of MLflow. It also packs many cleanups and improvements, such as simpler metadata management, search APIs and HDFS support. In this talk, we’ll present these new features in detail, and then discuss additional MLflow components that Databricks and other companies are working on for the rest of 2019. These new tools include a model registry to share and track models, as well as a multi-step workflow abstraction, both of which were announced at Spark + AI Summit 2019.
Команда Data Phoenix Events приглашает всех, 17 августа в 19:00, на первый вебинар из серии "The A-Z of Data", который будет посвящен MLOps. В рамках вводного вебинара, мы рассмотрим, что такое MLOps, основные принципы и практики, лучшие инструменты и возможные архитектуры. Мы начнем с простого жизненного цикла разработки ML решений и закончим сложным, максимально автоматизированным, циклом, который нам позволяет реализовать MLOps.
https://dataphoenix.info/the-a-z-of-data/
https://dataphoenix.info/the-a-z-of-data-introduction-to-mlops/
Simplifying AI integration on Apache SparkDatabricks
Spark is an ETL and Data Processing engine especially suited for big data. Most of the time an organization has different teams working on different languages, frameworks and libraries, which needs to be integrated in the ETL Pipelines or for general data processing. For example, a Spark ETL job may be written in Scala by data engineering team, but there is a need to integrate a machine learning solution written in python/R developed by Data Science team. These kinds of solutions are not very straightforward to integrate with spark engine, and it required great amount of collaboration between different teams, hence increasing overall project time and cost. Furthermore, these solutions will keep on changing/upgrading with time using latest versions of the technologies and with improved design and implementation, especially in Machine Learning domain where ML models/algorithms keep on improving with new data and new approaches. And so there is significant downtime involved in integrating the these upgraded version.
Productionzing ML Model Using MLflow Model ServingDatabricks
Productionzing ML Models are needs to ensure model integrity while it efficiently replicate runtime environments across servers besides it keep track of how each of our models were created. It helps us better trace the root cause of changes and issues over time as we acquire new data and update our model. We have greater accountability over our models and the results they generate.
MLflow Model Serving delivers cost-effective and on-click deployment of model for real-time inferences. Also the Model Version deployed in the Model Serving can also be conveniently managed with MLflow Model Registry. We will going to cover following topics Deployment, Consumption and Monitoring. For deployment, we will demo the different version deployment and validate the deployment. For consumption, we demo connecting power bi and generate prediction report using ML Model deployed in MLflow serving. Lastly will wrap up with managing the MLflow serving like, access rights and monitoring capabilities.
Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and PrometheusManasi Vartak
These are slides from Manasi Vartak's Strata Talk in March 2020 on Robust MLOps with Open-Source.
* Introduction to talk
* What is MLOps?
* Building an MLOps Pipeline
* Real-world Simulations
* Let’s fix the pipeline
* Wrap-up
A Microservices Framework for Real-Time Model Scoring Using Structured Stream...Databricks
Open-source technologies allow developers to build microservices framework to build myriad real-time applications. One such application is building the real-time model scoring. In this session,
we will showcase how to architect a microservice framework, in particular how to use it to build a low-latency, real-time model scoring system. At the core of the architecture lies Apache Spark’s Structured
Streaming capability to deliver low-latency predictions coupled with Docker and Flask as additional open source tools for model service. In this session, you will walk away with:
* Knowledge of enterprise-grade model as a service
* Streaming architecture design principles enabling real-time machine learning
* Key concepts and building blocks for real-time model scoring
* Real-time and production use cases across industries, such as IIOT, predictive maintenance, fraud detection, sepsis etc.
MLflow model serving
How to score models with MLflow
Offline scoring with Spark
Online scoring with MLflow model server
Custom model deployment and scoring
Databricks model server
Discuss the different ways model can be served with MLflow. We will cover both the open source MLflow and Databricks managed MLflow ways to serve models. Will cover the basic differences between batch scoring and real-time scoring. Special emphasis on the new upcoming Databricks production-ready model serving.
What MLflow is; what problem it solves for machine learning lifecycle; and how it solves; How it will be used with Databricks; and CI/CD pipeline with Databricks.
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...DataWorks Summit
Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.
Databricks is a popular tool used with large amounts of data, applying to many roles - including data analysts, data engineers, data scientists, and machine learning engineers. It can be found on many cloud platforms - including Azure, AWS, and GCP. In this talk, we will look at a flight-themed end-to-end solution using Azure Databricks, Azure Data Factory, Azure Storage, and Power BI. By the end of this session, you will have a better understanding of Databricks' capabilities and how it integrates with other Azure offerings.
MLflow: Infrastructure for a Complete Machine Learning Life Cycle with Mani ...Databricks
ML development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools, and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. To address these problems, many companies are building custom “ML platforms” that automate this lifecycle, but even these platforms are limited to a few supported algorithms and to each company’s internal infrastructure. In this session, we introduce MLflow, a new open source project from Databricks that aims to design an open ML platform where organizations can use any ML library and development tool of their choice to reliably build and share ML applications. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size. In this deep-dive session, through a complete ML model life-cycle example, you will walk away with:
MLflow concepts and abstractions for models, experiments, and projects
How to get started with MLFlow
Understand aspects of MLflow APIs
Using tracking APIs during model training
Using MLflow UI to visually compare and contrast experimental runs with different tuning parameters and evaluate metrics
Package, save, and deploy an MLflow model
Serve it using MLflow REST API
What’s next and how to contribute
Azure Resource Manager templates: Improve deployment time and reusabilityStephane Lapointe
Azure Resource Manager is the future of Azure and his templating features are a big improvement and simplification of how you provision resources on Azure. See how you can create ARM template in Visual Studio to create complex, multiple resources templates and how they can be combined and reused. Learn the different template functions available and how they can help you build more advanced template.
A Collaborative Data Science Development WorkflowDatabricks
Collaborative data science workflows have several moving parts, and many organizations struggle with developing an efficient and scalable process. Our solution consists of data scientists individually building and testing Kedro pipelines and measuring performance using MLflow tracking. Once a strong solution is created, the candidate pipeline is trained on cloud-agnostic, GPU-enabled containers. If this pipeline is production worthy, the resulting model is served to a production application through MLflow.
PLSSUG - Troubleshoot SQL Server performance problems like a Microsoft EngineerMarek Maśko
This is yet another session of the SQL Server performance troubleshooting category. But this time it is not focused on various techniques and methodologies, what is the case in many others presentations. On the contrary. This presentation is focused on tools which are available for a very long time. The tools which are every day used by Microsoft engineers. The tools which still are known by very small amount of people.
Running Apache Spark Jobs Using KubernetesDatabricks
Apache Spark has introduced a powerful engine for distributed data processing, providing unmatched capabilities to handle petabytes of data across multiple servers. Its capabilities and performance unseated other technologies in the Hadoop world, but while Spark provides a lot of power, it also comes with a high maintenance cost, which is why we now see innovations to simplify the Spark infrastructure.
Vertex AI is a managed machine learning platform that helps you build, deploy, and scale machine learning models faster and easier.
GitHub: https://github.com/TrilokiDA/Vertex-AI/tree/main
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent
Whether you are deploying a new application in Microservices or transitioning from a monolithic database application to a cloud-ready architecture, you will inevitably face the decision of either creating a service mesh of API’s – or – using an event bus for better durability, reliability and extensibility of your application. If you choose to go the event bus route, Kafka is an excellent choice for several reasons. One key technology not to overlook is Avro Schemas. They provide a definition for your event payload, just like an API, to ensure all of the event consumers can reliably consume the events. They also handle schema evolution as requirements change and much, much more.
In this talk we will discuss all the nuances and considerations around using Avro Schemas for your JSON event payloads. From developer tools, to DevOps approaches, versioning, governance and some “gotchas” we found when working with Avro Schemas and the Confluent Schema Registry.
Similar to DAIS Europe Nov. 2020 presentation on MLflow Model Serving (20)
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
2. Agenda
• MLflow model serving
• How to score models with MLflow
○ Online scoring with MLflow scoring server
○ Offline scoring with Spark
• Custom model deployment and scoring
3. Agenda
Offline Prediction
● High throughput
● Bulk predictions
● Predict with Spark
● Batch Prediction
● Structured Streaming
Prediction
Online Prediction
● Low latency
● Individual predictions
● Real-time model serving
● MLflow scoring server
● MLflow deployment plugin
● Serving outside MLflow
● Databricks model serving
4. Vocabulary - Synonyms
Model Serving
● Model serving
● Model scoring
● Model prediction
● Model inference
● Model deployment
Prediction
● Offline == batch prediction
○ Spark batch prediction
○ Spark streaming prediction
● Online == real-time prediction
6. Offline Scoring
• Spark is ideally suited for offline scoring
• Can use either Spark batch or structured streaming
• Use SQL with MLflow UDFs
• Distributed scoring with Spark
• Load model from MLflow Model Registry
7. MLflow Offline Scoring Example
Directly invoke model
udf = mlflow.pyfunc.spark_udf(spark, model_uri)
spark.udf.register("predict", udf)
%sql select *, predict(*) as prediction from my_data
UDF - SQL
model = mlflow.spark.load_model(model_uri)
predictions = model.transform(data)
UDF - Python
udf = mlflow.pyfunc.spark_udf(spark, model_uri)
predictions = data.withColumn("prediction", udf(*data.columns))
model_uri = "models:/sklearn-wine/production"
Model URI
8. Online Scoring
• Score models outside of Spark
• Variety of real-time deployment options:
○ REST API - web servers - self-managed or as containers in cloud providers
○ Embedded in browsers, .e.g TensorFlow Lite
○ Edge devices: mobile phones, sensors, routers, etc.
○ Hardware accelerators - GPU, NVIDIA, Intel
9. Options to serve real-time models
• Embed model in application code
• Model deployed as service
• Model published as data
• Martin Fowler’s Continuous Delivery for Machine Learning
10. MLflow Scoring Server
• Basis of different MLflow deployment targets
• Standard REST API:
○ Input: CSV or JSON (pandas-split or pandas-records formats)
○ Output: JSON list of predictions
• Implementation: Python Flask or Java Jetty server
• MLflow CLI builds a server with embedded model
• See Built-In Deployment Tools
11. MLflow Scoring Server deployment options
• Local web server
• SageMaker docker container
• Azure ML docker container
• Generic docker container
12. MLflow Scoring Server container types
Python Server
Model and its ML
framework embedded
in Flask server.
Only for non-Spark ML
models.
Python Server + Spark
Flask server delegates
scoring to Spark
server. Only for Spark
ML models.
Java MLeap Server
Model and MLeap
runtime are embedded
in Jetty server. Only
for Spark ML models.
No Spark runtime.
18. MLflow SageMaker and Azure ML containers
• Two types of containers to deploy to cloud providers
○ Python container - Flask web server process with embedded model
○ SparkML container - Flask web server process and Spark process
• SageMaker container
○ Most versatile container type
○ Can run in local mode on laptop as regular docker container
19. Python container
• Flask web server
• Model (e.g. sklearn or Keras/TF) is embedded inside web server
20. SparkML container
• Two processes:
○ Flask web server
○ Spark server - OSS Spark - no Databricks
• Web server delegates scoring to Spark server
21. MLflow SageMaker container deployment
• CLI:
○ mlflow sagemaker build-and-push-container
○ mlflow sagemaker deploy
○ mlflow sagemaker run-local
• API: mlflow.sagemaker API
• Deploy a model on Amazon SageMaker
23. MLflow AzureML container deployment
• Deploy to Azure Kubernetes Service (AKS) or Azure Container
Instances (ACI)
• CLI - mlflow azureml build-image
• API - mlflow.azureml.build-image
• Deploy a python_function model on Microsoft Azure ML
25. MLeap
• Non-Spark serialization format for Spark models
• Advantage is ability to score Spark model without overhead of
Spark
• Faster than scoring Spark ML models with Spark
• Problem is stability, maturity and lack of dedicated commercial
support
27. Databricks Model Serving
• Expose MLflow model predictions as REST endpoint
• One-node cluster is automatically provisioned
• HTTP endpoint publicly exposed
• Limited production capability - intended for light loads and testing
• Production-grade serving on the product roadmap
28. Model Serving on Databricks
Model Serving
Turnkey serving
solution to expose
models as REST
endpoints
Reports
Applications
...
REST
Endpoint
Tracking
Record and query
experiments: code,
metrics, parameters,
artifacts, models
Models
General format
that standardizes
deployment options
Logged
Model
Model Registry
Centralized and
collaborative
model lifecycle
management
Staging Production Archived
Data Scientists Deployment Engineers
30. Databricks Model Serving Resources
• Blog posts
○ Quickly Deploy, Test, and Manage ML Models as REST Endpoints with MLflow
Model Serving on Databricks - 2020-11-02
○ Announcing MLflow Model Serving on Databricks - 2020-06-25
• Documentation
○ MLflow Model Serving on Databricks
31. MLflow Deployment Plugins
• MLflow Deployment Plugins - Deploy model to custom serving tool
• Current deployment plugins:
○ https://github.com/RedisAI/mlflow-redisai
○ https://github.com/mlflow/mlflow-torchserve
32. MLflow PyTorch
• MLflow PyTorch integration released in MLflow 1.12.0 (Nov. 2020)
• Features:
○ PyTorch Autologging
○ TorchScript Models
○ TorchServing
• Deploy MLflow PyTorch models into TorchServe
33. MLflow PyTorch and TorchServe Resources
• PyTorch and MLflow Integration Announcement - 2020-11-12
• MLflow 1.12 Features Extended PyTorch Integration - 2012-11-13
• MLflow and PyTorch — Where Cutting Edge AI meets MLOps
• https://github.com/mlflow/mlflow-torchserve
34. MLflow and TensorFlow Serving Custom
Deployment
• Example how to deploy models to a custom deployment target
• MLflow deployment plugin has not yet been developed
• Deployment builds a standard TensorFlow Serving docker
container with a MLflow model from Model Registry
• https://github.com/amesar/mlflow-
tools/tree/master/mlflow_tools/tensorflow_serving
35. Keras/TensorFlow Model Formats
• HD5 format
○ Default in Keras TF 1.x but is legacy in Keras TF 2.x
○ Not supported by TensorFlow Serving
• SavedModel format
○ TensorFlow standard format is default in Keras TF 2.x
○ Supported by TensorFlow Serving
36. MLflow Keras/TensorFlow Flavors
• MLflow currently only supports Keras HD5 format as flavor
• MLflow does not support SavedModel flavor until this is resolved:
○ https://github.com/mlflow/mlflow/issues/3224
○ https://github.com/mlflow/mlflow/pull/3552
• Can save SavedModel as an unmanaged artifact for now
38. TensorFlow Serving Example
docker run -t --rm --publish 8501
--volume /opt/mlflow/mlruns/1/f48dafd70be044298f71488b0ae10df4/artifacts/tensorflow-model:/models/keras_wine
--env MODEL_NAME=keras_wine
tensorflow/serving
Launch server
Launch server in Docker container
curl -d '{"instances": [12.8, 0.03, 0.48, 0.98, 6.2, 29, 1.2, 0.4, 75 ] }'
-X POST http://localhost:8501/v1/models/keras_wine:predict
Launch server
Request
{ "predictions": [[-0.70597136]] }
Launch server
Response
39. MLflow Keras/TensorFlow Run Example
MLflow Flavors - Managed Models
● Models
○ keras-hd5-model
○ onnx-model
● Can be served as PyFunc models
MLflow Unmanaged Models
● Models
○ tensorflow-model - Standard TF SavedModel format
○ tensorflow-lite-model
○ tensorflow.js model
● Load as raw artifacts and then serve manually
● Sample code: wine_train.py and wine_predict.py
40. ONNX
• Interoperable model format supported by:
○ MSFT, Facebook, AWS, Nvidia, Intel, etc.
• Train once, deploy anywhere (in theory) - depends on ML tool
• Save MLflow model as ONNX flavor
• Deploy in standard MLflow scoring server
• ONNX runtime is embedded in Python server in MLflow container
What did they do with us?
what are they trying to do? recommendation? content curation?
how does that work?
How come Delta and Spark and those things can help with that thing (recommendation, or whatever they do)?
What did they do with us?
what are they trying to do? recommendation? content curation?
how does that work?
How come Delta and Spark and those things can help with that thing (recommendation, or whatever they do)?
What did they do with us?
what are they trying to do? recommendation? content curation?
how does that work?
How come Delta and Spark and those things can help with that thing (recommendation, or whatever they do)?
What did they do with us?
what are they trying to do? recommendation? content curation?
how does that work?
How come Delta and Spark and those things can help with that thing (recommendation, or whatever they do)?
1. Aha! feature
2. Description of the feature:
With MLflow we already have a Model format that abstracts away the ML framework. It doesn’t matter whether you use TensorFlow, Scikit, or SparkML, if you log it as an MLflow Model you can deploy the models in the same way.
The tracking server allows you to log those models in experiments, with metrics, parameters, etc.
Once you want to deploy a model, the Model Registry takes care of versioning, the deployment lifecycle, and hand-off between Data Scientists and Deployment Engineers
From the Model Registry, there are several deployment options. However, many customers have asked us for a turnkey solution to serve models as REST endpoints. So we are building a Serving solution to do exactly that, with two clicks.
3. Value of the feature (aka what databricks was before this feature, and what this feature will do for databricks)
Compared to other serving solutions this will require only 2 clicks to be enabled. However, important to highlight that this is for reporting / debug purposes and will not autoscale to arbitrary request loads.
4. Associated summary slide bullet point:
Native support for serving MLflow Models as REST endpoints
5. Cloud: All
6. Deployment (MT, ST, Azure)
All