Emeli Dral (Evidently AI) – Analyze it: production monitoring for machine learning models

•

0 likes•184 views

Machine learning models in production are susceptible to failures from data and concept drift issues that can degrade model performance over time. Effective monitoring of model health and data quality is needed to detect such issues. Key aspects to monitor include model accuracy, data schema changes, data distribution shifts, broken data pipelines, and the impact of concept drift. Starting simply by adding basic machine learning metrics to existing service monitoring for memory, latency, and uptime is pragmatic. The appropriate monitoring approach depends on factors like available resources, use case importance, and system complexity.

Technology

Analyze it:
production monitoring for
machine learning models
Emeli Dral
CTO Evidently AI

About me
https://www.linkedin.com/in/emelidral
• Сo-founder & CTO Evidently AI
• Ex Chief Data Scientist at Yandex Data Factory and Mechanica AI
• Co-founder of Data Mining in Action, largest oﬄine data science
course in Russia
• Co-author of two Coursera specializations in data science with >
100K students
• Lecturer at Harbour.Space University, GSOM MBA
50+ Industrial applications of machine
learning

Issues with data quality and integrity
Data processing issues
Broken pipelines,
infrastructure updates,
wrong source…
Data schema change
Change in the upstream
system, external APIs,
catalogue update…
Data loss at the source
Broken sensor, logging
error, database outage…
°C 🡪 °F

Broken upstream model
Aggregated
traﬃc data
Prediction of
traﬃc jams
Prediction of
ETA
Car in route
Route
suggestion

Gradual concept drift
Model accuracy
Time
Model accuracy
Time
model retraining

Sudden concept drift
predicted
actual
Time
Sales of
loungewear
national
lockdown
announced

Data drift: change in feature distribution
Example: users come from a new channel.
user acquisition channel (T1
)
paid
search
organic
social
user acquisition channel (T2
)
paid
search
organic
social

Model health
Data health DATA DRIFT
BROKEN PIPELINES
SCHEMA CHANGE
CONCEPT DRIFT
MODEL BIAS
UNDERPERFORMING
SEGMENTS
MODEL ACCURACY
UPTIME
DATA OUTAGE
Service health
How is machine learning monitoring diﬀerent?
MEMORY
LATENCY

How to monitor?
Memory Request time av.
Server requests MAPE
Add ML metrics to service
health monitoring
(e.g. Prometheus/Grafana)
ML-focused Reports /
Dashboards
(e.g. BI tools Tableau, Looker; or
custom in Matplotlib, Plotly)

Monitoring approach: factors to consider
Team resources
• Development
resources
Use case importance
• Economic value
• Cost of error
• Risks
Complexity
• Data source diversity
• Pipeline complexity
• Batch / real-time
• Immediate / delayed
response

According to Forrester Research, only 22% of companies are currently seeing a significant return from data science expenditures. Most data science implementations are high-cost IT projects, local applications that are not built to scale for production workflows, or laptop decision support projects that never impact customers. Despite this high failure rate, we keep hearing the same mantra and solutions over and over again. Everybody talks about how to create models, but not many people talk about getting them into production where they can impact customers. Harvinder Atwal offers an entertaining and practical introduction to DataOps, a new and independent approach to delivering data science value at scale, used at companies like Facebook, Uber, LinkedIn, Twitter, and eBay. The key to adding value through DataOps is to adapt and borrow principles from Agile, Lean, and DevOps. However, DataOps is not just about shipping working machine learning models; it starts with better alignment of data science with the rest of the organization and its goals. Harvinder shares experience-based solutions for increasing your velocity of value creation, including Agile prioritization and collaboration, new operational processes for an end-to-end data lifecycle, developer principles for data scientists, cloud solution architectures to reduce data friction, self-service tools giving data scientists freedom from bottlenecks, and more. The DataOps methodology will enable you to eliminate daily barriers, putting your data scientists in control of delivering ever-faster cutting-edge innovation for your organization and customers.

MLOps Using MLflow

Databricks

MLflow is an MLOps tool that enables data scientist to quickly productionize their Machine Learning projects. To achieve this, MLFlow has four major components which are Tracking, Projects, Models, and Registry. MLflow lets you train, reuse, and deploy models with any library and package them into reproducible steps. MLflow is designed to work with any machine learning library and require minimal changes to integrate into an existing codebase. In this session, we will cover the common pain points of machine learning developers such as tracking experiments, reproducibility, deployment tool and model versioning. Ready to get your hands dirty by doing quick ML project using mlflow and release to production to understand the ML-Ops lifecycle.

How to deploy machine learning models into production

DataWorks Summit

Data scientists spend a lot of time on data cleaning and munging, so that they can finally start with the fun part of their job: building models. After you have engineered the features and tested different models, you see how the prediction performance improves. However, the job is not done when you have a high performing model. The deployment of your models is a crucial step in the overall workflow and it is the point in time when your models actually become useful to your company. In this session you will learn about various possibilities and best practices to bring machine learning models into production environments. The goal is not only to make live prediction calls or have the models available as REST API, but also what needs to be considered to maintain them. This talk will focus on solutions with Python (flask, Cloud Foundry, Docker, and more) and the well established ML packages such as Spark MLlib, scikit-learn, and xgboost, but the concepts can be easily transferred to other languages and frameworks. Speaker Sumit Goyal, IBM, Software Engineer

Managing the Machine Learning Lifecycle with MLflow

Databricks

Drifting Away: Testing ML Models in Production

Databricks

Deploying machine learning models has become a relatively frictionless process. However, properly deploying a model with a robust testing and monitoring framework is a vastly more complex task. There is no one-size-fits-all solution when it comes to productionizing ML models, oftentimes requiring custom implementations utilising multiple libraries and tools. There are however, a set of core statistical tests and metrics one should have in place to detect phenomena such as data and concept drift to prevent models from becoming unknowingly stale and detrimental to the business. Combining our experiences from working with Databricks customers, we do a deep dive on how to test your ML models in production using open source tools such as MLflow, SciPy and statsmodels. You will come away from this talk armed with knowledge of the key tenets for testing both model and data validity in production, along with a generalizable demo which uses MLflow to assist with the reproducibility of this process.

ODSC May 2019 - The DataOps Manifesto

DataKitchen

Simplifying Model Management with MLflow

Databricks

<p>Last summer, Databricks launched MLflow, an open source platform to manage the machine learning lifecycle, including experiment tracking, reproducible runs and model packaging. MLflow has grown quickly since then, with over 120 contributors from dozens of companies, including major contributions from R Studio and Microsoft. It has also gained new capabilities such as automatic logging from TensorFlow and Keras, Kubernetes integrations, and a high-level Java API. In this talk, we’ll cover some of the new features that have come to MLflow, and then focus on a major upcoming feature: model management with the MLflow Model Registry. Many organizations face challenges tracking which models are available in the organization and which ones are in production. The MLflow Model Registry provides a centralized database to keep track of these models, share and describe new model versions, and deploy the latest version of a model through APIs. We’ll demonstrate how these features can simplify common ML lifecycle tasks.</p>

Introduction to MLflow

Databricks

ML development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. To address these problems, many companies are building custom “ML platforms” that automate this lifecycle, but even these platforms are limited to a few supported algorithms and to each company’s internal infrastructure. In this talk, I present MLflow, a new open source project from Databricks that aims to design an open ML platform where organizations can use any ML library and development tool of their choice to reliably build and share ML applications. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size.

When interacting with analytics dashboards, in order to achieve a smooth user experience, two major key requirements are quick response time and data freshness. To meet the requirements of creating fast interactive BI dashboards over streaming data, organizations often struggle with selecting a proper serving layer. Cluster computing frameworks such as Hadoop or Spark work well for storing large volumes of data, although they are not optimized for making it available for queries in real time. Long query latencies also make these systems suboptimal choices for powering interactive dashboards and BI use cases. This talk presents an open source real-time data analytics stack using Apache Kafka, Druid, and Superset. The stack combines the low-latency streaming and processing capabilities of Kafka with Druid, which enables immediate exploration and provides low-latency queries over the ingested data streams. Superset provides the visualization and dashboarding that integrates nicely with Druid. In this talk we will discuss why this architecture is well suited to interactive applications over streaming data, present an end-to-end demo of complete stack, discuss its key features, and discuss performance characteristics from real-world use cases. NISHANT BANGARWA, Software engineer, Hortonworks

Productionizing Machine Learning with a Microservices Architecture

Databricks

Introdution to Dataops and AIOps (or MLOps)

Adrien Blind

ML-Ops:Philosophy, Best-Practices and Tools

Jorge Davila-Chacon

Modern Data Platforms

Arne Roßmann

Washington DC DataOps Meetup -- Nov 2019

DataKitchen

Modern Data Flow

confluent

Building Robust ETL Pipelines with Apache Spark

Databricks

Stable and robust ETL pipelines are a critical component of the data infrastructure of modern enterprises. ETL pipelines ingest data from a variety of sources and must handle incorrect, incomplete or inconsistent records and produce curated, consistent data for consumption by downstream applications. In this talk, we’ll take a deep dive into the technical details of how Apache Spark “reads” data and discuss how Spark 2.2’s flexible APIs; support for a wide variety of datasources; state of art Tungsten execution engine; and the ability to provide diagnostic feedback to users, making it a robust framework for building end-to-end ETL pipelines.

"Managing the Complete Machine Learning Lifecycle with MLflow"

Databricks

Machine Learning development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools, and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. To address these problems, many companies are building custom “ML platforms” that automate this lifecycle, but even these platforms are limited to a few supported algorithms and to each company’s internal infrastructure. In this session, we introduce MLflow, a new open-source project from Databricks that aims to design an open ML platform where organizations can use any ML library and development tool of their choice to reliably build and share ML applications. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size.

MLOps and Data Quality: Deploying Reliable ML Models in Production

Provectus

Looking to build a robust machine learning infrastructure to streamline MLOps? Learn from Provectus experts how to ensure the success of your MLOps initiative by implementing Data QA components in your ML infrastructure. For most organizations, the development of multiple machine learning models, their deployment and maintenance in production are relatively new tasks. Join Provectus as we explain how to build an end-to-end infrastructure for machine learning, with a focus on data quality and metadata management, to standardize and streamline machine learning life cycle management (MLOps). Agenda - Data Quality and why it matters - Challenges and solutions of Data Testing - Challenges and solutions of Model Testing - MLOps pipelines and why they matter - How to expand validation pipelines for Data Quality

Leveraging Generative AI & Best practices

DianaGray10

Open core summit: Observability for data pipelines with OpenLineage

Julien Le Dem

Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...

Mihai Criveti

Mihai is the Principal Architect for Platform Engineering and Technology Solutions at IBM, responsible for Cloud Native and AI Solutions. He is a Red Hat Certified Architect, CKA/CKS, a leader in the IBM Open Innovation community, and advocate for open source development. Mihai is driving the development of Retrieval Augmentation Generation platforms, and solutions for Generative AI at IBM that leverage WatsonX, Vector databases, LangChain, HuggingFace and open source AI models. Mihai will share lessons learned building Retrieval Augmented Generation, or “Chat with Documents” platforms and APIs that scale, and deploy on Kubernetes. His talk will cover use cases for Generative AI, limitations of Large Language Models, use of RAG, Vector Databases and Fine Tuning to overcome model limitations and build solutions that connect to your data and provide content grounding, limit hallucinations and form the basis of explainable AI. In terms of technology, he will cover LLAMA2, HuggingFace TGIS, SentenceTransformers embedding models using Python, LangChain, and Weaviate and ChromaDB vector databases. He’ll also share tips on writing code using LLM, including building an agent for Ansible and containers. Scaling factors for Large Language Model Architectures: • Vector Database: consider sharding and High Availability • Fine Tuning: collecting data to be used for fine tuning • Governance and Model Benchmarking: how are you testing your model performance over time, with different prompts, one-shot, and various parameters • Chain of Reasoning and Agents • Caching embeddings and responses • Personalization and Conversational Memory Database • Streaming Responses and optimizing performance. A fine tuned 13B model may perform better than a poor 70B one! • Calling 3rd party functions or APIs for reasoning or other type of data (ex: LLMs are terrible at reasoning and prediction, consider calling other models) • Fallback techniques: fallback to a different model, or default answers • API scaling techniques, rate limiting, etc. • Async, streaming and parallelization, multiprocessing, GPU acceleration (including embeddings), generating your API using OpenAPI, etc.

LanGCHAIN Framework

Keymate.AI

Langchain Framework is an innovative approach to linguistic data processing, combining the principles of language sciences, blockchain technology, and artificial intelligence. This deck introduces the groundbreaking elements of the framework, detailing how it enhances security, transparency, and decentralization in language data management. It discusses its applications in various fields, including machine learning, translation services, content creation, and more. The deck also highlights its key features, such as immutability, peer-to-peer networks, and linguistic asset ownership, that could revolutionize how we handle linguistic data in the digital age.

Screw DevOps, Let's Talk DataOps

Kellyn Pot'Vin-Gorman

Actionable Requirements

Armond Mehrabian

Improving Apache Spark for Dynamic Allocation and Spot Instances

Databricks

This presentation will explore the new work in Spark 3.1 adding the concept of graceful decommissioning and how we can use this to improve Spark’s performance in both dynamic allocation and spot/preemptable instances. Together we’ll explore how Spark’s dynamic allocation has evolved over time, and why the different changes have been needed. We’ll also look at the multi-company collaboration that resulted in being able to deliver this feature and I’ll end with encouraging pointers on how to get more involved in Spark’s development.

Big Query - Utilizing Google Data Warehouse for Media Analytics

hafeeznazri

MLOps in action

Pieter de Bruin

Build an LLM-powered application using LangChain.pdf

StephenAmell4

LangChain is an advanced framework that allows developers to create language model-powered applications. It provides a set of tools, components, and interfaces that make building LLM-based applications easier. With LangChain, managing interactions with language models, chaining together various components, and integrating resources like APIs and databases is a breeze. The platform includes a set of APIs that can be integrated into applications, allowing developers to add language processing capabilities without having to start from scratch.

Data ops: Machine Learning in production

Stepan Pushkarev

Helping data scientists escape the seduction of the sandbox - Krish Swamy, We...

Sri Ambati

This talk was given at H2O World 2018 NYC and can be viewed here: https://youtu.be/xc3j20Om3UM Description: Data science is indeed one of the sexy jobs of the 21st century. But it is also a lot of hard work. And the hard work is seldom about the math or the algorithms. It is about building relevant machine learning products for the real world. We will go over some of the must-haves as you take your machine learning model out of the sandbox and make it work in the big, bad world outside. Speaker's Bio: Krish Swamy is an experienced professional with deep skills in applying analytics and BigData capabilities to challenging business problems and driving customer insights. Krish's analytic experience includes marketing and pricing, credit risk, digital analytics and most recently, big data analytics and data transformation. His key experiences lie in banking and financial services, the digital customer experience domain, with a background in management consulting. Other key skills include influencing organizational change towards a data and analytics driven culture, and building teams of analysts, statisticians and data scientists.

What's hot

Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...

DataWorks Summit

Productionizing Machine Learning with a Microservices Architecture

Databricks

Introdution to Dataops and AIOps (or MLOps)

Adrien Blind

ML-Ops:Philosophy, Best-Practices and Tools

Jorge Davila-Chacon

Modern Data Platforms

Arne Roßmann

Washington DC DataOps Meetup -- Nov 2019

DataKitchen

Modern Data Flow

confluent

Building Robust ETL Pipelines with Apache Spark

Databricks

"Managing the Complete Machine Learning Lifecycle with MLflow"

Databricks

MLOps and Data Quality: Deploying Reliable ML Models in Production

Provectus

Leveraging Generative AI & Best practices

DianaGray10

Open core summit: Observability for data pipelines with OpenLineage

Julien Le Dem

Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...

Mihai Criveti

LanGCHAIN Framework

Keymate.AI

Screw DevOps, Let's Talk DataOps

Kellyn Pot'Vin-Gorman

Actionable Requirements

Armond Mehrabian

Improving Apache Spark for Dynamic Allocation and Spot Instances

Databricks

Big Query - Utilizing Google Data Warehouse for Media Analytics

hafeeznazri

MLOps in action

Pieter de Bruin

Build an LLM-powered application using LangChain.pdf

StephenAmell4

What's hot (20)

Interactive real-time dashboards on data streams using Kafka, Druid, and Supe...

Productionizing Machine Learning with a Microservices Architecture

Introdution to Dataops and AIOps (or MLOps)

ML-Ops:Philosophy, Best-Practices and Tools

Modern Data Platforms

Washington DC DataOps Meetup -- Nov 2019

Modern Data Flow

Building Robust ETL Pipelines with Apache Spark

"Managing the Complete Machine Learning Lifecycle with MLflow"

MLOps and Data Quality: Deploying Reliable ML Models in Production

Leveraging Generative AI & Best practices

Open core summit: Observability for data pipelines with OpenLineage

Retrieval Augmented Generation in Practice: Scalable GenAI platforms with k8s...

LanGCHAIN Framework

Screw DevOps, Let's Talk DataOps

Actionable Requirements

Improving Apache Spark for Dynamic Allocation and Spot Instances

Big Query - Utilizing Google Data Warehouse for Media Analytics

MLOps in action

Build an LLM-powered application using LangChain.pdf

Similar to Emeli Dral (Evidently AI) – Analyze it: production monitoring for machine learning models

Data ops: Machine Learning in production

Stepan Pushkarev

Helping data scientists escape the seduction of the sandbox - Krish Swamy, We...

Sri Ambati

Making Data Science Scalable - 5 Lessons Learned

Laurenz Wuttke

Making Data Science Scalable - 5 Lessons Learned Making Data Science and Machine Learning scalable is not easy: #1 Data Science in silos is bad #2 ML-Feature stores should be at the heart of every ML-Platform #3 Auto ML works great if you have a Feature store #4 Treat Data Science Projekts more like Software Development #5 Cloude based Infrastructure makes it easy to get started Data Science MeetUp Cologne, Germany 16. May 2019 datasolut GmbH - https://datasolut.com

Managing Machines: The New AI Dev Stack

Peter Skomoroch

Machine learning drove massive growth at consumer internet companies over the last decade, and this was enabled by open software, datasets, and AI research. For many problems, machine learning will produce better, faster, and more repeatable decisions at scale. Unfortunately, building and maintaining these systems is still extremely difficult and expensive. As more machine learning software moves to production, many of our traditional tools and best practices in software development will change. Pete Skomoroch walks you through what you need to know as we shift from a world of deterministic programs to systems that give unpredictable results on ever-changing training data. To navigate this world powered by nondeterministic data-dependent programs, we’ll also need a new development stack to help us write, test, deploy, and monitor machine learning software. Presented at OSCON Portland July 18, 2019

Msst 2019 v4

Nisha Talagala

Machine learning systems for engineers

Cameron Joannidis

SESE 2021: Where Systems Engineering meets AI/ML

CARLOS III UNIVERSITY OF MADRID

ML platforms & auto ml - UEM annotated (2) - #digitalbusinessweek

Ed Fernandez

Storage Challenges for Production Machine Learning

Nisha Talagala

OSCON 2014: Data Workflows for Machine LearningPaco Nathan

Machine learning and big data

Poo Kuan Hoong

A Practical-ish Introduction to Data Science

Mark West

In this talk I will share insights and knowledge that I have gained from building up a Data Science department from scratch. This talk will be split into three sections: 1. I'll begin by defining what Data Science is, how it is related to Machine Learning and share some tips for introducing Data Science to your organisation. 2. Next up well run through some commonly used Machine Learning algorithms used by Data Scientists, along with examples for use cases where these algorithms can be applied. 3. The final third of the talk will be a demonstration of how you can quickly get started with Data Science and Machine Learning using Python and the Open Source scikit-learn Library.

AutoML - Heralding a New Era of Machine Learning - CASOUG Oct 2021

Sandesh Rao

GeeCon Prague 2018 - A Practical-ish Introduction to Data Science

Mark West

Data Science has been described as the sexiest job of the 21st Century. But what is Data Science? And what has Machine Learning got to do with all this? In this session I will share insights and knowledge that I have gained from building up a Data Science department from scratch. The talk will be split into three sections: 1. I’ll begin by defining what Data Science is, how it is related to Machine Learning and share some tips for introducing Data Science to your organization. 2. Next up we’ll run through some commonly used Machine Learning algorithms used by Data Scientists, along with examples for use cases where these algorithms can be applied. 3. The final third of the talk will be a demonstration of how you can quickly get started with Data Science and Machine Learning using Python and the Open Source scikit-learn Library.

Data Workflows for Machine Learning - Seattle DAML

Paco Nathan

First public meetup at Twitter Seattle, for Seattle DAML: http://www.meetup.com/Seattle-DAML/events/159043422/ We compare/contrast several open source frameworks which have emerged for Machine Learning workflows, including KNIME, IPython Notebook and related Py libraries, Cascading, Cascalog, Scalding, Summingbird, Spark/MLbase, MBrace on .NET, etc. The analysis develops several points for "best of breed" and what features would be great to see across the board for many frameworks... leading up to a "scorecard" to help evaluate different alternatives. We also review the PMML standard for migrating predictive models, e.g., from SAS to Hadoop.

Introducción al Machine Learning Automático

Sri Ambati

¿Cómo puede llevar el aprendizaje automático a las masas? Los proyectos de Machine Learning con la búsqueda de talento, el tiempo para construir e implementar modelos y confiar en los modelos que se construyen. ¿Cómo puede tener varios equipos en su organización para crear modelos de ML precisos sin ser expertos en ciencia de datos o aprendizaje automático? ¿Se pregunta sobre los diferentes sabores de AutoML? H2O Driverless AI emplea las técnicas de científicos expertos en datos en una aplicación fácil de usar que ayuda a escalar sus esfuerzos de ciencia de datos. La inteligencia artificial Driverless permite a los científicos de datos trabajar en proyectos más rápido utilizando la automatización y la potencia de computación de vanguardia de las GPU para realizar tareas en minutos que solían tomar meses. Con H2O Driverless AI, todos, incluyendo expertos y científicos de datos junior, científicos de dominio e ingenieros de datos pueden desarrollar modelos confiables de aprendizaje automático. Esta plataforma de aprendizaje automático de última generación ofrece una funcionalidad única y avanzada para la visualización de datos, la ingeniería de características, la interpretabilidad del modelo y la implementación de baja latencia. H2O Driverless AI hace: * Visualización automática de datos * Ingeniería automática de funciones a nivel de Grandmaster * Selección automática del modelo * Ajuste y capacitación automáticos del modelo * Paralelización automática utilizando múltiples CPU o GPU * Ensamblaje automático del modelo *automática del Interpretaciónaprendizaje automático (MLI) * Generación automática de código de puntuación ¿Quieres probarlo tú mismo? Puede obtener una prueba gratuita aquí: H2O Driverless AI trial. Venga a esta sesión y descubra cómo comenzar con el Aprendizaje automático automático con AI sin conductor H2O, y cree modelos potentes con solo unos pocos clics. ¡Te veo pronto! Acerca de H2O.ai H2O.ai es una empresa visionaria de software de código abierto de Silicon Valley que creó y reimaginó lo que es posible. Somos una empresa de fabricantes que trajeron al mercado nuevas plataformas y tecnologías para impulsar el movimiento de inteligencia artificial. Somos los creadores de, H2O, la principal plataforma de aprendizaje de ciencia de datos de fuente abierta y de aprendizaje automático utilizada por casi la mitad de Fortune 500 y en la que confían más de 14,000 organizaciones y cientos de miles de científicos de datos de todo el mundo.

Data Mining 101

Ali Septiandri

2020 09-16-ai-engineering challanges

Ivica Crnkovic

Practical Applications of Machine Learning in Cybersecurity

scoopnewsgroup

Machine learning

Saravanan Subburayal

Similar to Emeli Dral (Evidently AI) – Analyze it: production monitoring for machine learning models (20)

Data ops: Machine Learning in production

Helping data scientists escape the seduction of the sandbox - Krish Swamy, We...

Making Data Science Scalable - 5 Lessons Learned

Managing Machines: The New AI Dev Stack

Msst 2019 v4

Machine learning systems for engineers

SESE 2021: Where Systems Engineering meets AI/ML

ML platforms & auto ml - UEM annotated (2) - #digitalbusinessweek

Storage Challenges for Production Machine Learning

OSCON 2014: Data Workflows for Machine Learning

Machine learning and big data

A Practical-ish Introduction to Data Science

AutoML - Heralding a New Era of Machine Learning - CASOUG Oct 2021

GeeCon Prague 2018 - A Practical-ish Introduction to Data Science

Data Workflows for Machine Learning - Seattle DAML

Introducción al Machine Learning Automático

Data Mining 101

2020 09-16-ai-engineering challanges

Practical Applications of Machine Learning in Cybersecurity

Machine learning

Recently uploaded

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

20 Comprehensive Checklist of Designing and Developing a Website

Pixlogix Infotech

Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

Communications Mining Series - Zero to Hero - Session 1

DianaGray10

This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered: • Communication Mining Overview • Why is it important? • How can it help today’s business and the benefits • Phases in Communication Mining • Demo on Platform overview • Q/A

zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs

Alex Pruden

This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second). Paper: https://eprint.iacr.org/2023/1886

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Malak Abu Hammad

Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers: * What is Vector Search? * Importance and benefits of vector search * Practical use cases across various industries * Step-by-step implementation guide * Live demos with code snippets * Enhancing LLM capabilities with vector search * Best practices and optimization strategies Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications. #MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology

RESUME BUILDER APPLICATION Project for students

KAMESHS29

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

Neo4j

20240607 QFM018 Elixir Reading List May 2024

Matthew Sinclair

Full-RAG: A modern architecture for hyper-personalization

Zilliz

Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

SOFTTECHHUB

The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing. One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

National Security Agency - NSA mobile device best practices

Quotidiano Piemontese

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

Neo4j

Leonard Jayamohan, Partner & Generative AI Lead, Deloitte This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.

Large Language Model (LLM) and it’s Geospatial Applications

Rohit Gautam

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

Neo4j

Dr. Sean Tan, Head of Data Science, Changi Airport Group Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

Recently uploaded (20)

Securing your Kubernetes cluster_ a step-by-step guide to success !

20 Comprehensive Checklist of Designing and Developing a Website

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

A tale of scale & speed: How the US Navy is enabling software delivery from l...

Communications Mining Series - Zero to Hero - Session 1

zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

RESUME BUILDER APPLICATION Project for students

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

20240607 QFM018 Elixir Reading List May 2024

Full-RAG: A modern architecture for hyper-personalization

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

National Security Agency - NSA mobile device best practices

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

Large Language Model (LLM) and it’s Geospatial Applications

GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...

20240605 QFM017 Machine Intelligence Reading List May 2024

Emeli Dral (Evidently AI) – Analyze it: production monitoring for machine learning models

1. Analyze it: production monitoring for machine learning models Emeli Dral CTO Evidently AI

2. About me https://www.linkedin.com/in/emelidral • Сo-founder & CTO Evidently AI • Ex Chief Data Scientist at Yandex Data Factory and Mechanica AI • Co-founder of Data Mining in Action, largest oﬄine data science course in Russia • Co-author of two Coursera specializations in data science with > 100K students • Lecturer at Harbour.Space University, GSOM MBA 50+ Industrial applications of machine learning

3. How ML models fail in production?

4. Issues with data quality and integrity Data processing issues Broken pipelines, infrastructure updates, wrong source… Data schema change Change in the upstream system, external APIs, catalogue update… Data loss at the source Broken sensor, logging error, database outage… °C 🡪 °F

5. Broken upstream model Aggregated traﬃc data Prediction of traﬃc jams Prediction of ETA Car in route Route suggestion

6. Gradual concept drift Model accuracy Time Model accuracy Time model retraining

7. Sudden concept drift predicted actual Time Sales of loungewear national lockdown announced

8. Data drift: change in feature distribution Example: users come from a new channel. user acquisition channel (T1 ) paid search organic social user acquisition channel (T2 ) paid search organic social

9. How to prepare? Monitoring.

10. Model health Data health DATA DRIFT BROKEN PIPELINES SCHEMA CHANGE CONCEPT DRIFT MODEL BIAS UNDERPERFORMING SEGMENTS MODEL ACCURACY UPTIME DATA OUTAGE Service health How is machine learning monitoring diﬀerent? MEMORY LATENCY

11. How to monitor? Memory Request time av. Server requests MAPE Add ML metrics to service health monitoring (e.g. Prometheus/Grafana) ML-focused Reports / Dashboards (e.g. BI tools Tableau, Looker; or custom in Matplotlib, Plotly)

12. How to start? Let’s be pragmatic.

13. Monitoring approach: factors to consider Team resources • Development resources Use case importance • Economic value • Cost of error • Risks Complexity • Data source diversity • Pipeline complexity • Batch / real-time • Immediate / delayed response

14. Use Case: Bike Demand Prediction

15. Demo time!

Emeli Dral (Evidently AI) – Analyze it: production monitoring for machine learning models

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Emeli Dral (Evidently AI) – Analyze it: production monitoring for machine learning models

Similar to Emeli Dral (Evidently AI) – Analyze it: production monitoring for machine learning models (20)

More from Codiax

More from Codiax (20)

Recently uploaded

Recently uploaded (20)

Emeli Dral (Evidently AI) – Analyze it: production monitoring for machine learning models