Download to read offline
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
CNCF-Istanbul-MLOps for Devops Engineers.pptx
- 1. Bridging DevOps andMLOps: A Practitioner’s Guide CNCF Istanbul, October 2024
- 2.
- 3.
- 4. 🥜 DevOps ina Nutshell Plan & Code Build & Test Release & Deploy Monitor & Learn
- 5. 5 The practice ofdeploying machine learning models into production reliably and efficiently. 🤖 What is MLOps?
- 6. Because, many ofthe challenges facing developers also apply to data scientists 6 🥰 Why MLOps?
- 7. ● Siloed organizationand poor communication between teams ● “Works on my machine” ♀️ ️ ♀️ ♀️ ♀️ ♀️ ♀️ ♀️ ♀️ ♀️ ♀️ ♀️ ♀️ ♀️ ♀️ ● Lacking the ability to properly test, deploy, maintain software ● Not having access to decision makers ● Verify the model/feature you deploy is still relevant ● Reproducibility, traceability and explainability ● ... 7 🙊 Common Challenges
- 8. 8 The Machine LearningLifecycle Data Engineering Data Ingestion Data Cleansing Data Analysis Data Transformation Data Validation Data Science Data Splitting Feature Engineering Model Development Model Training Training Optimization Model Validation Continuous Integration & Deployment Data Preprocessing App Dev / Heuristics Inferencing Pipeline Deployment Targets Deployment Patterns Monitor / alerts Consumption & optimization metrics Satisficing (Gating) metric Logging & Visualization Explainability, Interpolation Drift, Decay, Skew, Shift Improvements Gather and Prep Data Deployment Monitoring Training MLOps DataOps Experimentation 🦄 MLOps Overview
- 9. 9 🦄 MLOps Overview OperationalizingAI/ML requires collaboration App developer ML platform engineer Data engineer Data scientist ML engineer Business leadership Set goals Gather and prepare data Training Monitoring Deployment Every member of your team plays a critical role in a complex process
- 10. 11 🦄 MLOps Overview TheMachine Learning Lifecycle Gather and prepare data Monitor model Develop model Retrain model Deploy model
- 11. 12 🦄 MLOps Overview We’veseen this before.. Gather and prepare data Monitor model Develop model Retrain model Code Deploy Operate & monitor QA Iterate Deploy model
- 12.
- 13. 15 🔥 Data ScienceEssentials What is a model really? In a nutshell, it is a set of parameters plus the algorithm or the neural network architecture, that can be packaged in a single (usually binary or compressed) file.
- 14. 16 🔥 Data ScienceEssentials Square Pentagon Triangle Raw Data Labeled Data Training Data Test Data Square Pentagon Squar e Triangle TrianglePentagon Square Pentagon Triangle Square Triangle Model Training Model + Model Artifact Model Evaluation 87% Accuracy .82 R-square .032 MSE Model Training Overview
- 15. 17 🔥 Data ScienceEssentials Model Serving Model Model in a Container Clients API Input Prediction
- 16. 18 🔥 Data ScienceEssentials What about LLMs?
- 17.
- 18. 20 Clark Kent ML Platform Engineer ckent@redhat.com (123)456-7890 Metropolis, NY SKILLS Kubernetes Day Two Ops GPU Management GitOps Containers Python Package Management CAREER OBJECTIVE To enable Data Science teams to scale development and experimentation of Data Science projects using Kube Native tooling to more rapidly prove value with machine learning. WORK EXPERIENCE ● Created GPU enabled kubernetes cluster to enable multiple data science teams to collaborate and rapidly iterate on data science experiments. Resulted in the a reduction from the time of experiment idea to proof of value and an increase in the number of value added experiments. ● Managed large multi-tenant environment and provided best practices for managing shared resources for large, distributed compute ML training jobs, including GPU, CPU, and large memory pools. This effort resulted in increased resource utilization, and faster training times for ML jobs. ● Created multiple Jupyter Notebook Images with team specific Python packages to increase collaboration and reduce number of python dependency issues. ● Established multi-cluster architecture to enable training and deployment of ML models alongside existing non-ML microservices. 🐈 ML Platform Engineer
- 19. 21 🐈 ML PlatformEngineer Multi Cluster Architecture ML Training Cluster Application Cluster Multi-Tenant Projects Multi-Tenant Projects GitOps Management GitOps Management Accelerated Compute S3 Compatible Storage Cluster Monitoring Cluster Monitoring IDE Distributed ML Training ML Pipelines Model Serving Model Monitoring Model Explainability S3 Compatible Storage Experiment Deployment
- 20.
- 21. 23 🐈 ⬛ ML Engineer Lois Lane MLEngineer llane@redhat.com (123) 987-6543 Metropolis, NY SKILLS Kubernetes GitOps CI/CD Automation Python Testing REST/GRCP APIs Observability CAREER OBJECTIVE Help businesses to evolve ML Experiments to production ready inference services by creating repeatable pipelines while building trust in ML services. WORK EXPERIENCE ● Assisted Data Scientist to transform ML Experiment into production ready model and deploy ML model as an API endpoint that is able to be consumed as a microservice. Enabled data science team to actualize value of experiment and get the model out of Jupyter. ● Created repeatable pipeline to orchestrate training and deployment of ML model as a REST API. Resulted in rapid iteration of ML model enabling increased accuracy of predictions. ● Created robust ML testing to ensure code changes result in accurate ML models and validate that deployed models using blue/green deployment strategies. Resulted in reduced number of rolled back models, and increased performance of model accuracy against production results. ● Created observability with dashboards and alerts of model performance with relation to inference time, accuracy of predictions, and impact on business objectives.
- 22. 24 🐈 ⬛ ML Engineer BuildTraining Container ML Pipeline Gather data Process data Train model Download existing model Compare new model with existing Deploy new model if better
- 23. 25 🐈 ⬛ ML Engineer PythonPackage Landscape Model Training Data Tools Package Management Poetry Code Quality and Testing Polars Other ML Tools Miscellaneous
- 24.
- 25. 27 DevOps methodology forML models. Operationalize CI/CD pipeline for ML. ❤️opendatahub.io github.com/opendatahub-io
- 26.
- 27. 29 📖 What’s inthe box? ● Scale distributed computing for AI ● Automatically adjust the underlying workers based on demands ● Multi-user Jupyter ● Used for data science and research ● Streamline the entire ML lifecycle ● Accelerate model development & deployment ● Design pipeline with drag and drop ease ● Kubeflow integration ● Supports multiple ML frameworks ● Deploy and scale AI models quickly and efficiently ● AI Explainability Toolkit ● Aims to mitigate AI bias, enhancing trust and fairness in AI systems Trusty AI
- 28. 30 Sounds interesting? 🎶 Finalthings Try Open Data Hub Try Red Hat Developer Sandbox - OpenShift AI Playground Check AI On OpenShift Join upstream communities like Kubeflow, KServe, TrustyAI
- 29.
Editor's Notes
- #3 I’m not a data scientist..I took several statistic classes, used some python numpy lib for a project back in the university but that’s all :D but luckily I have a sister who helps me to understand :) But I do know about DevOps and I know these seems two different world but there are some overlaps between two and I know that there are DevOps practices can help Data Science processes ….. So I’m going to focus on this part.. But before, a bit of definition.. Assistt process of data science...because data science is fastly embeded in our application and deployment and verification fastly and saglam getting more and more important What is this overlap looks llike
- #4 Super simple cycle of devops
- #5 Key Points: MLOps is an evolution of DevOps principles and tries to apply many of the same capabilities to Machine Learning: Collaboration Automation Continuous Improvement Many technologies overlap with traditional DevOps tools such as containers, but often times Data Science capabilities require some unique tooling or processes Primary goal of MLOps is to help get past the experiment phase and deploy machine learning models so that business value can be actualized
- #7 Make this one better.. Siloed org -> data analysts, data scientists, IT infra and business people/decision makers..
- #8 Key Points: High level overview of phases In 2024 most companies are still struggling to get out of the experiment phase Data Scientists know how build models but don’t know how to deploy and integrate them into traditional applications MLOps is primarily focused on automating Training, deployment and monitoring Good MLOps implementation can help to decrease time to production, and allow model development to iterate quicker, improving inference accuracy
- #9 Key Points: High level overview of phases In 2024 most companies are still struggling to get out of the experiment phase Data Scientists know how build models but don’t know how to deploy and integrate them into traditional applications MLOps is primarily focused on automating Training, deployment and monitoring Good MLOps implementation can help to decrease time to production, and allow model development to iterate quicker, improving inference accuracy
- #10 Here’s the technical detail of the lifecycle of machine learning. It begins with codifying the problem and then defining some key metrics. For example, as a supermarket, you might have a problem with shoplifters, especially at self checkout. What metrics could you use to inform a decision about whether or not someone is stealing groceries? What data do you have available to you? This leads to a team working on collecting and cleaning the data that’s available to provide it to a data scientist. The data scientist or team of scientists tries to come up with some ways to infer the likelihood of shoplifting, trains models based on data, and does some testing. Ultimately, those models have to be deployed somewhere so that they can serve their predictions in the greater context of the intelligent application. And, those models need to be continually monitored to validate whether or not they are performing as needed.
- #11 Here’s the technical detail of the lifecycle of machine learning. It begins with codifying the problem and then defining some key metrics. For example, as a supermarket, you might have a problem with shoplifters, especially at self checkout. What metrics could you use to inform a decision about whether or not someone is stealing groceries? What data do you have available to you? This leads to a team working on collecting and cleaning the data that’s available to provide it to a data scientist. The data scientist or team of scientists tries to come up with some ways to infer the likelihood of shoplifting, trains models based on data, and does some testing. Ultimately, those models have to be deployed somewhere so that they can serve their predictions in the greater context of the intelligent application. And, those models need to be continually monitored to validate whether or not they are performing as needed.
- #12 Here’s the technical detail of the lifecycle of machine learning. It begins with codifying the problem and then defining some key metrics. For example, as a supermarket, you might have a problem with shoplifters, especially at self checkout. What metrics could you use to inform a decision about whether or not someone is stealing groceries? What data do you have available to you? This leads to a team working on collecting and cleaning the data that’s available to provide it to a data scientist. The data scientist or team of scientists tries to come up with some ways to infer the likelihood of shoplifting, trains models based on data, and does some testing. Ultimately, those models have to be deployed somewhere so that they can serve their predictions in the greater context of the intelligent application. And, those models need to be continually monitored to validate whether or not they are performing as needed.
- #13 Interestingly, we believe that this machine learning lifecycle looks exactly like any other software development lifecycle. And, Red Hat has developed a powerful container platform in OpenShift that provides tremendous benefit to the software, or machine learning, development lifecycle.
- #14 No matter what role you play in a team you need to have some fundamental knowledge of how data science works.
- #15 Key Points: Models start as an architecture defined in code, which are trained using large amounts of data to determine the best parameters to predict the desired outcome Models can been a number of different types for traditional statistical models complex neural networks After models are trained, they can be saved as an artifact and that artifact can be deployed for future inferences
- #16 Process Overview: Start with a collection of Raw Data (shapes) Human adds labels to all of the shapes (this is what we want to have the model predict) Split data into training and test data sets Build a model, and process the training data through the model by iterating over the data, allowing the model to make small improvements at each step and repeating until it is good This process is the most compute intensive part of training and primarily where GPUs are utilized After the model is trained we use the reserved test dataset to evaluate the model
- #17 Process Overview: Model Artifact created in training process is loaded into a container and served with an API endpoint Model Artifacts can be built into the container image or dynamically loaded at runtime from S3 Traditional applications can create a request to the API endpoint to have it perform an inference on a single point of data Prediction is returned to client application and able to use the prediction to take some form of action
- #18 Key Points: LLM’s are the fastest growing area of interest in Machine Learning LLM’s are are something that everyone is interested in talking about, but there is still a ton of problems to be solved by non-LLM ML solutions LLM’s do introduce a large number of new concepts and requires new capabilities compared to traditional ML Attend the Delivering Generative AI Chatbots talk to learn more Trevor’s Personal Opinion: LLM’s are going to become a commodity solution before tooling for custom LLM development and deployment mature enough to make it truly viable for most customers.
- #19 No matter what role you play in a team you need to have some fundamental knowledge of how data science works.
- #20 Key Points: This is the role that RH consulting is most likely to play and help enable our customers to succeed Large portion of current OpenShift consultants should be able to help deliver in this role New Concerns for ML Platforms: Management of GPU resources adds new capabilities and complexities. They are expensive resources and companies want to manage them effectively. Model training can require massive amounts of GPU, CPU, and Memory for computing both in single containers or massive distributed jobs. Resource planning for burstable workloads can be very different from resource planning for long lived applications. Data Scientists need help managing container images and the Python packages in them used for development
- #21 Key Points: Not uncommon to separate out Training and Deployment of ML models to different clusters Many customers we are working with already have one or more application clusters that we work with to add serving capabilities to Training is a resource intensive task that can potentially disrupt traditional workloads or cause resource contention issues Training is more likely to require GPU resources. Most serving use cases don’t require GPU, except when you get into very large, compute intensive models like LLMs. Training environments may also have very different data access requirements compared to an application environment Just like customers may need multiple application clusters for different application tiers (dev/test/prod), customers may need multiple training environments with different levels of access depending on data access requirements.
- #22 No matter what role you play in a team you need to have some fundamental knowledge of how data science works.
- #23 Key Points: ML Engineer is a role that requires skills with K8s, data science, automation, and solid python skills Python skills are critical. Much of the focus of this role is taking Data Science experiment code and converting it into something that is maintainable in the long run. Key Deliverables: Deploying models as a REST API Automation for training and deploying models Help get Data Science processes out of Jupyter Notebooks Creating observability of models and helping to provide insights into when models need to be retrained For Red Hat this is an area where we tend to help deliver and skill up our customers as part of a “Phase II” engagement after the platform is deployed
- #24 Key Points: Automation of the model training process is one of the major deliverables for ML Engineering Many of these steps are defined and created by the Data Scientists but as an ML Engineer we need to help make those steps maintainable and repeatable
- #25 Key Points: Python skills are critical to ML Engineer path and these are all tools that an ML Engineer should have some level of familiarity with Model Training: SKLearn is the goto tool for most traditional classical statistical models. SKLearn can handle 90% of real world problems customers are facing. PyTorch and TensorFlow are the most talked about frameworks since they are Deep Learning Frameworks. PyTorch seems to be taking over in popularity. Keras used to be a third party tool for TensorFlow but now comes out of the box with TF2. Data Tools: 90% of data since is working with data and not with model development. If you were to only learn a single tool on this list it should be Pandas. Other ML Tools: Lots of specialized ML Tools designed to solve specific problems. Package Management Python’s Package Management is one of the worst parts of the language. Highly recommend that you learn something beyond pip such as poetry or pipenv. Code Quality and Testing One of the most undervalued aspects of ML development Expect this area to rapidly grow as more traditional development practices get pulled into ML development Miscellaneous Flask/FastAPI/LiteStar - All three are API Frameworks. Flask is the most popular and mature framework but lacks asynchronous capabilities. FastAPI is gaining a lot of popularity but has the challenge that the creator not accepting community PRs. LiteStar is newer and attempting to be a better FastAPI with strong community support. Good to know at least one. Deep Dive into Specific Packages: Model Training: PyTorch and Tensorflow are primary deem learning libraries. PyTorch is gaining traction on TF and the area I would recommend learning if you are going to learn one. Keras is an abstraction layer built into TensorFlow SKLearn is the goto tool for most traditional classical statistical models. SKLearn can handle 90% of real world problems customers are facing. Data Tools: Pandas is the single most important library for people to learn. Polars is a neat project that provides some possible performance gains over Pandas in a Pandas compatible SDK. Dask is another alternative to Pandas that provides distributed capabilities NumPy helps to power them all. Plotly, MatPlotLib, and Seaborn are all data visualization libraries Other ML Tools: Onnx is a cross framework model format that is heavily utilized in OpenShift AI Kubeflow Pipelines SDK and Airflow both provide pipeline capabilities. Lots of overlap in capabilities with distinct benefits. Ray/CodeFlare for distributed compute and model training MLFlow is the industry leading model registry for tracking model training metadata and artifacts. RH has its own model registry tool planned for development. Package Management PIP - default package manager for python. Easy to get started with, but can create long term problems with dependency management. PipEnv and Poetry are both popular and powerful alternatives to pip. Highly recommend learning one. Anaconda is a third party tool that became popular due to a lot of early challenges with installing ML packages. Less important in modern times. Code Quality and Testing Black - Opinionated code formatting. It just works. PyLint/Flake8 - Python style guide enforcement. Familiarize yourself with PEP8 standards. Ruff - New kid on the block that does everything the other tools do, but faster. PyTest - Python code testing framework. Testing one of the most underrepresented skills in ML. GreatExpectations - Data quality testing framework. Tox - Makefiles for python Miscellaneous Requests - Curl for python Boto3 - Tool for interacting with S3 buckets Pickle - Standard library in Python used for serializing files Loguru - No nonsense log configuration Flask/FastAPI/LiteStar - All three are API Frameworks. Flask is the most popular and mature framework but lacks asynchronous capabilities. FastAPI is gaining a lot of popularity but has the challenge that the creator not accepting community PRs. LiteStar is newer and attempting to be a better FastAPI with strong community support. Good to know at least one.
- #26 No matter what role you play in a team you need to have some fundamental knowledge of how data science works.
- #27 DevOps practices should be used to improve quality and speed up delivery of your machine learning models Leveraging a containerized environment (i.e. Kubernetes, OpenShift) to build, train, and deploy models is becoming the ideal you need ML software tool chain (e.g. Tensorflow, Jupyter notebooks, Python etc.), and data services (e.g. SQL Server, No SQL, data lakes, etc.) Open Data Hub is a Reference Architecture based on open source community projects that helps demonstrate value of Red Hat portfolio and open source technologies to accelerate the AI/ML lifecycle. Deployment of various components of Open Data Hub are fully automated with an Open Data Hub Kubernetes operator. An open source project based on Kubeflow that provides open source AI tools for running large and distributed AI workloads on OpenShift Container Platform Provide an end-to-end AI/ML platform on OpenShift Easy operator deployment for the platform on OCP Provide Tools for each stage in the AI/ML platform and for all AI/ML user personas optimized for OpenShift Provide monitoring tools for model and services used by DevOps Provide development tools for Data Scientists Provide ETL tools used by Data Engineers AI/ML pipelines and long processing tasks