Webinar kubernetes and-spark
•
0 likes•132 views
Spark is a unified analytics engine for large-scale data processing that can run on different platforms including Kubernetes. Spark on Kubernetes uses Kubernetes to manage Spark driver and executor pods, allowing Spark applications to leverage Kubernetes features like auto-scaling. The workshop demonstrated submitting a Spark Pi job directly to a Kubernetes cluster, and discussed how CNVRG can help manage, monitor, and automate Spark workloads on Kubernetes with features like reproducible jobs and a unified dashboard.
Report
Share
Report
Share
Download to read offline
Recommended
How to set up Kubernetes for all your machine learning workflows
How to set up Kubernetes for all your machine learning workflowscnvrg.io AI OS - Hands-on ML Workshops
The goal of data science teams are to build and deploy high impact models. Data scientists prefer to focus on building algorithms, while data engineers focus on performance and productionizing machine learning. Kubernetes is an orchestration platform that can be deployed anywhere and can serve any kind of machine and deep learning environment. Kubernetes is a great tool for data scientists to use to stay productive and for data engineers to get production-ready results. In this free workshop you’ll learn how to build your own Kubernetes to use in your next machine learning pipeline.
Watch all our webinars at https://cnvrg.io/webinars-and-workshops/Deploy your machine learning models to production with Kubernetes
Deploy your machine learning models to production with Kubernetescnvrg.io AI OS - Hands-on ML Workshops
Learn how to deploy your model to production in 30 minutes and..
• Reduce costs and man power with auto scaling
• Load balanced the traffic
• Natively monitored by Kubernetes
• Update your model continuously: canary deployments, blue/green deploymentsCnvrg webinar continual learning
Learn why continual learning is important, and how to use it in your machine learning models to improve accuracy. You can download the full webinar here: https://info.cnvrg.io/continual-learning-webinar
Machine learning on kubernetes
This document discusses machine learning infrastructure on Kubernetes. It describes how Kubernetes now supports stateful applications and data processing workloads through new abstractions. It introduces Kubeflow, which provides tools like JupyterHub, Tensorflow Training Controller, and Tensorflow Serving to make it easier to build and run machine learning workflows on Kubernetes. It also discusses efforts to run Apache Spark and Apache Airflow on Kubernetes to enable machine learning pipelines. The goal is for Kubernetes to provide a platform to orchestrate full machine learning workflows and leverage various frameworks.
Training And Serving ML Model Using Kubeflow by Jayesh Sharma
We will walk through the exploration, training and serving of a machine learning model by leveraging Kubeflow's main components. We will use Jupyter notebooks on the cluster to train the model and then introduce Kubeflow Pipelines to chain all the steps together, to automate the entire process.
Monitor Azure Kubernetes Cluster With Prometheus by Mamta Jha
Prometheus is a popular open source metric monitoring solution and Azure Monitor provides a seamless onboarding experience to collect Prometheus metrics. Learn how to configure scraping of Prometheus metrics with Azure Monitor for containers running in AKS cluster.
Alexandra johnson reducing operational barriers to model training
This document discusses reducing operational barriers to machine learning model training through building machine learning infrastructure. It presents challenges faced by both machine learning experts and infrastructure engineers. It then describes SigOpt's solution of building SigOpt Orchestrate to address these challenges through containerization, Kubernetes for parallel training, and a command line interface for viewing progress and debugging. The final slides invite connecting with SigOpt and note they are hiring.
Netflix Cloud Platform and Open Source
Andrew Spyker presented on Netflix's cloud platform and open source projects. Some key points included:
- Netflix has migrated from monolithic architectures to microservices and continuous delivery enabled by their open source libraries and services.
- Their platform focuses on elasticity, high availability through automation, and operational visibility.
- Netflix uses technologies like Eureka, Ribbon, Hystrix, and Servo to enable scalability, resilience, and monitoring across their distributed systems.
- They contribute over 50 open source projects to help others adopt their cloud-native approaches and are working on data and UI related projects.
Recommended
How to set up Kubernetes for all your machine learning workflows
How to set up Kubernetes for all your machine learning workflowscnvrg.io AI OS - Hands-on ML Workshops
The goal of data science teams are to build and deploy high impact models. Data scientists prefer to focus on building algorithms, while data engineers focus on performance and productionizing machine learning. Kubernetes is an orchestration platform that can be deployed anywhere and can serve any kind of machine and deep learning environment. Kubernetes is a great tool for data scientists to use to stay productive and for data engineers to get production-ready results. In this free workshop you’ll learn how to build your own Kubernetes to use in your next machine learning pipeline.
Watch all our webinars at https://cnvrg.io/webinars-and-workshops/Deploy your machine learning models to production with Kubernetes
Deploy your machine learning models to production with Kubernetescnvrg.io AI OS - Hands-on ML Workshops
Learn how to deploy your model to production in 30 minutes and..
• Reduce costs and man power with auto scaling
• Load balanced the traffic
• Natively monitored by Kubernetes
• Update your model continuously: canary deployments, blue/green deploymentsCnvrg webinar continual learning
Learn why continual learning is important, and how to use it in your machine learning models to improve accuracy. You can download the full webinar here: https://info.cnvrg.io/continual-learning-webinar
Machine learning on kubernetes
This document discusses machine learning infrastructure on Kubernetes. It describes how Kubernetes now supports stateful applications and data processing workloads through new abstractions. It introduces Kubeflow, which provides tools like JupyterHub, Tensorflow Training Controller, and Tensorflow Serving to make it easier to build and run machine learning workflows on Kubernetes. It also discusses efforts to run Apache Spark and Apache Airflow on Kubernetes to enable machine learning pipelines. The goal is for Kubernetes to provide a platform to orchestrate full machine learning workflows and leverage various frameworks.
Training And Serving ML Model Using Kubeflow by Jayesh Sharma
We will walk through the exploration, training and serving of a machine learning model by leveraging Kubeflow's main components. We will use Jupyter notebooks on the cluster to train the model and then introduce Kubeflow Pipelines to chain all the steps together, to automate the entire process.
Monitor Azure Kubernetes Cluster With Prometheus by Mamta Jha
Prometheus is a popular open source metric monitoring solution and Azure Monitor provides a seamless onboarding experience to collect Prometheus metrics. Learn how to configure scraping of Prometheus metrics with Azure Monitor for containers running in AKS cluster.
Alexandra johnson reducing operational barriers to model training
This document discusses reducing operational barriers to machine learning model training through building machine learning infrastructure. It presents challenges faced by both machine learning experts and infrastructure engineers. It then describes SigOpt's solution of building SigOpt Orchestrate to address these challenges through containerization, Kubernetes for parallel training, and a command line interface for viewing progress and debugging. The final slides invite connecting with SigOpt and note they are hiring.
Netflix Cloud Platform and Open Source
Andrew Spyker presented on Netflix's cloud platform and open source projects. Some key points included:
- Netflix has migrated from monolithic architectures to microservices and continuous delivery enabled by their open source libraries and services.
- Their platform focuses on elasticity, high availability through automation, and operational visibility.
- Netflix uses technologies like Eureka, Ribbon, Hystrix, and Servo to enable scalability, resilience, and monitoring across their distributed systems.
- They contribute over 50 open source projects to help others adopt their cloud-native approaches and are working on data and UI related projects.
Apache Airflow
Introductory talk on Apache Airflow (Incubator) by Sumit Maheshwari at recent Bangalore Big Data Meetup.
Machine Learning Infrastructure
Machine learning infrastructure solve data scientists' problems using infrastructure tools. This talk shows the case study of building SigOpt Orchestrate, an ML infrastructure tool. The talk highlights how data scientists' concerns as user mapped to solutions with some of today's most popular infrastructure tools.
To learn more about SigOpt Orchestrate: https://sigopt.com/orchestrate
Originally given as a talk for UC Berkeley's Women in Electrical Engineering and Computer Science group on January 24, 2019.
Matt Chung (Independent) - Serverless application with AWS Lambda
The talk will focus on how we are utilizing AWS Lambda for certain applications and the advantages/disadvantages, and the challenges we discovered along the way. It would help those who are looking to reduce technical debt with the infrastructure and costs.
Previously a Director of technical operations at fox networks (21st Century Fox/News Corporation) responsible for infrastructure and building deployment pipelines. Currently a Python programmer / DevOps engineer with roots in systems/networks administration. Focus is on infrastructure and application automation. Worked as an engineer for Cisco Systems with emphasis on video conferencing. Built microwave networks at Bel Air Internet. Find me on github and twitter @itsmemattchung
Video: https://www.youtube.com/watch?v=BLcElBUhfrQ
Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London
Follow DOXLON on twitter http://www.twitter.com/doxlon
Knative, Serverless on Kubernetes, and Openshift
This document provides an overview of serverless computing and Knative. It begins with defining serverless computing and distinguishing it from Function as a Service (FaaS). It then provides a high-level overview of the main components of Knative, including Knative Serving for building and deploying serverless applications and Knative Eventing for connecting those applications. It also introduces OpenShift Serverless, which is Red Hat's implementation of Knative on OpenShift. The document includes several demos and labs for hands-on experience with Knative and serverless concepts.
Nextflow and AWS Batch - GCC/BOSC 2018
This document discusses using Nextflow and AWS Batch for bioinformatics analysis. It describes how Nextflow provides orchestration, parallelization, scalability, portability, and reproducibility capabilities. AWS Batch automates dynamic computing clusters and optimizes EC2 instance types used for jobs. The document shows how Nextflow can use AWS Batch as an executor, with Nextflow automatically converting tasks to AWS Batch jobs and managing input/output data transfer with S3. It shares the author's experience using Nextflow and AWS Batch together at their company, noting benefits of scalability and reproducibility as well as some optimization opportunities for large-scale workflows.
Hydrosphere.io for ODSC: Webinar on Kubeflow
Webinar video: https://www.youtube.com/watch?v=Y3_fcJBgpMw
Kubeflow and Beyond: Automation of Model Training, Deployment, Testing, Monitoring, and Retraining
Speakers:
Stepan Pushkarev, CTO, Hydrosphere.io and Ilnur Garifullin is an ML Engineer, Hydrosphere.io
Abstract: Very often a workflow of training models and delivering them to the production environment contains loads of manual work. Those could be either building a Docker image and deploying it to the Kubernetes cluster or packing the model to the Python package and installing it to your Python application. Or even changing your Java classes with the defined weights and re-compiling the whole project. Not to mention that all of this should be followed by testing your model's performance. It hardly could be named "continuous delivery" if you do it all manually. Imagine you could run the whole process of assembling/training/deploying/testing/running model via a single command in your terminal. In this webinar, we will present a way to build the whole workflow of data gathering/model training/model deployment/model testing into a single flow and run it with a single command.
Java & Microservices in Azure
Deploy & Run your Java Application on Azure and Azure Stack, in a hybrid environment spanning public and private clouds.
Apache Airflow Architecture
Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.
Cloudsolutionday 2016: Docker & FAAS at getvero.com
This document discusses Docker and cloud functions at Vero. It provides details on:
1. How Vero uses containers and Docker to run over 20 kinds of worker functions, handling up to 16 million emails per day and tracking customer actions.
2. How Vero built its own cloud function service to have more control over scaling and security compared to other cloud providers' services.
3. Challenges faced including startup latency initially and horizontal scaling, and how Vero overcame these challenges through auto-scaling and other techniques.
Serverless spark
Building and deploying an analytic service on Cloud is a challenge. A bigger challenge is to maintain the service. In a world where users are gravitating towards a model where cluster instances are to provisioned on the fly, in order for these to be used for analytics or other purposes, and then to have these cluster instances shut down when the jobs get done, the relevance of containers and container orchestration is more important than ever. In short Customers are looking for Serverless Spark Clusters. The Intent of this presentation is to share what is Serverless Spark and what are the benefits of running Spark in serverless manner.
DevOps for Databricks
Applying DevOps to Databricks can be a daunting task. In this talk this will be broken down into bite size chunks. Common DevOps subject areas will be covered, including CI/CD (Continuous Integration/Continuous Deployment), IAC (Infrastructure as Code) and Build Agents.
We will explore how to apply DevOps to Databricks (in Azure), primarily using Azure DevOps tooling. As a lot of Spark/Databricks users are Python users, will will focus on the Databricks Rest API (using Python) to perform our tasks.
Kubernetes in Adform
The document discusses Adform's use of Kubernetes for container orchestration. It describes Adform's Kubernetes clusters including sandbox, development, staging, and production clusters. It discusses why Adform uses containers and Kubernetes for benefits like build locally-run anywhere, immutable infrastructure, and improved workflow. The presentation also covers technologies around Kubernetes like monitoring and provides a live demo of Adform's self-service portal and deployment workflow.
TechTalk Webinar Series - Getting Started with Apache OpenWhisk
1) Apache OpenWhisk is an open source serverless computing platform that allows executing code in response to events from web/mobile apps without provisioning servers or resources.
2) OpenWhisk uses a programming model where services define triggers as events emitted, developers associate actions to handle the events via rules, and the system handles executing the code.
3) A demo was presented exploring OpenWhisk through IBM Cloud Functions, a hosted service on IBM Bluemix that allows running functions in response to various events from databases, object storage, and other triggers.
AWS Community Day Bangkok 2019 - How AWS Parallel Cluster can accelerate high...
AWS Community Day Bangkok 2019 - How AWS Parallel Cluster can accelerate high...AWS User Group - Thailand
(1) AWS Parallel Cluster can accelerate high performance scientific computing by providing flexible compute clusters in AWS that are optimized for HPC workloads.
(2) Case studies show how computational chemistry and deep learning for drug discovery benefit from HPC resources for tasks like molecular simulations and property predictions.
(3) AWS offers compute instances, storage, networking and other services for building HPC clusters, along with tools like Parallel Cluster that simplify setup and management of these clusters for scientific applications.OpenShift Meetup - Red Hat OpenShift Container Storage explained
OpenShift Meetup - Red Hat OpenShift Container Storage explainedConSol Consulting & Solutions Software GmbH
Red Hat OpenShift Container Storage (RHOCS) is software-defined storage optimized for container environments that addresses challenges around platform services, application portability, and persistence. It uses Red Hat Ceph Storage, the Rook.io operator, and NooBaa's Multicloud Object Gateway technology. RHOCS provides simplified deployment and management of storage and allows applications and storage to run on the same or different nodes in an OpenShift cluster, or to connect multiple OpenShift clusters to an external Ceph storage cluster. The presentation demonstrated a proof-of-concept using an external Red Hat Ceph storage cluster with OpenShift Container Platform 4.6.Tensorflow London 13: Barbara Fusinska 'Hassle Free, Scalable, Machine Learni...
Speaker: Barbara Fusinska, Machine Learning Strategic Cloud Engineer at Google
Title: Hassle Free, Scalable, Machine Learning with Kubeflow
Abstract: Kubeflow uses Kubernetes strengths to build a toolkit for data scientists where they can create, train and publish the models in a hassle-free and scalable way. The goal is to run machine learning workflow without a need to think about the infrastructure. In this talk, Barbara will discuss the capabilities of Kubeflow from the data scientist perspective. The presentation will introduce how you can use the platform to build the models and deploy it adjusting the computation environment.
Bio: Barbara is a Machine Learning Strategic Cloud Engineer at Google with strong software development background. While working with a variety of different companies, she gained experience in building diverse software systems. This experience brought her focus to the Data Science and Big Data field. She believes in the importance of the data and metrics when growing a successful business. Alongside collaborating around data architectures, Barbara still enjoys programming activities. Currently speaking at conferences in-between working in London. She tweets at @BasiaFusinska and you can follow her blog.
Thanks to all TensorFlow London meetup organisers and supporters:
Seldon.io
Altoros
Rewired
Google Developers
Rise London
Brendon Foxen (Channel 4) - Speeding up Software Delivery at Channel 4
A look into why Channel 4 needed to speed up the software delivery pipeline and how a Micro services architecture, a CI/CD approach and Docker is helping to make that happen.
Video: Coming Soon
Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London
Follow DOXLON on twitter http://www.twitter.com/doxlon
stackconf 2021 | How we finally migrated an eCommerce-Platform to GCP
As Squad Architect Platform I supported the platform-team to migrate a complete ecommerce-environment to Google Cloud Platform. By sketching out various migration-steps, technical concepts and tooling I will explain we did the migration exactly this way.
DevOps in real life
Александр Снеговой, DevOps Software Engineer в DataArt Kherson. Более шести лет в IT. Сертифицированный AWS Solutions Architect Associate. Докладчик на международных научных конференциях. Религиозный фанат Docker.
Презентация:
1. Докеризация приложения.
2. Настройка CI/CD.
3. Развертывание инфраструктуры в AWS с помощью Terraform.
Shift Remote AI: Build and deploy PyTorch Models with Azure Machine Learning ...
Take a deep look into Azure Machine Learning, a cloud service that helps you build, train, deploy, and manage models. Walk through the data science process and then have some fun creating a ML recognition model based on the Simpsons cartoon with PyTorch. You'll leave this session with a better grasp of the technological components of Azure Machine Learning services.
Apache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
Kubernetes is a fast growing open-source platform which provides container-centric infrastructure. Conceived by Google in 2014, and leveraging over a decade of experience running containers at scale internally, it is one of the fastest moving projects on GitHub with 1000+ contributors and 40,000+ commits. Kubernetes has first class support on Google Cloud Platform, Amazon Web Services, and Microsoft Azure.
Unlike YARN, Kubernetes started as a general purpose orchestration framework with a focus on serving jobs. Support for long-running, data intensive batch workloads required some careful design decisions. Engineers across several organizations have been working on Kubernetes support as a cluster scheduler backend within Spark. During this process, we encountered several challenges in translating Spark considerations into idiomatic Kubernetes constructs. In this talk, we describe the challenges and the ways in which we solved them. This talk will be technical and is aimed at people who are looking to run Spark effectively on their clusters. The talk assumes basic familiarity with cluster orchestration and containers.
[Spark Summit 2017 NA] Apache Spark on Kubernetes
This document summarizes a presentation about running Apache Spark on Kubernetes. It discusses how Spark jobs can be scheduled and run on Kubernetes, including scheduling the driver and executor pods. Key points of the design include the Kubernetes scheduler backend for Spark and components like the file staging server. The roadmap outlines upcoming support for features like Spark Streaming and improvements to dynamic allocation.
More Related Content
What's hot
Apache Airflow
Introductory talk on Apache Airflow (Incubator) by Sumit Maheshwari at recent Bangalore Big Data Meetup.
Machine Learning Infrastructure
Machine learning infrastructure solve data scientists' problems using infrastructure tools. This talk shows the case study of building SigOpt Orchestrate, an ML infrastructure tool. The talk highlights how data scientists' concerns as user mapped to solutions with some of today's most popular infrastructure tools.
To learn more about SigOpt Orchestrate: https://sigopt.com/orchestrate
Originally given as a talk for UC Berkeley's Women in Electrical Engineering and Computer Science group on January 24, 2019.
Matt Chung (Independent) - Serverless application with AWS Lambda
The talk will focus on how we are utilizing AWS Lambda for certain applications and the advantages/disadvantages, and the challenges we discovered along the way. It would help those who are looking to reduce technical debt with the infrastructure and costs.
Previously a Director of technical operations at fox networks (21st Century Fox/News Corporation) responsible for infrastructure and building deployment pipelines. Currently a Python programmer / DevOps engineer with roots in systems/networks administration. Focus is on infrastructure and application automation. Worked as an engineer for Cisco Systems with emphasis on video conferencing. Built microwave networks at Bel Air Internet. Find me on github and twitter @itsmemattchung
Video: https://www.youtube.com/watch?v=BLcElBUhfrQ
Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London
Follow DOXLON on twitter http://www.twitter.com/doxlon
Knative, Serverless on Kubernetes, and Openshift
This document provides an overview of serverless computing and Knative. It begins with defining serverless computing and distinguishing it from Function as a Service (FaaS). It then provides a high-level overview of the main components of Knative, including Knative Serving for building and deploying serverless applications and Knative Eventing for connecting those applications. It also introduces OpenShift Serverless, which is Red Hat's implementation of Knative on OpenShift. The document includes several demos and labs for hands-on experience with Knative and serverless concepts.
Nextflow and AWS Batch - GCC/BOSC 2018
This document discusses using Nextflow and AWS Batch for bioinformatics analysis. It describes how Nextflow provides orchestration, parallelization, scalability, portability, and reproducibility capabilities. AWS Batch automates dynamic computing clusters and optimizes EC2 instance types used for jobs. The document shows how Nextflow can use AWS Batch as an executor, with Nextflow automatically converting tasks to AWS Batch jobs and managing input/output data transfer with S3. It shares the author's experience using Nextflow and AWS Batch together at their company, noting benefits of scalability and reproducibility as well as some optimization opportunities for large-scale workflows.
Hydrosphere.io for ODSC: Webinar on Kubeflow
Webinar video: https://www.youtube.com/watch?v=Y3_fcJBgpMw
Kubeflow and Beyond: Automation of Model Training, Deployment, Testing, Monitoring, and Retraining
Speakers:
Stepan Pushkarev, CTO, Hydrosphere.io and Ilnur Garifullin is an ML Engineer, Hydrosphere.io
Abstract: Very often a workflow of training models and delivering them to the production environment contains loads of manual work. Those could be either building a Docker image and deploying it to the Kubernetes cluster or packing the model to the Python package and installing it to your Python application. Or even changing your Java classes with the defined weights and re-compiling the whole project. Not to mention that all of this should be followed by testing your model's performance. It hardly could be named "continuous delivery" if you do it all manually. Imagine you could run the whole process of assembling/training/deploying/testing/running model via a single command in your terminal. In this webinar, we will present a way to build the whole workflow of data gathering/model training/model deployment/model testing into a single flow and run it with a single command.
Java & Microservices in Azure
Deploy & Run your Java Application on Azure and Azure Stack, in a hybrid environment spanning public and private clouds.
Apache Airflow Architecture
Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.
Cloudsolutionday 2016: Docker & FAAS at getvero.com
This document discusses Docker and cloud functions at Vero. It provides details on:
1. How Vero uses containers and Docker to run over 20 kinds of worker functions, handling up to 16 million emails per day and tracking customer actions.
2. How Vero built its own cloud function service to have more control over scaling and security compared to other cloud providers' services.
3. Challenges faced including startup latency initially and horizontal scaling, and how Vero overcame these challenges through auto-scaling and other techniques.
Serverless spark
Building and deploying an analytic service on Cloud is a challenge. A bigger challenge is to maintain the service. In a world where users are gravitating towards a model where cluster instances are to provisioned on the fly, in order for these to be used for analytics or other purposes, and then to have these cluster instances shut down when the jobs get done, the relevance of containers and container orchestration is more important than ever. In short Customers are looking for Serverless Spark Clusters. The Intent of this presentation is to share what is Serverless Spark and what are the benefits of running Spark in serverless manner.
DevOps for Databricks
Applying DevOps to Databricks can be a daunting task. In this talk this will be broken down into bite size chunks. Common DevOps subject areas will be covered, including CI/CD (Continuous Integration/Continuous Deployment), IAC (Infrastructure as Code) and Build Agents.
We will explore how to apply DevOps to Databricks (in Azure), primarily using Azure DevOps tooling. As a lot of Spark/Databricks users are Python users, will will focus on the Databricks Rest API (using Python) to perform our tasks.
Kubernetes in Adform
The document discusses Adform's use of Kubernetes for container orchestration. It describes Adform's Kubernetes clusters including sandbox, development, staging, and production clusters. It discusses why Adform uses containers and Kubernetes for benefits like build locally-run anywhere, immutable infrastructure, and improved workflow. The presentation also covers technologies around Kubernetes like monitoring and provides a live demo of Adform's self-service portal and deployment workflow.
TechTalk Webinar Series - Getting Started with Apache OpenWhisk
1) Apache OpenWhisk is an open source serverless computing platform that allows executing code in response to events from web/mobile apps without provisioning servers or resources.
2) OpenWhisk uses a programming model where services define triggers as events emitted, developers associate actions to handle the events via rules, and the system handles executing the code.
3) A demo was presented exploring OpenWhisk through IBM Cloud Functions, a hosted service on IBM Bluemix that allows running functions in response to various events from databases, object storage, and other triggers.
AWS Community Day Bangkok 2019 - How AWS Parallel Cluster can accelerate high...
AWS Community Day Bangkok 2019 - How AWS Parallel Cluster can accelerate high...AWS User Group - Thailand
(1) AWS Parallel Cluster can accelerate high performance scientific computing by providing flexible compute clusters in AWS that are optimized for HPC workloads.
(2) Case studies show how computational chemistry and deep learning for drug discovery benefit from HPC resources for tasks like molecular simulations and property predictions.
(3) AWS offers compute instances, storage, networking and other services for building HPC clusters, along with tools like Parallel Cluster that simplify setup and management of these clusters for scientific applications.OpenShift Meetup - Red Hat OpenShift Container Storage explained
OpenShift Meetup - Red Hat OpenShift Container Storage explainedConSol Consulting & Solutions Software GmbH
Red Hat OpenShift Container Storage (RHOCS) is software-defined storage optimized for container environments that addresses challenges around platform services, application portability, and persistence. It uses Red Hat Ceph Storage, the Rook.io operator, and NooBaa's Multicloud Object Gateway technology. RHOCS provides simplified deployment and management of storage and allows applications and storage to run on the same or different nodes in an OpenShift cluster, or to connect multiple OpenShift clusters to an external Ceph storage cluster. The presentation demonstrated a proof-of-concept using an external Red Hat Ceph storage cluster with OpenShift Container Platform 4.6.Tensorflow London 13: Barbara Fusinska 'Hassle Free, Scalable, Machine Learni...
Speaker: Barbara Fusinska, Machine Learning Strategic Cloud Engineer at Google
Title: Hassle Free, Scalable, Machine Learning with Kubeflow
Abstract: Kubeflow uses Kubernetes strengths to build a toolkit for data scientists where they can create, train and publish the models in a hassle-free and scalable way. The goal is to run machine learning workflow without a need to think about the infrastructure. In this talk, Barbara will discuss the capabilities of Kubeflow from the data scientist perspective. The presentation will introduce how you can use the platform to build the models and deploy it adjusting the computation environment.
Bio: Barbara is a Machine Learning Strategic Cloud Engineer at Google with strong software development background. While working with a variety of different companies, she gained experience in building diverse software systems. This experience brought her focus to the Data Science and Big Data field. She believes in the importance of the data and metrics when growing a successful business. Alongside collaborating around data architectures, Barbara still enjoys programming activities. Currently speaking at conferences in-between working in London. She tweets at @BasiaFusinska and you can follow her blog.
Thanks to all TensorFlow London meetup organisers and supporters:
Seldon.io
Altoros
Rewired
Google Developers
Rise London
Brendon Foxen (Channel 4) - Speeding up Software Delivery at Channel 4
A look into why Channel 4 needed to speed up the software delivery pipeline and how a Micro services architecture, a CI/CD approach and Docker is helping to make that happen.
Video: Coming Soon
Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London
Follow DOXLON on twitter http://www.twitter.com/doxlon
stackconf 2021 | How we finally migrated an eCommerce-Platform to GCP
As Squad Architect Platform I supported the platform-team to migrate a complete ecommerce-environment to Google Cloud Platform. By sketching out various migration-steps, technical concepts and tooling I will explain we did the migration exactly this way.
DevOps in real life
Александр Снеговой, DevOps Software Engineer в DataArt Kherson. Более шести лет в IT. Сертифицированный AWS Solutions Architect Associate. Докладчик на международных научных конференциях. Религиозный фанат Docker.
Презентация:
1. Докеризация приложения.
2. Настройка CI/CD.
3. Развертывание инфраструктуры в AWS с помощью Terraform.
Shift Remote AI: Build and deploy PyTorch Models with Azure Machine Learning ...
Take a deep look into Azure Machine Learning, a cloud service that helps you build, train, deploy, and manage models. Walk through the data science process and then have some fun creating a ML recognition model based on the Simpsons cartoon with PyTorch. You'll leave this session with a better grasp of the technological components of Azure Machine Learning services.
What's hot (20)
Matt Chung (Independent) - Serverless application with AWS Lambda
Matt Chung (Independent) - Serverless application with AWS Lambda
Cloudsolutionday 2016: Docker & FAAS at getvero.com
Cloudsolutionday 2016: Docker & FAAS at getvero.com
TechTalk Webinar Series - Getting Started with Apache OpenWhisk
TechTalk Webinar Series - Getting Started with Apache OpenWhisk
AWS Community Day Bangkok 2019 - How AWS Parallel Cluster can accelerate high...
AWS Community Day Bangkok 2019 - How AWS Parallel Cluster can accelerate high...
OpenShift Meetup - Red Hat OpenShift Container Storage explained
OpenShift Meetup - Red Hat OpenShift Container Storage explained
Tensorflow London 13: Barbara Fusinska 'Hassle Free, Scalable, Machine Learni...
Tensorflow London 13: Barbara Fusinska 'Hassle Free, Scalable, Machine Learni...
Brendon Foxen (Channel 4) - Speeding up Software Delivery at Channel 4
Brendon Foxen (Channel 4) - Speeding up Software Delivery at Channel 4
stackconf 2021 | How we finally migrated an eCommerce-Platform to GCP
stackconf 2021 | How we finally migrated an eCommerce-Platform to GCP
Shift Remote AI: Build and deploy PyTorch Models with Azure Machine Learning ...
Shift Remote AI: Build and deploy PyTorch Models with Azure Machine Learning ...
Similar to Webinar kubernetes and-spark
Apache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
Kubernetes is a fast growing open-source platform which provides container-centric infrastructure. Conceived by Google in 2014, and leveraging over a decade of experience running containers at scale internally, it is one of the fastest moving projects on GitHub with 1000+ contributors and 40,000+ commits. Kubernetes has first class support on Google Cloud Platform, Amazon Web Services, and Microsoft Azure.
Unlike YARN, Kubernetes started as a general purpose orchestration framework with a focus on serving jobs. Support for long-running, data intensive batch workloads required some careful design decisions. Engineers across several organizations have been working on Kubernetes support as a cluster scheduler backend within Spark. During this process, we encountered several challenges in translating Spark considerations into idiomatic Kubernetes constructs. In this talk, we describe the challenges and the ways in which we solved them. This talk will be technical and is aimed at people who are looking to run Spark effectively on their clusters. The talk assumes basic familiarity with cluster orchestration and containers.
[Spark Summit 2017 NA] Apache Spark on Kubernetes
This document summarizes a presentation about running Apache Spark on Kubernetes. It discusses how Spark jobs can be scheduled and run on Kubernetes, including scheduling the driver and executor pods. Key points of the design include the Kubernetes scheduler backend for Spark and components like the file staging server. The roadmap outlines upcoming support for features like Spark Streaming and improvements to dynamic allocation.
Containerized architectures for deep learning
This document discusses containerized architectures for deep learning using Kubernetes and Kubeflow. It motivates the use of machine learning pipelines for improving data wrangling, system unification, composability, complexity management, portability, and model quality. It then summarizes popular machine learning pipeline tools like Apache Airflow, Kubeflow, TensorFlow Extended, and MLflow. It introduces Kubernetes and containers, and argues that Kubernetes and Kubeflow can provide composability, portability, and scalability for machine learning workloads. It demonstrates a deep learning model deployed on Kubernetes with Kubeflow through a doppelganger image similarity search app.
Innovating faster with SBT, Continuous Delivery, and LXC
A case study of the tools and techniques used at Gilt Groupe to develop and deploy a system composed of over 200 micro-services.
Containers, Serverless and Functions in a nutshell
This document provides an overview of containers, microservices, Docker, Kubernetes, serverless computing, and functions. It discusses how containers package software for distribution and are more lightweight than virtual machines. Microservices decompose monolithic applications into loosely coupled services. Docker is a popular container platform, while Kubernetes is an open source orchestration system for containers. Serverless computing focuses on writing code without managing infrastructure, using functions as units of work. Functions are stateless and triggered by events. Platforms like AWS Lambda, Azure Functions, Fn, and OpenFaaS support serverless development.
Building Efficient Parallel Testing Platforms with Docker
We often use containers to maintain parity across development, testing, and production environments, but we can also use containerization to significantly reduce time needed for testing by spinning up multiple instances of fully isolated testing environments and executing tests in parallel. This strategy also helps you maximize the utilization of infrastructure resources. The enhanced toolset provided by Docker makes this process simple and unobtrusive, and you’ll see how Docker Engine, Registry, and Compose can work together to make your tests fast.
Running Spark Inside Containers with Haohai Ma and Khalid Ahmed
This presentation describes the journey we went through in containerizing Spark workload into multiple elastic Spark clusters in a multi-tenant kubernetes environment. Initially we deployed Spark binaries onto a host-level filesystem, and then the Spark drivers, executors and master can transparently migrate to run inside a Docker container by automatically mounting host-level volumes. In this environment, we do not need to prepare a specific Spark image in order to run Spark workload in containers. We then utilized Kubernetes helm charts to deploy a Spark cluster. The administrator could further create a Spark instance group for each tenant. A Spark instance group, which is akin to the Spark notion of a tenant, is logically an independent kingdom for a tenant’s Spark applications in which they own dedicated Spark masters, history server, shuffle service and notebooks. Once a Spark instance group is created, it automatically generates its image and commits to a specified repository. Meanwhile, from Kubernetes’ perspective, each Spark instance group is a first-class deployment and thus the administrator can scale up/down its size according to the tenant’s SLA and demand. In a cloud-based data center, each Spark cluster can provide a Spark as a service while sharing the Kubernetes cluster. Each tenant that is registered into the service gets a fully isolated Spark instance group. In an on-prem Kubernetes cluster, each Spark cluster can map to a Business Unit, and thus each user in the BU can get a dedicated Spark instance group. The next step on this journey will address the resource sharing across Spark instance groups by leveraging new Kubernetes’ features (Kubernetes31068/9), as well as the Elastic workload containers depending on job demands (Spark18278). Demo: https://www.youtube.com/watch?v=eFYu6o3-Ea4&t=5s
Adf with docker
This document discusses developing, delivering, and running Oracle ADF applications with Docker containers. It provides an overview of using containers and Docker to build application images, deploy them to Kubernetes clusters in the cloud, and set up continuous delivery pipelines for automated testing and deployment. Sample applications are packaged into Docker containers along with required dependencies. Kubernetes is used to orchestrate and manage container deployments across different environments.
AI & Machine Learning Pipelines with Knative
The document discusses the need for Knative to build cloud-native AI platforms. It describes that an AI lifecycle involves multiple iterative phases like data preparation, model training, deployment, and monitoring. It states that Kubernetes alone is not sufficient and that concepts like building, serving, eventing and pipelines are required to automate the end-to-end AI workflow. It introduces Knative as a set of building blocks on top of Kubernetes that provide these capabilities through custom resource definitions. Specifically, Knative provides capabilities for source-to-container builds, event delivery and subscription, request-driven scalable serving of models, and configuration of CI/CD-style pipelines for Kubernetes applications.
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...
Devops engineers have applied a great deal of creativity and energy to invent tools that automate infrastructure management, in the service of deploying capable and functional applications. For data-driven applications running on Apache Spark, the details of instantiating and managing the backing Spark cluster can be a distraction from focusing on the application logic. In the spirit of devops, automating Spark cluster management tasks allows engineers to focus their attention on application code that provides value to end-users.
Using Openshift Origin as a laboratory, we implemented a platform where Apache Spark applications create their own clusters and then dynamically manage their own scale via host-platform APIs. This makes it possible to launch a fully elastic Spark application with little more than the click of a button.
We will present a live demo of turn-key deployment for elastic Apache Spark applications, and share what we’ve learned about developing Spark applications that manage their own resources dynamically with platform APIs.
The audience for this talk will be anyone looking for ways to streamline their Apache Spark cluster management, reduce the workload for Spark application deployment, or create self-scaling elastic applications. Attendees can expect to learn about leveraging APIs in the Kubernetes ecosystem that enable application deployments to manipulate their own scale elastically.
Why Kubernetes as a container orchestrator is a right choice for running spar...
Building and deploying an analytic service on Cloud is a challenge. A bigger challenge is to maintain the service. In a world where users are gravitating towards a model where cluster instances are to be provisioned on the fly, in order for these to be used for analytics or other purposes, and then to have these cluster instances shut down when the jobs get done, the relevance of containers and container orchestration is more important than ever.
Container orchestrators like Kubernetes can be used to deploy and distribute modules quickly, easily, and reliably. The intent of this talk is to share the experience of building such a service and deploying it on a Kubernetes cluster. In this talk, we will discuss all the requirements which an enterprise grade Hadoop/Spark cluster running on containers bring in for a container orchestrator.
This talk will cover in details how Kubernetes orchestrator can be used to meet all our needs of resource management, scheduling, networking, and network isolation, volume management, etc. We will discuss how we have replaced our home grown container orchestrator with Kubernetes which used to manage the container lifecycle and manage resources in accordance to our requirements. We will also discuss the feature list as container orchestrator which is helping us deploy and patch 1000s of containers and also a list which we believe need improvement or can be enhanced in a container orchestrator.
Speaker
Rachit Arora, SSE, IBM
Kubernetes 101 Workshop
This document provides an overview and agenda for a hands-on Kubernetes 101 workshop hosted by Vishal Biyani from InfraCloud technologies. The 180-minute workshop includes introductions, conceptual overviews of Kubernetes components like pods and deployments, demonstrations of setting up a Kubernetes cluster on Google Cloud and the local machine, and hands-on labs for working with pods, deployments, services, secrets and Helm. The goal is to help attendees gain practical experience with common Kubernetes patterns, architectures, and tools.
Container Conf 2017: Rancher Kubernetes
Deploy, manage and scale Kubernetes with Rancher - a session given in ContainerConf 2017 (7th April) in Bangalore
Container management with docker & kubernetes
Kasun Rajapakse provides an overview of container management with Azure Kubernetes Service (AKS). AKS is a managed Kubernetes service provided by Azure that makes it easy to get started with enterprise-scale container environments. Key points include:
- Containers are more lightweight than VMs and share the host OS kernel for efficiency. Docker is a tool for building and running containers.
- Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications. It was originally developed by Google.
- AKS removes the need to manage the Kubernetes master nodes, allowing users to focus on containerized applications. It provides automated upgrades, scaling
CNCF Projects Overview
This document discusses several open source projects from the Cloud Native Computing Foundation (CNCF). The CNCF aims to accelerate adoption of containers, microservices, and cloud native applications. It hosts projects like Containerd, Kubernetes, CoreDNS, Linkerd, Prometheus, Fluentd, OpenTracing, and others. Containerd and rkt are container runtimes. Kubernetes is an orchestration system. CNCI, CoreDNS, and Linkerd provide container networking, DNS/service discovery, and service mesh capabilities. Prometheus, Fluentd, OpenTracing provide monitoring, logging, and distributed tracing.
How kubernetes operators can rescue dev secops in midst of a pandemic updated
This document discusses how Kubernetes operators can help automate DevSecOps processes. It begins by explaining why organizations adopt containers and Kubernetes. It then discusses the challenges of managing containerized workloads at scale and how Kubernetes operators can provide orchestration and management. It provides an overview of what operators are, how the Operator Framework works, and the phases of building an operator. It demonstrates building a sample memcached operator in Golang using the Operator SDK tools. Finally, it discusses different options for installing operators like Helm, Ansible, and custom operators and provides some useful links for learning more.
Getting Started with OpenStack, Red Hat Summit 2016
Hearing a lot about OpenStack and want to check it out for yourself? See how quick and easy it is to install and start using OpenStack using containers running within a VM on your own laptop. Familiarize yourself with the environment. Learn to use the Horizon (GUI) and the CLI to view and operate an OpenStack cloud, both as a cloud administrator as well as a tenant/user of the cloud.
Spark on Kubernetes
Spark can run on Kubernetes containers in two ways - as a static cluster or with native integration. As a static cluster, Spark pods are manually deployed without autoscaling. Native integration treats Kubernetes as a resource manager, allowing Spark to dynamically acquire and release containers like in YARN. It uses Kubernetes custom controllers to create driver pods that then launch worker pods. This provides autoscaling of resources based on job demands.
How to monitor your ML models in production with Kubernetes
Once a model is deployed, you have a responsibility to ensure its reliability and performance in production. That means that in addition to system monitoring, you should be checking and monitoring its ML health and vitals such as accuracy, bias, and variance as new data comes in. In this online workshop we’ll discuss how to build a system to monitor your machine learning model in production on Kubernetes. You’ll learn to keep track of different models and their model performance over time, and how to set up custom alerts for your models. We’ll discuss what types of variants to monitor, and how to measure its performance. Join CTO of cnvrg.io, Leah Kolben in this hands-on workshop on critical practices for monitoring your machine learning models in production. Using the power of Kubernetes, we’ll build a complete system for model tracking that ensures high performing models in production.
Watch the full presentation with video and audio here: https://info.cnvrg.io/monitor-machine-learning-model-workshop
What you’ll learn:
- Why we monitor models in production
- The critical vitals to track and monitor performance
- How to set up automated alerts
- How to set up Kubernetes for monitoring
- Use tools like Grafana and Kibana to monitor and visualize your system and ML health
To watch the full live presentation click here: https://info.cnvrg.io/monitor-machine-learning-model-workshop
DevJam 2019 - Introduction to Kubernetes
Introduction talk from Alejandro Galue about Kubernetes and how to run OpenNMS services on Kubernetes based platforms.
Similar to Webinar kubernetes and-spark (20)
Apache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
Apache Spark on Kubernetes Anirudh Ramanathan and Tim Chen
Innovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXC
Containers, Serverless and Functions in a nutshell
Containers, Serverless and Functions in a nutshell
Building Efficient Parallel Testing Platforms with Docker
Building Efficient Parallel Testing Platforms with Docker
Running Spark Inside Containers with Haohai Ma and Khalid Ahmed
Running Spark Inside Containers with Haohai Ma and Khalid Ahmed
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...
Teaching Apache Spark Clusters to Manage Their Workers Elastically: Spark Sum...
Why Kubernetes as a container orchestrator is a right choice for running spar...
Why Kubernetes as a container orchestrator is a right choice for running spar...
How kubernetes operators can rescue dev secops in midst of a pandemic updated
How kubernetes operators can rescue dev secops in midst of a pandemic updated
Getting Started with OpenStack, Red Hat Summit 2016
Getting Started with OpenStack, Red Hat Summit 2016
How to monitor your ML models in production with Kubernetes
How to monitor your ML models in production with Kubernetes
More from cnvrg.io AI OS - Hands-on ML Workshops
CI/CD for Machine Learning
CI/CD (Continuous Integration/ Continuous Deployment) has long been a successful process for most software applications. The same can be done with Machine Learning applications, offering an automated and continuous training and continuous deployment of machine learning models. Using CI/CD for machine learning applications creates a truly end-to-end pipeline that closes the feedback loop at every step of the way, and maintains high performing ML models. It can also bridge science and engineering tasks, causing less friction from data, to modeling, to production and back again. Join CEO of cnvrg.io Yochay Ettun as he brings you through how to create a CI/CD pipeline for machine learning, and set up continuous deployment in just one click. With a depth of knowledge in all the latest research, Yochay will share with you today's top methods for applying CI/CD to machine learning.
Webinar takeaways:
Configure and execute continuous training and continuous deployment for ML
Define dependencies and triggers
Automatically connect data pipeline, machine learning pipeline and deployment pipelines
Integrate model bias detection or fairness and accuracy validations
Build monitoring infrastructure to close the data feedback loop
Collect live data for improved model performance
Watch all our webinars at https://cnvrg.io/webinars-and-workshops/
How to use continual learning in your ML models
In this webinar, data science expert and CEO of cnvrg.io Yochay Ettun discusses continual learning in production. This webinar examines continual learning, and will help you apply continual learning into your production models using tools like Tensorflow, Kubernetes, and cnvrg.io. This webinar for professional data scientists will go over how to monitor models when in production, and how to set up automatically adaptive machine learning.
Key webinar takeaways:
Understanding of continual learning
Optimizing your models for accuracy with continual learning
How to use TensorFlow, Kubernetes and cnvrg.io to apply CL to your models
How you can build automatically adaptive machine learning
Adapting to shifting data distributions
Coping with outliers
Retraining in production
Adapting to new tasks
A/B test your models
Deploying your machine learning pipeline to production
Watch all our webinars at https://cnvrg.io/webinars-and-workshops/
How To Build Auto-Adaptive Machine Learning Models with Kubernetes
How To Build Auto-Adaptive Machine Learning Models with Kubernetescnvrg.io AI OS - Hands-on ML Workshops
This webinar will instruct data scientists and machine learning engineers how to build manage and deploy auto-adaptive machine learning models in production. Data is ever changing, leaving your models outdated and built on old data. This can lead to underperforming models and a lot of manual work to fix it. By allowing your models to continually learn you’ll ensure that they run at peak performance. Using state of the art Kubernetes infrastructure, we’ll show you how to automatically track and manage your auto-adaptive machine learning models while in production. By building auto-adaptive machine learning models, data engineers can bridge the gap between research and production. After this webinar you’ll be able to build and deploy machine learning pipelines that automatically adapt and retrain based on any validation trigger you choose.
Key webinar takeaways:
How to build auto-adaptive machine learning pipelines
How to use Kubernetes to manage and scale models in production
How to automatically monitor for peak performance
How to set up continuous deployment of ML pipeline
Watch all our webinars at https://cnvrg.io/webinars-and-workshops/MLOps for production-level machine learning
MLOps (a compound of “machine learning” and “operations”) is a practice for collaboration and communication between data scientists and operations professionals to help manage the production machine learning lifecycle. Similar to the DevOps term in the software development world, MLOps looks to increase automation and improve the quality of production ML while also focusing on business and regulatory requirements. MLOps applies to the entire ML lifecycle - from integrating with model generation (software development lifecycle, continuous integration/continuous delivery), orchestration, and deployment, to health, diagnostics, governance, and business metrics.
To watch the full presentation click here: https://info.cnvrg.io/mlopsformachinelearning
In this webinar, we’ll discuss core practices in MLOps that will help data science teams scale to the enterprise level. You’ll learn the primary functions of MLOps, and what tasks are suggested to accelerate your teams machine learning pipeline. Join us in a discussion with cnvrg.io Solutions Architect, Aaron Schneider, and learn how teams use MLOps for more productive machine learning workflows.
- Reduce friction between science and engineering
- Deploy your models to production faster
- Health, diagnostics and governance of ML models
- Kubernetes as a core platform for MLOps
- Support advanced use-cases like continual learning with MLOps
Continual learning with human in-the-loop
Continual learning allows machine learning models to learn new tasks over time without forgetting previous ones, improving performance and efficiency. It supports large-scale data processing with in-memory computing for faster applications. Machine learning pipelines with continual learning can include automated model deployment, hyperparameter optimization, retraining triggers, and labeling at scale.
Build machine learning pipelines from research to production
Streamlining your machine learning pipeline is critical for enterprise data science to deliver better business results. Accelerating the process from data, to processing to training to deployment and back again will help you get better performing models, faster. Watch the full presentation with audio and video here: https://info.cnvrg.io/build-machine-learning-pipelines
This presentation will offer solutions to the common challenges data scientists and data engineers face when building a machine learning pipeline.
We will dissect each part of the pipeline and offer strategies on how to design your machine learning pipelines for a more efficient, integrated and automated process. We’ll tackle ways to connect all your data sourcing in one unified location. How to create modular ML components for easy reproducibility, and automate MLOps for quick training of models and hyperparameter optimization. Streamline frequent deployment of models leveraging the power of Kubernetes. And lastly, you’ll learn to design a monitoring toolkit with Grafana and Kibana for easy CI/CD. Join Solutions Architect, Aaron Schneider as he builds and end-to-end machine learning pipeline, and explains how to optimize each part for a more efficient workflow.
Key webinar takeaways:
- Set up an efficient machine learning pipeline
- Learn key MLOps solutions streamlining science and engineering
- Create reusable ML components
- Build a suite of monitoring and visualization tools
- Instantly train and deploy ML models with Kubernetes
- Use CI/CD to design an auto-adaptive machine learning pipeline
Watch the full presentation here: https://info.cnvrg.io/build-machine-learning-pipelines
Why more than half of ML models don't make it to production
As the commercial world accelerates investment into AI and Machine Learning one theme continually appears. Models are being built, but they are not being used. Teams of Data Scientists around the world are training versatile models but due to managerial, logistical and infrastructural problems, these models are not making it to production.
To watch the full presentation with visual and audio click here: https://info.cnvrg.io/ml-models-to-production
In this webinar, Solutions Architect Aaron Schneider will diagnose the problem and identify the symptoms. He’ll explain how reproducibility, scalability and collaboration can increase the gap between research and production. The webinar will examine best practices for building a machine learning pipeline that enables quick iteration, deployment and CI/CD to ensure that your company is deploying and maintaining the best services for you customers and clients.
Key takeaways:
- The common issues that block deployment and increase time to production
- How different stakeholders can resolve key issues
- How to accelerate from research to production
Tools that can make productionizing models easy
- Leveraging Kubernetes and container-based architecture for faster deployment
Watch the full presentation here: https://info.cnvrg.io/ml-models-to-production
Training Machine Learning models directly from GitHub with cnvrg.io MLOps
Training Machine Learning models directly from GitHub with cnvrg.io MLOpscnvrg.io AI OS - Hands-on ML Workshops
To watch the full presentation with video and audio click here: https://cnvrg.io/webinars-and-workshops/how-to-train-ml-models-directly-from-github/
Join our hands-on workshop series and deep dive into machine learning and deep learning with cnvrg.io ML experts. Each session cnvrg.io will guide you through a hands on ML project using cnvrg.io CORE – a free ML platform we released for the data science community – and tools and frameworks like TensorFlow, Spark on Kubernetes, GitHub and more to help you build and deploy high impact machine learning models.
This session is about how to deploy your machine learning model directly with GitHub.
To watch the full presentation with video and audio click here: https://cnvrg.io/webinars-and-workshops/how-to-train-ml-models-directly-from-github/ Scaling MLOps on NVIDIA DGX Systems
Developing, experimenting, and deploying ML models at scale requires substantial tooling, scripting, tracking, versioning, and monitoring.
Watch full video here: https://cnvrg.io/webinars-and-workshops/scaling-mlops-on-nvidia-dgx-systems/
Data scientists want to do data science – and are slowed down by MLOps and DevOps tasks.
They lack user friendly tools needed to track experiments, attach resources, manage datasets and launch multiple ML pipelines.
In this presentation cnvrg.io CEO, Yochay Ettun will host a special guest from NVIDIA, Sr. Product Manager for NVIDIA DGX systems, Michael Balint, and discuss how to optimize the use of any NVIDIA DGX and NVIDIA GPU asset both on-prem or in the cloud with the cnvrg.io machine learning platform.
We will show best practices to reach high utilization of NVIDIA DGX systems, while conducting meta-scheduling across multiple heterogeneous Kubernetes/OpenShift/Linux server clusters.
In addition, we will introduce the concept of production flows, which automate hundreds of models from the data hub to deployment. We will wrap up with a real-life demo of flows, exercising many experiments across DGX platforms.
What you will learn:
- Creating a data science flow: from data to deployment, while attaching different NVIDIA DGX Kubernetes clusters to each step of the flow
- The concept of meta-scheduler: scheduling experiments disperse resources or other schedulers, accomplishing high utilization at scale
- How the NVIDIA DGX ecosystem with cnvrg.io makes GPU assets consumed easily, with one-click, bypassing complexity of MLOps
- How to leverage NGC containers in ML pipelines
You can watch the full presentation along with audio and video in the link here: https://cnvrg.io/webinars-and-workshops/scaling-mlops-on-nvidia-dgx-systems/
More from cnvrg.io AI OS - Hands-on ML Workshops (9)
How To Build Auto-Adaptive Machine Learning Models with Kubernetes
How To Build Auto-Adaptive Machine Learning Models with Kubernetes
Build machine learning pipelines from research to production
Build machine learning pipelines from research to production
Why more than half of ML models don't make it to production
Why more than half of ML models don't make it to production
Training Machine Learning models directly from GitHub with cnvrg.io MLOps
Training Machine Learning models directly from GitHub with cnvrg.io MLOps
Recently uploaded
Analysis insight about a Flyball dog competition team's performance
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
"Join us for STATATHON, a dynamic 2-day event dedicated to exploring statistical knowledge and its real-world applications. From theory to practice, participants engage in intensive learning sessions, workshops, and challenges, fostering a deeper understanding of statistical methodologies and their significance in various fields."
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
毕业原版【微信:176555708】【(UCSB毕业证书)圣芭芭拉分校毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
学校原件一模一样【微信：741003700 】《(英国UCA毕业证书)创意艺术大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
This webinar will explore cutting-edge, less familiar but powerful experimentation methodologies which address well-known limitations of standard A/B Testing. Designed for data and product leaders, this session aims to inspire the embrace of innovative approaches and provide insights into the frontiers of experimentation!
Global Situational Awareness of A.I. and where its headed
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
毕业原版【微信:176555708】【(GWU,GW毕业证书)乔治·华盛顿大学毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness in Kansas City, MO
To: Tom Tresser
From: Roger Warren
The Building Blocks of QuestDB, a Time Series Database
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Learn SQL from basic queries to Advance queries
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
Intelligence supported media monitoring in veterinary medicine
Media monitoring in veterinary medicien
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
The Modern Marketing Reckoner (MMR) is a comprehensive resource packed with POVs from 60+ industry leaders on how AI is transforming the 4 key pillars of marketing – product, place, price and promotions.
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
by
Timothy Spann
Principal Developer Advocate
https://budapestdata.hu/2024/en/
https://budapestml.hu/2024/en/
tim.spann@zilliz.com
https://www.linkedin.com/in/timothyspann/
https://x.com/paasdev
https://github.com/tspannhw
https://www.youtube.com/@flank-stack
milvus
vector database
gen ai
generative ai
deep learning
machine learning
apache nifi
apache pulsar
apache kafka
apache flink
Experts live - Improving user adoption with AI
Bekijk de slides van onze sessie Enhancing Modern Workplace Efficiency op Experts Live 2024.
Recently uploaded (20)
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Influence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business Plan
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Global Situational Awareness of A.I. and where its headed
Global Situational Awareness of A.I. and where its headed
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
Webinar kubernetes and-spark
- 1. WORKSHOP Spark on Kubernetes Create and set up your Spark cluster on Kubernetes Leah Kolben, CTO @leah4kosh leah@cnvrg.io
- 2. whoami • Developer/Data scientist => CTO • cnvrg.io = built by data scientists, for data scientists to help teams: • Get from data to models to production in the most efficient and fast way • bridge science and engineering
- 3. agenda • Introduction • What’s spark • Different spark implementations • What’s kubernetes • Spark on Kubernetes – pros vs cons • Live Workshop • Summary
- 4. What is Spark? • Unified analytics engine for large-scale data processing • Faster processing speed of applications due to In-memory cluster computing (100x improvement) • Support different workloads – batch, iterative, streaming, interactive SQL etc. • Support multiple languages and different environments
- 5. Spark deployment modes • Hadoop Yarn • Apache Mesos • Kubernetes • Standalone
- 6. Kubernetes - recap • Provides a runtime environment for Docker containers • Provides an abstraction layer for containers to run on • All services are natively load balanced • Can scale up and down dynamically • Monitor the health of the containers • Schedule runs and cronjobs • Use the same API across EVERY cloud provider and bare metal!
- 7. Spark Architecture on Kubernetes
- 8. Spark Architecture on Kubernetes • Spark-submit will be used to submit a spark application using kubectl: • Spark will create a spark driver as a pod • Driver will create executors (pods) and run the application code • When job is done – terminate jobs and clean resources (terminate nodes)
- 9. Spark on kubernetes • Kubernetes can manage unified containerized pipelines • Optimization for resource sharing • Leverage kubernetes resources: PV, service mesh, scheduling • Beta stage
- 10. Running out first workload • Use GKE for kubernetes cluster ( with auto scaling enabled) • Build spark docker image for kubernetes to use • Run pi.py on the cluster
- 11. Let’s do it!
- 12. Why use cnvrg to run your spark workloads? • Leverages the spark & kubernetes to one unified system • Reproducible jobs: artifacts are linked to workloads • Monitor your SPARK workload health • One unified dashboard for all your projects and workloads • Simple & fast • Clarity
- 13. DEMO
- 14. Summary • Spark is a unified analytics engine for large-scale data processing • Kubernetes is a platform for containers orchestration • Overview on Spark different implementaitons • Overview of spark on Kubernetes architecture • Overview of spark on Kubernetes: Pros vs. Cons • Submit a spark job directly on kubernetes cluster • Manage, monitor and automate your workloads using cnvrg