2
Operationalizing AI/ML is not trivial
Overview of Red Hat OpenShift AI
App developer
IT operations
Data engineer
Data scientist
ML engineer
Business leadership
Set goals Gather and prepare data Develop model
Model monitoring
and management
Integrate models
in app dev
Every member of your team plays a critical role in a complex process

3
Source: Gartner Peer Insights, Open Source AI for Enterprise survey, 2023
Operationalizing AI is still a challenging process
What is the average
AI/ML timeline from idea to
operationalizing the model?
Half of respondents (50%) say
their average AI/ML timeline
from idea to operationalizing
the model is 7-12 months.
50%
7-12 months
15%
3-6 months
4%
Unsure
26%
1 year or more
5%
Haven't done
this yet / Still in
experiment phase

Red Hat offers generative AI and MLOps capabilities for building flexible, trusted AI solutions at scale
Red Hat AI platforms
5
Generative
Models
Serving and
monitoring
Lifecycle
management
Resource optimization
and management
Model development
tooling
Scalability, operations
and automation
Hybrid-cloud
support
24x7 Global
premium support
Trust
Meet data location, privacy and security constraints while
controlling costs by owning the decision of where to train,
deploy and run AI models
Consistency
Streamlines the process of managing and monitoring the
lifecycle of models and AI-enabled applications at scale and
across clouds
Choice
Access to ready-to-use environments, models and AI/ML
capabilities from curated open source and emerging
technologies to expedite AI adoption
Alignment
tooling

6
Integrated MLOps platform for model lifecycle
management at scale anywhere
▸ Provides support for both generative and predictive
AI models with a BYOM approach
▸ Includes distributed compute, collaborative
workflows, model serving and monitoring
▸ Offers enterprise MLOps capabilities and the ability
to scale across hybrid-clouds
▸ Includes Red Hat Enterprise Linux AI, including the
Granite family models
Foundation model platform for developing,
testing, and running Granite family LLMs
▸ Provides a simplified approach to get started with
generative AI that includes open source models
▸ Makes AI accessible to developers and domain
experts with little data science expertise
▸ Provides the ability to do training & inference on
individual production server deployments

▸ Provide Granite LLM models and tooling for model customization as part of
RHEL AI and OpenShift AI
▸ Developing the infrastructure stack for single server and distributed workloads,
scheduling for building, fine-tuning and serving foundation models
▸ Enable out-of-the-box “bring your own model” use cases to OpenShift AI to
incorporate open source and partner ecosystem model integration
▸ Red Hat is infusing Generative AI capabilities into more of its portfolio.
･ OpenShift Lightspeed preview and RHEL Lightspeed vision
announced at Red Hat Summit
7
Red Hat strategy around generative AI and foundation models

8
8
Red Hat’s AI/ML engineering is 100% open source
Upstream projects Product
Community projects
CodeFlare

Red Hat OpenShift AI
9
9
Integrated AI platform
Create and deliver gen AI and predictive
models at scale across hybrid cloud
environments.
Available as
● Fully managed cloud service
● Traditional software product on-site or in the cloud!
Model development
Bring your own models or customize Granite models to your
use case with your data. Supports integration of multiple
AI/ML libraries, frameworks, and runtimes.
Lifecycle management
Expand DevOps practices to MLOps to manage the entire
AI/ML lifecycle.
Model serving and monitoring
Deploy models across any OpenShift footprint and centrally
monitor their performance.
Resource optimization and management
Scale to meet workload demands of gen AI and predictive
models. Share resources, projects, and models across
environments.

10
Red Hat software and
cloud services
OpenShift AI components Retrain
Model monitoring
and management
Gather and prepare data Develop models
Deploy models in
an application
ISV software and cloud services
Accelerators:
On-premise, cloud or edge infrastructure
Data science pipelines
Red Hat on-premise and
cloud platform
Model serving
Model performance
monitoring
Distributed
workloads
Data
streaming
Integration

Gather and prepare data Deploy models in an application
Model monitoring
and management
Develop models
11
Detailed look integrating our partner ecosystem
Application platform
Accelerators
Infrastructure
ISV software and
services
Customer managed applications
3rd party models and
repositories
Red Hat software and
cloud services
Red Hat cloud
platform
NVIDIA NIM
Granite
models
Physical Virtual Edge

12
… and the model operationalization life cycle
Starburst Enterprise or Galaxy
Unlock the value of your data by making it fast and easy to access data across hybrid cloud.
Pachyderm
Brings data versioning and governance to your most precious asset
Anaconda Professional
Curated access to an extensive set of data science packages to be used in your Jupyter projects.
RHEL AI and Granite LLM models
Unlock the power of efficient open source models with the RHEL AI component of OpenShift AI,
or bring your own gen AI and predictive AI models from vendors like Stability AI
IBM Watsonx.ai
Build, run, and manage generative AI models at scale.
Elastic Search AI Platform
Build transformative RAG applications with multilingual vector search
NVIDIA AI Enterprise and NIM
GPU-enabled hardware, software and NIM microservices makes it easier to stand up
resource-intensive environments and accelerate data science experiments.
Intel OpenVINO Notebook Images and Model Server
Toolkit of pre-trained models optimized for intel processors and GPUs. And a scalable,
high-performance serving engine.
Extract and
transform data
Run experiment and
create or tune models
Deploy models
as services
Monitor models and
track performance

Dashboard user interface
13

Dashboard resources
14

15
Gain hybrid cloud flexibility
Simplify AI adoption
Streamline the process of moving
models from experiments to
production
Drive AI/ML operational consistency
Deploy models in containerized
format across on-prem, clouds and
edge, including disconnected
environments
Promotes freedom of choice and
access to latest innovation on AI/ML
technologies
What differentiates us?
The value of Red Hat OpenShift AI

Source: Red Hat Summit presentation - May 2024
“Red Hat’s work with
AGESIC exemplifies
our dedication to
improving the user
experience for both
our and their
customers.”
Steven Huels
Vice President and General
Manager – AI Business Unit,
Red Hat
Government: LATAM
Presentation abstract
AGESIC, Uruguay’s Agency for Electronic Government and Information and
Knowledge Society, is responsible for e-government strategy and implementation.
With Red Hat®, it led Uruguay’s AI strategy and provided a more consistent, hybrid
AI/ML platform to build and host models while delivering innovative applications.
Presentation summary
● With the proliferation of AI, AGESIC knew that infusing it into its operations
would be key to meeting Uruguay’s evolving needs.
● AGESIC optimized its AI infrastructure with Red Hat OpenShift®, which
brought a containerized approach to workload management and automation
of key processes while also bringing development, operations, and systems
security functions together on a centralized platform.
● AGESIC evolved its offerings to include Platform as a Service (PaaS),
enabling other government agencies to develop, run, and manage
applications without the build and maintenance of complex infrastructure.
Products and services
Red Hat OpenShift Red Hat OpenShift AI

17
Source: Red Hat Q&A: Clalit accelerates research and innovation with AI and GPUs. Interview with Eyal Dviri, Innovation Team Leader in the Data Department at
Clalit. May 2024.
Overview
The patient lies at the center of everything Clalit Health Services (Clalit) does. Its 14
hospitals include 8 general hospitals, 2 mental health hospitals, 2 geriatric hospitals, and a
children’s hospital. It also operates community clinics, dental clinics, imaging facilities, and a
lifestyle program. Clalit recently established an advanced AI platform based on Red Hat
OpenShift AI and Red Hat® OpenShift®. We recently interviewed Eyal Dviri, Innovation Team
Leader in the Data Department at Clalit:
What led you to Red Hat OpenShift AI?
“Our central IT department implemented Red Hat OpenShift to provide a modern solution
for a wide range of use cases across the caregiving part of our organization. When they
made it available to us, we adopted it immediately for our research needs. We didn’t think
twice. To address our ML/AI use cases, because we had OpenShift already, we decided to
go with Red Hat OpenShift AI.”
What’s next?
“We have two challenges ahead. The first thing is the increase in demand when people hear
about our OpenShift AI platform. We need to figure out how to use our computing
resources—our GPUs—optimally. The second thing is the pipeline. We need to understand
best practices for deploying AI algorithms from training to production.”
Red Hat OpenShift AI use cases:
● A joint venture with Harvard
University Medical School. Red
Hat OpenShift AI is helping them
process their large quantity of
structured data using an NVIDIA
GPU.
● Using LLM to pinpoint patients
for preventive medication or
closer inspections. We need to
look at doctors’ notes to find
these people; for that, we use
LLMs such as LLaMa.
● Another use case allows us to
develop image-based machine
learning processes.
Customer perspective: Healthcare

18
Source: Red Hat blog. Red Hat, Team Guidehouse named winner in Mission Daybreak challenge to reduce Veteran suicides, Mar. 2023.
Challenge
Develop new data-driven means of identifying Veterans at risk for suicide.
Solution
Red Hat teamed with global consulting services provider Guidehouse and Philip Held,
Ph.D. of Rush University Medical Center, to develop a new data-driven means of
identifying Veterans at risk for suicide running on Red Hat OpenShift, leveraging Red Hat
OpenShift API Management and Red Hat OpenShift AI.
Results
● Named a winner in the Mission Daybreak challenge, Phase 2, of the U.S.
Department of Veterans Affairs’ (VA) Mission Daybreak Grand Challenge in
support of cutting-edge suicide prevention solutions
● Moved forward with a solution for the VA’s efforts to reduce Veteran suicides
● Showcased the repeatability and scalability of open source-enabled solutions
Suicide has no single cause, and no
single strategy can end this complex
problem. That’s why Mission
Daybreak is fostering solutions
across a broad spectrum of focus
areas.
A diversity of solutions will only be
possible if a diversity of solvers
answer the call to collaborate and
share their expertise.
Red Hat, Team Guidehouse named winner in
Mission Daybreak challenge to reduce Veteran
suicides
U.S. Department of
Veterans Affairs

19
▸ Implemented interactive lecture and lab environment for computer
scientists and engineers based on Red Hat OpenShift AI
▸ Currently over 300 users including over 100 concurrent
▸ Integrates with the Boston University online textbook material,
also authored using the Red Hat OpenShift AI
▸ Fast time to solution: cloud services environment enabled BU to
configure and deploy in December for classes that started in January
▸ Lowers cost: auto-scales based on demand; enables bursty
interactive use cases at optimized cost

20
Award Logos
2024 Finalist - Artificial Intelligence Excellence Awards
2024 AI Awards for OpenShift AI

Red Hat Consulting AI Offerings
Learn how to maximize your technology investments
MLOps Foundation
AI Accelerator AI Incubator
Rapidly deploy and adopt Red Hat
OpenShift AI while advancing your AI
practices:
● Upskill your ML Platform team and
data scientists and adopt new AI
capabilities
● Deploy an AI platform and tools
with GitOps
● Review your current MLOps
processes and identify improvements
using Red Hat OpenShift AI
Roll out automated MLOps pipelines
and practices throughout your
organization
● Establish self-service of your
MLOps platform
● Automate and template ML
pipelines
● Establish patterns and best
practices for managing production
ready solutions
● Transform models to microservices
Upskill MLOps teams on AI tooling
through delivery of gen AI use
cases
● Work side-by-side with Red Hat
experts utilizing best practices while
learning MLOps and AI Engineering
skills
● Prototype your custom gen AI
solution
● Release a solution in your
environment
Red Hat Learning: Developing and Deploying AI/ML Applications on Red Hat OpenShift AI (AI267)

22
Learn more ▸
Try it ▸

linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning
support, training, and consulting services make
Red Hat a trusted adviser to the Fortune 500.
Thank you
23

25
Foundation models
25
An open source
platform for
foundation models
Train or fine tune
conversational
and generative AI
Kueue
Job dispatching, queuing, and scheduling
TorchX
KubeRay
TGIS
Optimized text generation interface server
Caikit
Dev APIs, prompt tuning interface
Training and validation | Workflows Tuning and Inference | Domain specific APIs
vLLM
Memory efficient inference for serving LLMs

Model training
Model training highlights
Support a variety of use cases
including generative AI by accelerating and
managing model training and tuning workloads
Initiate and manage batch training
in single- or multi-cluster environments with
an easy-to-use interface
Automate foundation model pipelines
Improve performance and scalability
with distributed training
Meet scale and performance needs
by selecting from a range of accelerators

Model training
27
Distribute workloads to enhance efficiency
Focus on maintaining models, not
infrastructure
by dynamically allocating computing power
Automate setup and deployment
so you can get up and running with minimal
effort
Manage resources and submit jobs
using a Python-friendly SDK, which is a
natural fit for data scientists
Prioritize and distribute job execution
using advanced queuing for tasks like
large-scale data analysis
Streamline data science workflows
with seamless integration
into the OpenShift AI ecosystem

Model training
28
Configure distributed workload clusters more easily
1. Send request for creation of cluster
2. Evaluate clusters for aggregated resources
and dispatch when available
3. Watch pending clusters to provide
aggregated resources and guarantee
workload execution
4. Develop and submit jobs to retrieve statuses
and logs from the clusters
5. Send request for deletion of clusters
Process
Data
scientist
Multi-cluster
dispatching
Dynamic scaling
Distributed clusters
Create Delete Add

29
Model serving
Make model serving more flexible and transparent
▸ Use model-serving user interface (UI)
integrated within product dashboard and projects workspace
▸ Support a variety of model frameworks
including TensorFlow, PyTorch, and ONNX
▸ Choose inference servers
either out-of-the-box options optimized for model types
or your own custom inference server
▸ Serve models
from providers like Hugging Face and Stability AI or Granite models
▸ Scale cluster resources
up or down as your workload requires

Model serving
30
Serve, scale, and monitor your models
Select the required
resources and
scale model serving
as needed
Make your model
public and secure
Select your model
framework
View your
deployed model
fleet endpoints

Model monitoring
31
Model performance metrics
Access a range of model performance metrics
to build your own visualizations or integrate
data with other observability services
▸ Out-of-the-box visualizations for performance
and operations metrics

Data Science Pipelines
32
Data science pipelines component
▸ Orchestrate data science tasks into pipelines
▸ Chain together processes like data prep, build models,
and serve models
▸ Data science pipelines are based on upstream
Kubeflow pipelines
▸ Continuously deliver and test models in production
▸ Schedule, track, and manage pipeline runs
▸ Easily build pipelines using graphical front end
Gather data Process data Train model
Download
existing model
Compare new
model with existing
Push new model
if better

Data Science Pipelines
33
Red Hat OpenShift data science pipelines user interface
The OpenShift AI user
interface enables you to
track and manage pipelines
and pipeline runs.

CONFIDENTIAL designator
z[[
Far edge Near edge Enterprise
Last-mile network
Unreliable connection
Device edge
Edge Core
Code, configuration, master data, machine learning models, control, commands, etc.
Sensor data, telemetry, events, operational data, general information, etc.
Model monitoring
Model registry
Model serving
component
Model training
Device
or sensor
Pipelines
Edge AI
34
Flexibility at the edge

Edge AI
35
Red Hat OpenShift AI at the edge
▸ Deploy centrally to the near edge using GitOps approach
▸ Monitor operations using centralized Grafana dashboard
▸ Provide data scientists with actionable insights
▸ Automate deployment throughout stages with
repeatable MLOps pipelines
Consistently deploy and manage
intelligent applications

Red Hat Enterprise
Linux AI
36

Seamlessly develop, test, and run Granite
family large language models (LLMs) for
enterprise applications.
Foundation Model Platform
37
InstructLab model alignment tools
Scalable, cost-effective solution for enhancing LLM
capabilities and making AI model development open and
accessible to all users.
Optimized bootable model runtime instances
Granite models & InstructLab tooling packaged as a
bootable RHEL image, including Pytorch/runtime libraries
and hardware optimization (NVIDIA, Intel and AMD).
Enterprise support, lifecycle & indemnification
Trusted enterprise platform, 24x7 production support,
extended model lifecycle and model IP indemnification by
Red Hat.
Overview of Red Hat Enterprise Linux AI
Granite family models
Open source-licensed LLMs, distributed under the
Apache-2.0 license, with complete transparency on training
datasets.

39
Move models from experimentation to production faster
Operationalize AI is the catalyst for incorporating AI into practical applications
Operationalize AI is the process of integrating AI capabilities into the day-to-day
operations of an organization. It involves taking your models from experimentation to production
while contributing to the overall goals of the organization.
Set goals Gather and prepare data Develop or tune model
Model monitoring
and management
Integrate models
in app dev

40
Gather and prepare data Monitor model
Develop model
Model
App
Retrain model
Code Deploy Operate & monitor
QA
Iterate
Automation Validation
Deploy model
RH1 - OpenShift AI
Lifecycle for operationalizing models

41
AI at Red Hat
▸ Integrate and curate open source technologies to bring AI use cases to life
▸ Support the e2e AI lifecycle: build, deploy, manage, and operate AI models securely and at scale
▸ Provide the tools for developing predictive models or tuning foundation models
▸ We don’t offer models, but partner with model builders and support the motion for ‘bring your
own model’
▸ Provide support for cloud, on-premises, and the edge
Red Hat provides enterprise-class AI solutions to help customers deploy
AI-enabled applications anywhere
The power of AI is open

42
Red Hat OpenShift
A platform for continuous
development, integration,
and deployment for
AI/ML models, such as
GPU support
OpenShift
Operators
OpenShift
GitOps
OpenShift
Pipelines
OpenShift
Serverless
Prometheus
Operating system
OpenShift
ServiceMesh
▸ One platform for all. Collaborative environments for dev, data
engineers, data scientists and DevOps.
▸ Extend capabilities with operators. Operators allow bringing
AI/ML capabilities to OpenShift.
▸ Hybrid-cloud support. On-premise support for model
development, delivery and deployment.
▸ Enhanced security. Expand DevSecOps practices to protect
the AI/Ml lifecycle.
OpenShift AI

43
Red Hat OpenShift AI provides an integrated platform for building,
training, tuning, deploying and monitoring AI-enabled applications,
predictive and foundation models securely and at scale across
hybrid-cloud environments.
Built on top of Red Hat OpenShift delivers a consistent, streamlined
and automated experience to help organizations rapidly innovate and
deliver AI-enabled apps into production.

44
Red Hat OpenShift AI - Key features
Interactive, collaborative UI for
seamless access AI/ML tooling,
libraries, frameworks, etc.
Model development
Model serving routing for
deploying models to production
environments
Model serving
Centralized monitoring for tracking
models performance and
accuracy
Model monitoring
Visual editor for creating and
automating data science pipelines
Data & model pipelines
Seamless experience for efficient
data processing, model training,
and tuning
Distributed workloads

Operating system
45
AI/ML platform
Platform
services
Hardware
accelerators
Bare metal Virtualization Edge
Cloud (Secured)
Cloud
Operating system
Operating system
Data & model pipelines
Model serving Model monitoring
Model development
Distributed workflows
GPU support
Deploy anywhere
Gather and prepare
data
Develop or tune model
Model monitoring
and management
Integrate models
in app dev
Retrain
Red Hat’s AI/ML platform

46
OpenShift
Operators
OpenShift
GitOps
OpenShift
Pipelines
OpenShift
Serverless
Prometheus
Dashboard Application
Model Development, Training & Tuning
Data Science Projects Admin Features
Object
Storage
Model Serving Model Monitoring
Performance metrics
Operations metrics
Quality metrics
Serving Engines
Serving Runtimes
Kserve
ModelMesh
OVMS
vLLM, Caikit/TGIS
Custom
Distributed workloads
KubeRay
CodeFlare
Data and model Pipelines
Workbenches
Custom images
ISV images
- Minimal Python
- PyTorch
- CUDA
- Standard Data Science
- TensorFlow
- VS Code
- RStudio
- TrustyAI
CodeFlare SDK
Operating system
Model Registry
OpenShift
ServiceMesh
Kueue
Models
RHEL AI
Ecosystem models

partners-red-hat-openshift-ai-101-deck.pdf