An overview of foundation models.pdf

9/11/23, 12:04 PM An overview of foundation models
https://www.leewayhertz.com/foundation-models/ 1/19
An overview of foundation models
leewayhertz.com/foundation-models
Artificial intelligence has experienced a remarkable surge in adoption thanks to exciting
technological breakthroughs and our ever-increasing access to vast amounts of data. AI
systems now have the power to learn from enormous datasets comprising thousands or
even millions of examples, fueling their ability to comprehend and interpret spoken or
written language. From smart recommendation systems on e-com apps that suggest
items based on our preferences to virtual assistants making lives easier, we rely heavily
on AI in our daily experiences. In fact, as per Grand View Research, the market size for
artificial intelligence globally was worth USD 136.55 billion in 2022, and it is estimated to
grow at a CAGR of 37.3% between 2023 and 2030.
However, building and deploying new AI systems can be time-consuming and resource-
heavy. Developing a new system requires a sizeable dataset that is well-labeled and
specific to the task at hand. If such a dataset does not exist, significant effort and time are
required to find and label suitable images, text, or graphs for the dataset. This is where
the concept of foundation models steps in, empowering AI systems to leverage pre-
existing knowledge and significantly mitigate the demand for extensive training.
Foundation models have been instrumental in advancing AI by serving as powerful
building blocks for creative output generation. Popularized by the Stanford Institute for
Human-centered Artificial Intelligence, foundation models have shown great potential in
areas like imagery and language. Examples like GPT-3, BERT, and DALL-E 2 have
demonstrated their capabilities by generating complex images and essays based on short
prompts, even if they were not explicitly trained to execute such tasks.
Undoubtedly, foundation models are a groundbreaking development in the field of Natural
Language Processing (NLP) and serve as the core architecture upon which various
language models are built, providing a solid base of understanding and generating high-
quality text.
Now, let’s dive deeper into foundation models and understand their operational
mechanics, architecture, capabilities and applications.
What are foundation models?
The need for a foundation model
Transformer architecture
Types of foundation models and how do they work?
LLM’s
Diffusion models
Capabilities of the foundation model
Components of the foundation models

How can we use the foundation models?
Fine-tuning
Prompting
Applications of foundation models
What are foundation models?
Foundation models are pre-trained models that can be fine-tuned on specific tasks or
domains. These highly adaptable and high-performing models find applications across
diverse domains, including Natural Language Processing (NLP), computer vision, and
multimodal tasks. They play a key role in driving innovation and advancements in artificial
intelligence. They are large-scale models with billions of parameters that can generate
output in various forms, such as text, images and even code. Unlike traditional models
that rely on labeled data for training (supervised learning), foundation models are trained
on unlabeled data using deep neural networks. Unsupervised learning enables these
models to discover patterns and structures within data more efficiently.
Delving deeper into their functioning, foundation models are pre-trained on a massive and
diverse dataset, such as images or text. The pretraining process involves exposing the
model to vast amounts of data, enabling it to learn patterns and features from the data.
This makes foundation models extremely powerful and versatile, allowing for seamless
utilization across a wide range of use cases with minimal training data. After pretraining,
the model is ready to be fine-tuned.
AI foundation models use deep neural networks that allow them to emulate the human
brain’s functionality and handle complex tasks such as generating code or tackling
intricate mathematical problems. This power stems from their ability to match patterns,
which is vital for AI applications. For instance, a deep neural network can analyze millions
of image sets and associate a particular word (say, “cat”) and can further enhance the
ability to recognize, visualize, and predict image components with additional data and
examples. As a result, the model’s scope expands as it analyzes more complex patterns
and correlations. However, creating a foundation model in AI is a demanding and
expensive process that involves large amounts of computational resources and expertise.
Instead, developers can use existing foundation models as the base for building task-
specific models through adaptation. The term “foundation” highlights these models’
importance of stability, safety, and security.
The need for foundation models
Pretrained models have become ubiquitous in machine learning, particularly for text and
image tasks. Earlier, these models were trained on large labeled data, which enabled
them to generalize well to new tasks. However, this approach had drawbacks since the
model’s ability to leverage the vast amounts of available unlabeled data was limited. To
overcome this, researchers have developed foundation models which are capable of
making use of unlabeled data.

Foundation models, such as BERT and GPT, utilize the transformer architecture, which
employs self-attention to weigh the importance of different input elements. Introduced in
2017, transformers have replaced traditional recurrent neural networks, offering
advantages like parallel processing and bidirectionality for better language
understanding.
The transformer architecture
Transformers are a type of neural network architecture that has witnessed a remarkable
surge in popularity within the field of artificial intelligence. The development of
transformers was intended to address the issue of sequence transduction, which includes
tasks such as neural machine translation, speech recognition, text-to-speech conversion,
and more. Transformers utilize an encoder-decoder architecture based on attention
layers. The transformer model has proven to be highly effective in capturing
dependencies between elements in a sequence and has significantly advanced the field
of natural language processing. Let’s now understand in detail how this encoder-decoder
architecture works:
Encoder
The encoder in a transformer model plays a crucial role in enabling the model to
comprehend and encode the input sequence, facilitating effective understanding and
subsequent processing of the text data. The encoder architecture consists of several
components and processes that contribute to its functionality:
Embeddings
Computers do not comprehend language in its textual form but rather in the form of
numerical data such as vectors or matrices. Therefore, it is necessary to convert words
into vectors to enable machine processing. This is where the concept of an embedding
space comes into play. An embedding space is a space or dictionary where words with
similar meanings are clustered or close to one another. Each word is assigned a
particular value in the embedding space according to its meaning. This process allows us
to convert words into vectors for machine processing.
Positional encoding
Positional encoders address the issue of word ambiguity by providing context based on
the position of words within a sentence. Each word is converted into an embedding and
augmented with a positional embedding. These combined embeddings create a context
vector, which is then processed by the encoder block. The positional encoders enable the
model to understand word meanings in different sentence contexts, enhancing its ability
to capture relationships and dependencies in the input sequence.
Multi-head attention

The central aspect of the transformer architecture is its “self-attention” mechanism, which
determines the importance of each word with other words within a sentence. This is
accomplished by generating an attention vector for each word, which captures the
contextual relationships between the words in the sentence.
One limitation of the self-attention mechanism is that it assigns a higher weight to a
word’s value in the sentence rather than its interactions with other words. Multiple
attention vectors are calculated for each word to overcome this issue, and a weighted
average is taken to compute the final attention vector for that word. This enables the
model to capture the relevant contextual relationships between the words in the
sentence. The use of multiple attention vectors is referred to as the multi-head attention
block.
Feed-forward network
The second step involves the feed-forward neural network applied to each attention
vector. The goal is to transform the attention vectors into a format that the subsequent
encoder or decoder layers can process. The feed-forward network processes the
attention vectors individually, one at a time. Unlike Recurrent Neural Networks (RNNs),
the attention vectors are independent of each other. As a result, parallelization can be
utilized, which can significantly improve processing speed and efficiency.
With the ability to process attention vectors independently and in parallel, we can pass all
the words into the encoder block at the same time, resulting in a set of encoded vectors
for each word, which can be computed simultaneously.
Decoder
In a transformer model, the decoder plays a crucial role in generating the output
sequence based on the encoded input representation. While the encoder focuses on
understanding the input sequence, the decoder focuses on generating the target
sequence by attending to the encoded information.
For example, let’s consider training a language translator from English to French. We
provide an English sentence and the corresponding French translation to train the model.
The English sentence is processed through the encoder block, while the French sentence
is processed through the decoder block.
Like the encoder block, the decoder block also includes an embedding layer and a
positional encoder component, which converts the words in the input sentence into
corresponding vectors.
The decoder architecture consists of several components and processes that contribute
to its functionality, such as:
Masked multi-head attention

In the masked multi-head attention process, each word in a sentence dynamically
interacts with the surrounding words to uncover the intricate relationships between them.
The term “masked” is used to prevent the model from looking at future words during this
process.
Let’s consider the example of the language translator for better understanding. The input
sentence in French is processed through the self-attention mechanism. The self-attention
mechanism is a fundamental component used in both the encoder and decoder blocks. It
enables the model to capture the relationships and dependencies between different
words within a sentence or sequence.
To elaborate on the learning mechanism, during the training process, the model first
predicts the French translation of each English word using its previous results. These
predicted translations are then compared with the actual French translation (which we
provide as input to the decoder block). Based on the comparison, the model updates its
matrix values, allowing it to improve its predictions with each iteration. This iterative
learning process continues until the model accurately translates the input sentences.
During training, we must hide (or mask) the next French word from the input sequence.
This ensures that the model learns to predict each French word based solely on its
corresponding English and previously predicted French words without knowledge of the
next French word. By masking the next French word, we prevent the model from simply
memorizing the input-output pairs and encourage it to learn the underlying patterns and
relationships between the two languages. When processing the French sentence in the
decoder block, we can only use the previously generated French words to predict the
next word. We can’t use any information from future French words as that information is
unavailable during inference.
To achieve this, we mask all future French words by transforming them into 0s. This is
done by creating a mask matrix with the same shape as the attention matrix, with values
of 1 where the corresponding French word is available and 0 where it is masked. During
the attention operation, we use this mask matrix to zero out the attention weights for the
masked French words, ensuring they don’t contribute to predicting the next word.
Multi-head attention block
The next step involves passing the attention vectors obtained from the previous layer and
the encoded vectors from the encoder block through another multi-head attention block.
This step is called the encoder-decoder attention block, where the results from the
encoder block are used. The encoder-decoder attention block is where the main mapping
between English and French words happens. The attention vectors generated in this
block capture the contextual relationship between the words in the English sentence and
those in the corresponding French sentence. This is important for generating accurate
translations, as the model needs to comprehend the relationship between the words in
both languages to translate them accurately.

Feed-forward network
The feed-forward unit is typically a simple two-layer neural network with a ReLU
activation function in between. It is applied to each attention vector independently,
allowing for parallelization. This layer helps transform the attention vectors into a form
easily digestible by the next layer of the model.
The linear layer’s output is passed via the softmax layer, producing a probability
distribution over the possible output words in French. Each word is assigned a probability
based on if the translation accurately renders the English word entered. The word with
the highest probability is then selected as the translated word for that position in the
sentence. Repeating this method for each word in the original English sentence produces
the translated French sentence.
Types of foundation models and how they work
AI foundation models can broadly be categorized into two types: LLMs and diffusion
models. Let’s discuss the two in detail:
LLMs
LLMs, or Large Language Models, are machine learning models that utilize deep learning
techniques to process and generate natural language. They are trained on vast amounts
of textual data and can perform various language-related tasks, such as language
translation, text summarization, and question-answering. Transformer-based LLMs have
received significant attention owing to their remarkable performance on various natural
language processing tasks. Popular examples of LLMs include GPT-3, BERT, and
RoBERTa. Let us understand the working of LLMs in detail:
Pretraining
During pertaining, LLMs are trained on vast and diverse datasets that contain a wide
range of text, such as books, articles, and websites. Pretraining aims to enable the model
to learn language patterns, including grammar, syntax, and semantics.
Pretraining is typically achieved through unsupervised learning, and LLMs can be trained
in various ways during this stage. For example, OpenAI’s GPT models are trained to
predict subsequent words in a partially complete sentence, while Google’s BERT is
trained using a technique called masked language modeling, where the model is required
to guess the randomly blanked words in a sentence.
During pretraining, the LLM regularly updates the weights of its parameters to minimize
the prediction error, which enables it to learn to generate coherent and contextually
relevant text.
Fine-tuning

After pretraining, the LLM is fine-tuned on a smaller, task-specific dataset using
supervised learning. During fine-tuning, the model is provided with labeled examples of
the desired output for a specific task.
The fine-tuning process allows the model to adapt its pre-trained knowledge to the
specific requirements of the target task, such as translation, summarization, sentiment
analysis, and more. This adaptation is achieved by adjusting the model’s parameters
through techniques such as gradient descent and backpropagation, which optimize the
model’s performance on the task.
Pretraining and fine-tuning are highly effective in building LLMs that can perform a wide
range of NLP tasks with state-of-the-art accuracy.
In-context learning
In-context learning refers to the ability of a language model to learn and perform a task
based on a few examples or specific context, even if it was not explicitly trained for that
task. It implies that the model can generalize its knowledge from the provided examples
to similar scenarios without the need for retraining or additional labeled data.
For example, GPT-3, a language model, can accurately determine the sentiment (positive
or negative) of a new sentence after being exposed to several sentences with known
sentiments. This capability arises from the model’s capacity to update its parameters and
adapt its behavior based on the given context. In other words, the model can leverage the
contextual information it has learned to make informed predictions or decisions without
requiring explicit training for each specific task.
In this context, the model appears to have learned something new without undergoing
additional training because it can apply its existing knowledge and adapt it to the specific
task or context at hand. This phenomenon of in-context learning allows the model to
exhibit a level of flexibility and generalization, enabling it to perform tasks beyond its initial
training scope.
This is the basic working of an LLM model. The process may vary depending on the type
of LLM model.
Diffusion models
Diffusion models are generative models used to generate data similar to the data they
were trained on. These models work by adding Gaussian noise to the training data and
then learning to reverse this noising process to recover the original data. Diffusion models
have shown promising results in various applications, including image and speech
synthesis, and are known for generating high-quality samples with fine details. The most
popular examples of diffusion models are Dall-E, Imagen, Glide etc.

In essence, diffusion models are a type of probabilistic generative model that learn the
underlying structure of a dataset by modeling how data points diffuse through the latent
space. The diffusion process describes how a data point moves through the latent space
over time. In a diffusion model, the process is modeled using a markov chain, where the
markov chain’s current state represents the data point’s current location in the latent
space. The diffusion process is usually defined as a series of stochastic transformations
that gradually spread out the data points in the latent space. Neural networks often
parameterize these transformations and may depend on additional input, such as the
noise level in the data.
Once the diffusion process is defined, the diffusion model is trained using variational
inference. The goal of variational inference is to maximize the log-likelihood of the training
data for the model parameters. This is done by introducing a variational distribution over
the latent variables and minimizing the Kullback-Leibler (KL) divergence between this
distribution and the true posterior distribution. In the case of diffusion models, the
variational distribution is often defined as a normal distribution with mean and variance
parameters that are functions of the observed data. The KL divergence between the
variational and true posterior distributions can be estimated using monte carlo methods
such as importance sampling or Markov Chain Monte Carlo (MCMC).
During training, the diffusion process is applied to the training data in reverse order,
starting from a fully blurred or corrupted version of the original data and then iteratively
applying the inverse transformations to estimate the true data. By optimizing the
variational parameters and the diffusion process parameters jointly, the model learns to
denoise.
After training, the diffusion model can be used for various tasks such as denoising,
inpainting, super-resolution, and image generation. For image generation, the model
takes a random noise image as input and then applies the learned diffusion process in
the forward direction to generate a new image that follows the same distribution as the
training data.
Capabilities of the foundation model
Foundation models have diverse capabilities that can be utilized to power various
applications. From processing different modalities to affecting the physical world and
interacting with humans, the capabilities of foundation models are almost limitless. Let’s
discuss more about its capabilities below:
Natural language processing
The most impactful feature of AI foundation models in NLP is not their generation abilities
but their remarkable generality and adaptability. A single foundation model can be
adapted in various ways to accomplish multiple linguistic tasks. NLP encompasses
various tasks, including sentence or document classification, sequence labeling, span

relation classification, and text generation. Generating general-purpose language was
widely considered a challenging and practically unattainable task in NLP and was
believed to require the accomplishment of other linguistic sub-tasks. However, the advent
of foundation models in AI that are predominantly trained to generate language has
marked a significant change in the role of language generation in NLP.
Highly coherent foundation models can be trained with a simple language generation
objective, such as predicting the next word in a sentence. These generative models have
become the primary means for language-based machine learning, including analysis and
understanding tasks once considered prerequisites for a generation. Before the
emergence of foundation models, researchers focused on tackling complex NLP tasks.
However, most of these tasks are handled at an almost-human level using publicly
released foundation models.
Visual comprehension
Computer vision has been one of the key areas that have driven the adoption of deep
learning in AI. Deep learning models have demonstrated remarkable success in image
classification, object detection, and other standard tasks in computer vision, thanks to
their ability to learn complex representations directly from raw data. However, traditional
deep learning models require large annotated datasets for training, which can be time-
consuming and expensive.
In recent years, there has been a growing interest in pretraining deep learning models on
web-scale raw data rather than curated datasets. These foundation models have shown
great promise in computer vision, enabling knowledge transfer to various downstream
tasks. Training on multimodal and embodied data has paved the way for progress in more
challenging areas, such as 3D geometric and physical understanding and commonsense
reasoning. The emergence of foundation models in AI has facilitated acquiring a
contextual understanding of the visual world using large quantities of raw data. With the
use of self-supervision techniques in foundation models, there has been significant
progress in traditional computer vision tasks like image classification, object detection,
and visual synthesis. Foundation models in AI have enabled training without explicit
annotations and with massive amounts of diverse and raw visual data, resulting in
competitive performance.
Robotics
In robotics, foundation models could address task specification and task learning
challenges, which are more complex than those in other domains like NLP. Unlike in NLP,
where many problems can be formulated as “text-in, text-out” tasks, robotic tasks often
have unique input-output that require specialized models. As such, there is a need for AI
foundation models that can generalize across different tasks, environments, and robot
embodiments. In artificial intelligence, there has been a shift towards developing more
advanced models that can not only perform specific tasks but also learn from their

experiences, adapt to new situations, and reason about the physical world in a way
generalizable to different tasks and contexts. This means that instead of building
separate models for each task or situation, a more general model can be created that can
be applied to various scenarios.
Foundation models for task specification need to understand and interpret various forms
of human communication, including natural language, gestures, demonstrations, and
other forms of feedback. They should also be able to incorporate prior knowledge about
the task and environment and adapt to changes in task requirements and environmental
conditions. This requires the development of models that can reason about task structure,
incorporate probabilistic inference to deal with uncertainty, and learn from few or even
single demonstrations. Developing foundation models for task specification is a critical
step toward enabling robots to operate in various environments and perform various
tasks without requiring significant manual engineering or reprogramming for each new
setting. Robotic foundation models can leverage self-supervised learning to acquire
knowledge from interaction data, enabling quick adaptation to new tasks and
environments, thus reducing the need for extensive new data. This approach enhances
robots’ learning efficiency and consistency.
Reasoning and search
Due to the huge combinatorial search space involved, AI has faced long-standing
challenges in solving reasoning and search problems, such as theorem proving and
program synthesis. Nevertheless, humans possess an innate capacity for intuitive
reasoning in most mathematical domains and the ability to transfer knowledge across
tasks to facilitate efficient adaptation and abstract reasoning. Recent AI research has
shown that deep neural networks can effectively guide the search space of these
problems. However, these approaches still have limitations in transferring knowledge and
reasoning abstractly. To address these challenges, these multi-purpose models possess
strong generative and multimodal capabilities, potentially controlling the inherent
combinatorial explosion in search and enabling more efficient adaptation and abstract
reasoning.
Thus, combining deep neural networks and AI foundation models holds great potential for
advancing state-of-the-art reasoning and search problems in AI.
Human engagement
Foundation models in AI can provide a starting point for developers with little experience
in creating effective AI applications. By using pre-trained models, developers can
leverage the knowledge and experience already in the model, allowing them to create
more effective and efficient applications.
Regarding user experience, AI foundation models can help improve the interaction
between humans and AI by focusing on human agency and reflecting user values. For
example, a foundation model could provide a chatbot application’s underlying language

processing capabilities while incorporating user-specific information and preferences to
provide a more personalized experience. This leads to more natural and engaging
interactions, improving the user experience.
Components of foundation models
Foundation models in AI are based on a few important components, which are discussed
below:
Data
Data is a fundamental component of foundation models. It serves as the raw material
upon which these models are trained, enabling them to learn language patterns, context,
and semantics. The data’s quality, diversity, and size play a crucial role in determining the
performance and capabilities of foundation models.
Training a foundation model involves exposing it to vast amounts of text data from
various sources. This data can include books, articles, websites, social media posts, and
more. The larger and more diverse the dataset, the better the model’s ability to
understand and generate human-like language.
Data serves multiple purposes in the training of foundation models:
1. Language understanding: By training on a wide range of texts, the model learns a
language’s statistical patterns and linguistic structures. This enables it to
comprehend the meanings, context, and relationships between words and phrases.
2. Contextual knowledge: Data provides the model with a wealth of knowledge about
different domains, topics, and cultural nuances. This contextual knowledge helps
the model generate responses and make predictions that align with human
understanding.
3. Generalization: Exposure to diverse data enables the model to generalize its
knowledge beyond the specific examples it has seen during training. It learns to
infer patterns and apply them to unseen or similar situations, allowing it to perform
well on a variety of language-related tasks.
4. Bias and fairness: Data plays a crucial role in addressing bias and fairness in
foundation models. The quality and diversity of the training data help mitigate
biases and ensure that the model’s responses are more inclusive and
representative of diverse perspectives.
It is essential to note that the data used for training foundation models should be carefully
curated and preprocessed to maintain quality and mitigate potential biases. Pre-training
often involves large-scale unsupervised learning, where the model learns to predict the
next word in a sentence based on the context provided by the preceding words. This
process helps the model acquire a general understanding of language.

Modality
Modality is another important aspect of foundation models. It refers to the different types
of input or output that a model can handle, such as text, images, speech, or a
combination of these modalities. The ability to handle diverse modalities allows
foundation models to understand and generate content in various formats, enhancing
their versatility and applicability across different domains.
Here’s how modality functions as a core component of foundation models:
1. Input modality: Foundation models can accept inputs in different modalities. For
example, they can process text inputs to understand the meaning and context of
written language. They can also handle image inputs, analyzing visual features and
extracting information from images. Additionally, models may support speech
inputs, enabling them to transcribe spoken language or perform tasks like voice
recognition.
2. Output modality: Foundation models can generate output in various modalities.
They can generate coherent text responses, providing detailed answers, creative
writing, or natural language conversation. They can also produce image-based
outputs, generating images based on given prompts or completing missing parts of
an image. Furthermore, models can generate speech outputs, synthesizing natural-
sounding speech in different languages or voices.
3. Multimodal fusion: Foundation models can effectively combine information from
multiple modalities. They can process and analyze inputs that contain both text and
images or text and speech, enabling them to understand complex scenarios that
involve different forms of data. By fusing information from multiple modalities, the
models can provide richer and more comprehensive responses or generate content
that incorporates multiple modalities.
4. Transfer learning: The inclusion of multiple modalities in foundation models allows
for transfer learning between different tasks and modalities. Knowledge acquired
from one modality can be transferred and applied to another. For example, a model
trained on text-to-speech synthesis can leverage that knowledge to improve its
performance in other speech-related tasks. This transferability enhances the
efficiency and effectiveness of the models across various domains and applications.
By incorporating different modalities, foundation models can handle a wide range of
inputs and generate outputs in diverse formats. This flexibility makes them valuable tools
for various applications, such as natural language processing, computer vision, speech
recognition, multimodal understanding, and more. The modality component broadens the
scope of foundation models, enabling them to process and generate content across
multiple domains and modalities, advancing the capabilities of AI systems.

Training
parameters to enable it to learn and improve its performance on specific tasks. Training is
a critical step in building robust and effective foundation models.
Here’s how training functions as a core component of foundation models:
1. Dataset preparation: The training process begins with the preparation of a suitable
dataset. The dataset should be diverse, representative of the target domain, and
appropriately labeled or annotated, depending on the task. This dataset serves as
the training input, allowing the model to learn from the examples and patterns
present in the data.
2. Model architecture: Before training, the model’s architecture needs to be defined.
This includes determining the structure, layers, and connections of the neural
network that constitutes the foundation model. The architecture defines the model’s
capacity to learn and represent information.
3. Backpropagation and optimization: During training, the model is exposed to the
training dataset, and its parameters are adjusted iteratively to minimize the
difference between the model’s predictions and the ground truth labels. This
process, known as backpropagation, uses techniques like gradient descent to
update the model’s parameters and optimize its performance. The goal is to find the
optimal set of parameters that minimizes the overall loss or error of the model.
4. Iterative learning: Training occurs over multiple iterations or epochs. In each
iteration, the model is presented with batches of training data, and the parameters
are updated based on the computed gradients. By repeatedly exposing the model
to the dataset and updating its parameters, the model gradually learns to generalize
patterns, understand relationships, and improve its performance on the target task.
5. Regularization and hyperparameter tuning: To prevent overfitting and improve
generalization, regularization techniques such as dropout, weight decay, or batch
normalization may be applied during training. Hyperparameters, including learning
rate, batch size, and regularization strength, are tuned to find the optimal settings
for training the model.
6. Evaluation and validation: Throughout training, the model’s performance is
evaluated using separate validation datasets or metrics. This helps assess its
generalization ability and prevent overfitting. The evaluation results guide the
decision-making process, such as early stopping criteria or further adjustments to
the training process.
7. Transfer learning: Foundation models can also benefit from transfer learning,
where pre-trained models on large-scale datasets are fine-tuned or used as starting
points for specific tasks. Transfer learning leverages the knowledge and
representations learned from one task or dataset to improve performance on related
tasks with smaller or domain-specific datasets.

Training is an iterative and resource-intensive process that involves adjusting model
parameters, optimizing performance, and fine-tuning to achieve the desired task-specific
outcomes. The quality of the training data, the architecture design, and the training
techniques employed significantly influence the effectiveness and capabilities of
foundation models.
Adapting
Adapting foundation models in AI is crucial for effective deployment in various
applications. There are different approaches to adaptation, including priming, fine-tuning,
and updating. Priming involves providing additional input, such as a prompt, to condition
the model for a specific task. Fine-tuning involves updating some or all of the model
parameters to reflect new information, while updating involves modifying the model’s
architecture to fit the specific application better.
Adaptation can be used in various settings, such as domain-specific tasks, where a
foundation model is specialized to a particular domain, improving its accuracy. It can also
be used for test-time data removal, where the model is adapted to work without access to
specific data. Another use case is editing the model behavior on specific inputs, such as
increasing or decreasing the sentiment of a text.
Several factors determine the suitability of a particular adaptation procedure for a given
application, including the available data, the desired level of accuracy, and the task
requirements. Additionally, ethical considerations, such as potential biases and fairness,
must be considered.
A long-term goal for future research in foundation model adaptation is to develop more
efficient and effective adaptation techniques that can quickly adapt a foundation model to
new tasks and domains while addressing ethical considerations. The goal is to ensure
that the adaptation process does not degrade the generalization capabilities of the model,
which is essential for its long-term usefulness. Many methods are available for adapting
foundation models, making it challenging for practitioners to choose the best one for their
specific problem or computing environment. To help with this decision-making process,
three important factors should be taken into consideration, which are computing budget,
data availability and access to foundation model gradients.
Evaluation metrics
Evaluation is an important part of machine learning models because it helps us
understand how well the model performs and how we can improve it. Evaluation is even
more important for foundation models, which are very large and complex because they
are generally used for many different tasks. However, evaluating foundation models
comes with its challenges because they need to be adapted to perform specific tasks and
can acquire unexpected skills. Additionally, documenting the performance of foundation
models can be difficult because they can be used for many different applications. To
frame the evaluation of foundation models, we identify two types of evaluation that stem

from the abstraction of these models: intrinsic evaluation of the foundation model, which
is inherently independent of any particular task as foundation models are task-agnostic,
and extrinsic evaluation of task-specific models, which relies on both the foundation
model and the method of adaptation.
Intrinsic evaluation: An approach to evaluate foundation models in AI is to assess them
based on the task associated with their training objective. For instance, a language model
trained to predict the next word given the preceding context can be evaluated by
measuring the probabilities it assigns to words given their context in held-out test data
(perplexity on language modeling benchmarks such as LAMBADA). While this approach
has shown promise in NLP, it has two primary limitations. Firstly, it needs more generality,
as foundation models trained using different, incompatible objectives cannot be easily
compared or understood within a consistent framework. Secondly, evaluation based on
the training objective relies on a proxy relationship to be meaningful. Although this proxy
relationship has worked well in some contexts, it is likely to break down when assessing
the diverse capabilities of foundation models, their behavior in different domains, and
factors beyond in-domain accuracy. Hence, two complementary approaches will be
necessary to evaluate foundation models effectively.
Extrinsic evaluation: When evaluating AI models, it’s important to measure their
performance on specific tasks, but comparing their performance can be tricky because
different models may require different resources to achieve the same level of accuracy.
This is especially true for AI foundation models, which are the basis for many task-
specific models. The proposal to address this issue is to track the resources used for pre-
training the foundation models. Another option is to consider the resources used to adapt
the foundation model to specific tasks, not just the data used for adaptation. This would
help identify the most effective adaptation methods and ensure fair model comparisons.
To evaluate task-specific models, we must consider the resources used to adapt the
foundation models to these tasks. This involves understanding all the data used to inform
adaptation and access requirements for different adaptation methods. While current
evaluations may provide some understanding of specific models, they do not necessarily
help us understand how different adaptation methods perform or which one to choose in
a given context.
System
Foundation models are so large that they often require custom software systems like
Megatron, DeepSpeed, or Mesh Transformer JAX to train efficiently at scale. These
systems are built on top of standard frameworks like PyTorch, TensorFlow, and JAX and
rely on several innovations to optimize training, such as new parallelization dimensions
like pipeline parallelism and state-sharding optimizers to reduce memory usage. JIT
compilers are used to optimize the computation graph, and optimized libraries like cuDNN
and NCCL accelerate the training process. Overall, these innovations enable the training
of large-scale foundation models but still require careful co-design across algorithms,

models, software, and hardware systems, as well as new interfaces for programming and
deploying ML applications. In addition, training large-scale models can be very time-
consuming and expensive due to the computational power required. For example,
training the GPT-3 model required over 1000 petaFLOP/s-days, a massive amount of
computational resources. As a result, training such models requires access to specialized
hardware, such as high-end GPUs or TPUs, which can be expensive and difficult to
obtain.
Moreover, once these large models are trained, they can be difficult to deploy and
maintain in production applications due to their size and complexity. Performing inference
with these models can be computationally intensive and may require specialized
hardware. Debugging and monitoring these models can also be challenging, particularly
in complex systems with many interacting components.
Addressing these challenges will require careful co-design across algorithms, models,
software, hardware systems, and new interfaces for programming and deploying ML
applications.
Using foundation models for downstream tasks
We have covered components of foundation models in AI and their significance. Now, we
will explore how to utilize these models for downstream tasks after training them. There
are two approaches to achieving this: Fine-tuning and prompting.
The crucial distinction between fine-tuning and prompting is that fine-tuning modifies the
model while prompting alters how the model is utilized by providing input prompts or
instructions to guide its output generation.
Fine-tuning
Fine-tuning is a technique where a pre-trained foundation model is further trained on a
smaller, task-specific dataset. The aim is to adapt the pre-trained model’s parameters to
the specific task. This process typically involves adding a few additional layers to the pre-
trained model and training the entire network on the new dataset. We need a smaller,
task-specific dataset of labeled examples to fine-tune a foundation model. For example, if
we use a pre-trained foundation model for sentiment analysis on a specific product review
type, we would need a dataset of product reviews for that specific product.
The advantage of fine-tuning is that it significantly reduces the labeled dataset required to
train a high-quality model for a specific task. Since the pre-trained model has already
learned general features from a large dataset, fine-tuning on a smaller dataset can
perform well with fewer labeled examples. However, it’s important to note that fine-tuning
changes the pre-trained model’s parameters, which may affect its performance on other
tasks. Additionally, fine-tuning requires access to a labeled dataset, which may only
sometimes be available or costly.

Prompting
Prompting is a technique to use a pre-trained foundation model to solve a specific task
without requiring any further model training. Instead of fine-tuning the model on a new
dataset, prompting involves providing a few task-specific examples or cues to the model,
which helps it to understand the context and make predictions accordingly. For example,
in the case of language models, the prompting technique involves providing a few task-
related examples and asking the model to fill in the blank space or predict the next token
in a sentence. The model uses the examples provided to it to understand the context and
generate the appropriate output.
Applications of foundation models across industries
Some applications of AI foundation models are:
Healthcare
Using foundation model-powered solutions in healthcare can improve efficiency and
accuracy for healthcare providers by reducing time spent editing Electronic Health
Records (EHRs) and preventing medical errors. These solutions can be used to help
providers create patient visit summaries, retrieve relevant cases and literature, and
suggest lab tests, diagnoses, treatments, and discharges. Additionally, AI foundation
models can be adapted to help surgical robots achieve accurate surgeries.
Foundation model-based solutions can also serve as an interface to patients, providing
relevant information about clinical appointments, answering patient questions about
preventive care, and providing explanatory medical information. They can also assist
assistive-care robots for patients. Furthermore, solutions based on these models can
serve as an interface to the general public, answering public health and pandemic
prevention questions. It is crucial to note that the interface must guarantee factual
accuracy to ensure public trust in medical advice.
Biomedicine
Foundation models have also positively impacted biomedical research, particularly drug
discovery and disease understanding. This is crucial as biomedical discovery is a
complex and expensive process that requires significant human resources and
experimental time. Foundation models can facilitate this process by using existing data
and published findings, accelerating the discovery of new disease drugs and treatments.
Additionally, foundation models can integrate diverse data modalities in medicine,
enabling the investigation of biomedical concepts from multiple scales and knowledge
sources. This leads to biomedical discoveries that are difficult to obtain using single-
modality data. Overall, foundation models enable knowledge transfer across modalities
and are helpful for generative tasks in biomedical research, such as generating
experimental protocols and designing molecules that work given existing data.

Law
Foundation model-powered solutions offer significant enhancements to legal businesses
by automating and optimizing various aspects of legal operations. These solutions can
streamline legal research and document analysis, accelerating the extraction of relevant
information from vast volumes of legal documents. Virtual legal assistants and chatbots
powered by foundation models provide accessible and accurate legal guidance,
improving client interactions and support. Predictive analytics based on historical legal
data enable lawyers to make informed decisions and develop effective strategies.
Contract generation and review are simplified through automated drafting and risk
identification. Additionally, foundation models facilitate compliance and regulatory
analysis, ensuring adherence to complex legal frameworks. By leveraging these
technologies, legal businesses can increase efficiency, improve decision-making, and
deliver enhanced services to clients while empowering legal professionals to focus on
high-value tasks.
Education
Foundation model-powered solutions have the potential to revolutionize education
businesses by offering personalized learning experiences, intelligent tutoring systems,
automated content generation, and advanced data analytics. These solutions can tailor
education to individual students, providing personalized recommendations and adaptive
assessments. They can assist in creating high-quality educational resources, automate
tasks like essay grading, and facilitate language learning. Additionally, foundation models
enable educators to gain valuable insights from educational data, empowering data-
informed decision-making. Virtual assistants and chatbots powered by these models can
provide round-the-clock support to students. These solutions enhance the learning
process, optimize educational outcomes, and empower educators to deliver personalized
and data-driven instruction.
Endnote
Foundation models represent a significant breakthrough in artificial intelligence. While
they have made a remarkable impact in natural language processing, they also show
promise in other domains, such as computer vision, speech recognition and
reinforcement learning. Their potential is vast, and we can expect AI foundation models to
transform how AI is used in various fields, like biomedical research and education. With
the ability to automate processes within enterprises, foundation models in AI can
significantly reduce the time, cost, and resources needed to train models for each task,
leading to faster and more efficient deployment of AI systems. We should only expect
more fascinating uses of AI foundation models as technology develops, ushering in a new
era of artificial intelligence.

Leverage the power of generative AI with our robust generative AI solutions based on
foundation models fine-tuned to your business needs. Contact LeewayHertz for all your
consultancy and development needs.

An overview of foundation models.pdf

More Related Content

What's hot

Similar to An overview of foundation models.pdf

More from StephenAmell4

Recently uploaded

An overview of foundation models.pdf