In this talk, we will explore the underlying mechanisms of ChatGPT, a large-scale language model developed by OpenAI, from the perspective of Information Retrieval (IR). We will delve into the process of training the model using massive amounts of data, the techniques used to optimize the model’s performance, and how the IR concepts such as tokenization, vectorization, and ranking are used in generating responses. We will also discuss how ChatGPT handles contextual understanding and how it leverages the power of transfer learning to generate high-quality and relevant responses. Software engineers will gain insights into how a modern conversational AI system like ChatGPT works, providing a better understanding of its strengths and limitations, and how to best integrate it into their software applications. This abstract has been fully written by ChatGPT with the simple prompt in input <Write an abstract for a talk called “How does ChatGPT work? An Information Retrieval perspective”, the audience is software engineers>.

1. Alessandro Benedetti, Director @ Sease
21/02/2023
London Information Retrieval Meetup
How ChatGPT works: an Information
Retrieval Perspective

2. ‣ Born in Tarquinia(ancient Etruscan city in Italy)
‣ R&D Software Engineer
‣ Director
‣ Master degree in Computer Science
‣ PC member for ECIR, SIGIR and Desires
‣ Apache Lucene/Solr PMC member/committer
‣ Elasticsearch expert
‣ Semantic, NLP, Machine Learning
technologies passionate
‣ Beach Volleyball player and Snowboarder
Who I am
Alessandro Benedetti

3. ● Headquarter in London/distributed
● Open Source Enthusiasts
● Apache Lucene/Solr/Es experts
● Community Contributors
● Active Researchers
● Hot Trends :
Neural Search,
Learning To Rank,
Document Similarity,
Search Quality Evaluation,
Relevance Tuning
www.sease.io
Search Services

4. Sease
● Website: www.sease.io
● Blog: https://sease.io/blog
● Github: https://github.com/SeaseLtd
● Twitter: https://twitter.com/SeaseLtd

5. T The AI techniques in ChatGPT
Supervised Fine Tuning (SFT) Model
Reward Model
Proximal Policy Optimisation (PPO)
What’s the impact on Information Retrieval?
Overview

6. ChatGPT: what is it?
● Generative Pre-training Transformer
● product capable of generating text in a wide range of styles and
for different purposes responding to a prompt
● (based on) generative AI Large Language Models
● sibling model of InstructGPT
most of our explanations come from
here

7. ChatGPT: main tech behind it
From https://openai.com/blog/chatgpt/ :
“We trained this model using Reinforcement Learning from Human
Feedback (RLHF), using the same methods as InstructGPT, but with
slight differences in the data collection setup. ”
● Supervised Learning
● Deep Learning
● Pre-trained Large Language Models
● (Deep) Reinforcement Learning from Human Feedback
(RLHF)

8. AI, Machine learning and Deep Learning
https://sease.io/2021/07/artificial-intelligence-applied-to-search-introduction.html

9. Pre-trained Large Language Models
● Transformers
● Next-token-prediction and masked-
language-modeling
● estimate the likelihood of each possible
word (in its vocabulary) given the
previous sequence
● learn the statistical structure of
language
● pre-trained on huge quantities of text
https://towardsdatascience.com/how-chatgpt-works-the-models-behind-the-bot-1ce5fca96286

10. Deep Reinforcement Learning
● Input status -> vector
● Policy network: A probability for
the actions is estimated by a policy
(neural network)
● An action is sampled from the
probability distribution
● the action is performed on the real
system
● the reward is observed
● Policy Gradients: the reward is
back-propagated to the policy(to
affect next probability estimations)
http://karpathy.github.io/2016/05/31/rl/

11. Reinforcement Learning from Human Feedback
1. Supervised fine-tuning step
a pre-trained language model is fine-tuned on a relatively small human-curated dataset, to
learn a supervised policy (the SFT model) that generates text from a prompt
2. Reward estimation step
a pre-trained language model is fine-tuned on a relatively large human-curated dataset, to
learn a reward function that generates a rating from a prompt and a response
3. Proximal Policy Optimization (PPO) step: the reward model is used to fine-tune the SFT
model. The outcome of this step is the final model (that can be iteratively improved).
● 2-3 are iteratively repeated

12. Supervised Fine-Tuning (SFT) Model
● training sample <prompt, text> ->
human-curated
○ directly from Human labellers
○ from GPT3 clients
○ 10-15.000 ‘ish samples
● starting from GPT-3.5 series.
○ Presumably the baseline model used
is the latest one text-davinci-003, a
GPT-3 model which was fine-tuned
mostly on programming code.
● expensive -> scale this up is not a
solution to improve the model

13. Reward model
● Scope: fine-tune a model that estimates a score for <prompt, text> pair
● A list of prompts is selected and the SFT model generates multiple
outputs (4…9) for each prompt.
● Training Set: Humans rank the outputs. The size of this dataset is
approximately 10 times bigger than the dataset used for the SFT model.
● The fine-tuned model takes as input a few of the SFT model outputs and
ranks them in order of preference. (Learning to Rank, sounds familiar?)
● easier for humans to rate, rather than write text
● the reward function can be further updated with users’ feedback

14. Fine-tuning the SFT model via Proximal Policy Optimization (PPO)
● PPO is a reinforcement learning algorithm.
● "on-policy"
PPO is continuously adapting the current policy
according to the actions that the agent is
taking(sampling) and the rewards it is receiving
● PPO uses a trust region optimization method -> it
constrains the change in the policy to be within a
certain distance of the previous policy in order to
ensure stability

15. Fine-tuning the SFT model via Proximal Policy Optimization (PPO)
● PPO policy is initialized from the SFT model
● value function is initialized from the reward model.
● The environment presents a random prompt and expects a
response
● Given the prompt and response, it produces a reward
● policy get updated and the episode ends.
● During the fine-tuning many episodes happen

16. Proximal Policy Optimisation 2
● PPO2 is simply an updated version of the algorithm
● optimized for GPU and better supports parallel training.
● It has a number of other differences (e.g., advantages are normalized
automatically and value functions are clipped as well), but uses the same
mathematical foundations
● OpenAI implementation -> simply remember that PPO is obsolete and
you should use PPO2.
https://openai.com/blog/openai-baselines-ppo/

17. What’s the impact on Information Retrieval?
● start from one of the fine-tuned models available online
● build datasets from your own data to additionally fine-tune them
○ e.g.
○ from a query and top-k documents, write a snippet summarizing them
○ fine-tune a reward model, to just do re-ranking of results
○ integrate it out of the box to just add on top of your results
○ … be creative!

18. References
Reinforcement Learning
http://karpathy.github.io/2016/05/31/rl/
https://towardsdatascience.com/proximal-policy-optimization-ppo-explained-abed1952457b
Short Blogs
https://openai.com/blog/chatgpt/
https://www.assemblyai.com/blog/how-chatgpt-actually-works/
https://towardsdatascience.com/how-chatgpt-works-the-models-behind-the-bot-1ce5fca96286
Detailed Resources
https://gist.github.com/veekaybee/6f8885e9906aa9c5408ebe5c7e870698
InstructGPT:
https://openai.com/blog/instruction-following/
https://arxiv.org/pdf/2203.02155.pdf

19. THANK YOU!
@seaseltd @sease-
ltd
@seaseltd @sease_ltd