More Related Content

Similar to Generative Models and ChatGPT(20)


Generative Models and ChatGPT

  1. Generative Models & ChatGPT Loic Merckel Image generated with DALL.E: “Billboard advertising a robot in a futuristic city at night with bluish neon lit” (and slightly modified with The Gimp) March 19, 2023
  2. Generative Models Models that learn from a given dataset how to generate new data instances. ♫ ♬ ♪♪ A generative models is trained using a dataset: It can subsequently generate new data instances: ♫ Music—Google Research introduced MusicLM that generates music from text. OpenAI released Jukebox, “provided with genre, artist, and lyrics as input, Jukebox outputs a new music sample produced from scratch.” Image—Both Google (Imagen) and OpenAI (DALL.E) have developed impressive models that generate novel images from text. Text—OpenAI’s ChatGPT has become widely known, but other players have similar, possibly even better, technology (including Google, with Bard, and Meta with BlenderBot3). Others—Recommender (movies, books, flight destinations), drug discovery… ■ ChatGPT: ■ Bard: ■ Recommender: ■ Drug discovery: ■ MusicLM: ■ Jukebox: ■ Imagegen: ■ DALL.E:
  3. Discriminative vs. Generative Models GLM, GBM, SVM, RF, Feedforward ANN, … GMM, VAE, GAN, Transformers … Given a set of data instances X (and a set of labels Y) “Discriminative models capture the conditional probability P(Y | X).” “Generative models capture the joint probability P(X, Y), or just P(X) if there are no labels.” Source: Y1 Y2 In a regression analysis, Y is continuous. We are then interested in the conditional expectation E(Y|X)—which depends on the conditional probability density function.
  4. Discriminative Model: 2016 Olympics Athletes ● We know the gender (y) and the weight (X) of each athlete. ● Given a weight, what is the probability of the gender, i.e., P(y | X)? ● P(y = Female | X = 50 kg) ≈ 89.6% ● P(y = Female | X = 65 kg) ≈ 60.4% ● P(y = Female | X = 100 kg) ≈ 2.6% (Obtained by fitting a simple logistic regression model) Dataset: ≈ 69 kg Female Male
  5. Generative Model: 2016 Olympics Athletes Let us imagine a situation where we have only the weights data of athletes (no gender information). We wish to generate more synthetic data that cannot easily be discerned from the real world observed data. In this toy case, a Gaussian mixture model can be fitted. Although the model identifies two components, it cannot label them. The labels (‘Female’ and ‘Male’) have been set via our knowledge of the context. Newly generated data instances
  6. Text Generation Models Image generated with DALL.E: “Writing with a fountain pen”
  7. 1966: ELIZA Image source: “While ELIZA was capable of engaging in discourse, it could not converse with true understanding. However, many early users were convinced of ELIZA's intelligence and understanding, despite Weizenbaum's insistence to the contrary.” Source: (and references therein).
  8. 2005: SCIgen - An Automatic CS Paper Generator A project using a rather rudimentary technology that aimed to "maximize amusement, rather than coherence" is still the cause of troubles today...
  9. 2017: Google Revolutionized Text Generation ■ Vaswani (2017), Attention Is All You Need ( ■ Image generated with DALL.E: “A small robot standing on the shoulder of a giant robot” (and slightly modified with The Gimp) OpenAI’s Generative Pre-trained Transformer (DALL.E, 2021; ChatGPT, 2022), as the name suggests, reposes on Transformers. Google introduced the Transformer, which rapidly became the state-of-the-art approach to solve most NLP problems.
  10. ● Kiela et al. (2021), Dynabench: Rethinking Benchmarking in NLP: ● Roser (2022), The brief history of artificial intelligence: The world has changed fast – what might be next?: Transformers 2017 Text and shapes in blue have been added to the original work from Max Roser.
  11. What are Transformers? Images source: Encoder Decoder Encoder—Self-attention mechanism: each word is encoded in a numerical sequence, which is contextualized, for this sequence is formed taking into account the other surrounding words (left and right, the “context”). Decoder—Masked self-attention mechanism (left xor right context), cross-attention and auto-regressive (re-uses its past outputs as inputs of the following steps) Transformer (1-layer ) Transformer (4-layer ) Both encoder and decoder can be used as a standalone model. Popular LLMs rely only on decoders. Whereas, e.g., machine translations may leverage the “full” transformer architecture. Source: Vaswani (2017), Attention Is All You Need (
  12. Going Further… For a rather high-level understanding: For getting your hands dirty: uS-nNzVg7IYNtrK0pFYCGY Video lecture on Embeddings: machine-learning/crash-course/ embeddings/video-lecture
  13. The Mushrooming of Transformer-based LLMs PaML (540b), LaMDA (137b) and others (Bard relies on LaMDA) OPT-IML (175b), Galactica (120b), BlenderBot3 (175b) and perhaps others? ERNIE 3.0 Titan (260b) GPT-3 (175b), GPT-3.5 (?b), more versions coming… (ChatGPT relies on GPT-3.5) BLOOM (176b) PanGu-𝛼 (200b) Jurassic-1 (178b), Jurassic-2 (?b) Exaone (300b) Megatron-Turing NLG (530b) (It appears that all those models rely only on transformer-based decoders)
  14. ChatGPT
  15. 2022: ChatGPT “ChatGPT, the popular chatbot from OpenAI, is estimated to have reached 100 million monthly active users in January, just two months after launch, making it the fastest-growing consumer application in history” Reuters, Feb 1, 2023
  16. “ChatGPT is 'not particularly innovative,' and 'nothing revolutionary', says Meta's chief AI scientist The public perceives OpenAI's ChatGPT as revolutionary, but the same techniques are being used and the same kind of work is going on at many research labs, says the deep learning pioneer.” Irrational Exuberance?,
  17. Google’s Bard, Meta’s Galactica & Baidu’s Ernie “Bard and ChatGPT are large language models, not knowledge models. They are great at generating human-sounding text, they are not good at ensuring their text is fact-based.” —Jack Krawczyk, the product lead for Bard, March 2, 2023 ( (March 16, 2023) “Baidu Inc. surged more than 14% Friday after brokerages including Citigroup tested the company’s just-unveiled ChatGPT-like service and granted it their preliminary approval.” —Bloomberg, March 17, 2023 (
  18. “we’ll see”
  19. Except where otherwise noted, this work is licensed under