Recommended
More Related Content
What's hot
What's hot (20)
Similar to GCD ChatGPT.pptx
Similar to GCD ChatGPT.pptx (20)
More from Karthikeyan VK
More from Karthikeyan VK (20)
Recently uploaded
Recently uploaded (20)
GCD ChatGPT.pptx
- 2. How ChatGPT Works: A Deep Dive into the Architecture and Mechanics of OpenAI's Language Model
- 3. How ChatGPT Works: A Deep Dive 13th May, 2023 Chennai Speaker Karthikeyan VK Designation Cloud Native Architect
- 4. Agenda ● Why ChatGPT ● What is ChatGPT ● ChatGPT vs GPT-4 ● Internal Architecture ● How it actually works ● Tools Available
- 5. Why ChatGPT ? ● Personalized assistance ● Increased efficiency ● Enhanced language translation ● Improved customer service ● Fast response times
- 6. What is ChatGPT ● Large language model which can keep context
- 7. Initial Training of ChatGPT
- 8. High Level Components ● Pre-processing ● Encoding ● Training ● Decoding ● Postprocessing
- 9. Preprocessing ● Tokenization ● Stop word removal ● Stemming /Lemmatization
- 11. Encoding ● Four types of Attributes ○ Nominal - Zipcode ○ Ordinal – Good, bad ○ Interval – 78.5 F ○ Ratio – 21 years old ● Categorical Variables Vs Numerical ● Conversion - Numerical Format
- 12. Decoding
- 13. Training ● Transformer architecture ○ NLP ○ Feed Forward Networks ○ Transformers
- 14. Transformer Architecture - NLP ○ Tokenization - ["ChatGPT", "is", "a", "language", "model", "."] ○ Part-of-speech tagging ■ "The cat sat on the mat", a POS tagger might label "The" as a determiner (DT), "cat" as a noun (NN), "sat" as a past tense verb (VBD), "on" as a preposition (IN), "the" as a determiner (DT), and "mat" as a noun (NN). ○ Named entity recognition ■ Identifying mentions of entities such as people, locations, and organizations in text. ○ Sentiment analysis
- 15. Transformer Architecture - Basics ● Feed Forward Networks
- 16. Transformer Architecture ● Self-attention mechanism in this architecture, it does a really good job of learning how to apply context in a data-driven way
- 17. Transformer Architecture ● To solve this problem, transformer models use neural networks to generate a vector called query, and a vector called key for each word. ● When the query from one word matches the key from another word, that means the second word has a relevant context for the first word. In order to provide appropriate context from the second word to the first word, a third vector called value is generated which is then combined with the first word to get a more contextualized meaning of the first word.
- 18. How it actually works
- 19. Main Take Aways ● Chat GPT is a LLM ● Chat GPT is form of probabilistic text generator ● Strength is hold to context ● Transformer Architecture – Query, Key and Value
- 21. Linked in – To Connect
- 22. Thank You
Editor's Notes
- Proximal policy optimization
- The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form
- For example, in the sentence "The cat sat on the mat", a POS tagger might label "The" as a determiner (DT), "cat" as a noun (NN), "sat" as a past tense verb (VBD), "on" as a preposition (IN), "the" as a determiner (DT), and "mat" as a noun (NN).
- Compute Query, Key, and Value Vectors: For each word in the input sequence, the model generates three vectors: a query vector, a key vector, and a value vector. These vectors are computed by multiplying the word's embedding (a vector representation of the word) by three weight matrices that the model learns during training. Calculate Attention Scores: The model calculates an "attention score" for each word in the sequence relative to every other word. This is done by taking the dot product of the query vector of the word we're focusing on and the key vector of the other word, and then applying a softmax function. This gives us a probability distribution that sums to 1, with higher values indicating words that should receive more attention. Compute Weighted Sum of Values: Each value vector is then multiplied by the corresponding softmax score (this gives higher weight to the words that should get more attention) and then summed to produce the output vector for the word we're focusing on. Generate Output: The output vector is then fed through the rest of the model (which might include additional self-attention layers, feed-forward layers, etc.).
- Compute Query, Key, and Value Vectors