SlideShare a Scribd company logo

LLMs in Production: Tooling, Process, and Team Structure

โ€ข
โ€ข
A
Aggregage

Join Dr. Greg Loughnane and Chris Alexiuk in this exciting webinar to learn all about the tooling, processes, and team structure you need to build and operate performant, reliable, and scalable production-grade LLM applications!

1 of 77
LLMs in Production:
Tooling, Process, and Team Structure
December 6th, 2023
10:30am PST
Have a question or comment for our
panelists?
Use this QR code to engage with our
speakers, or visit the link in the chat!
Having an audio issue?
Try dialing in by phone!
Dial: +1 312 626 6799
Webinar ID: 819 1469 6007
Passcode: 385318
Closed Captioning is available
for this webinar!
Our Panelists
Tony Karrer
Founder & CEO TechEmpower,
Founder & CTO Aggregage
Greg Loughnane
Founder & CEO of
AI Makerspace
Chris Alexiuk
Co-Founder & CTO at
AI Makerspace
๐ŸŽฏALIGNING OUR AIM
BY THE END OF TODAY...
Understand processes for building and
improving production LLM applications
Overview of industry-standard tooling
How to leverage LangSmith
OVERVIEW
LLM Ops, LLM OS, and โ€œThe New Stackโ€
Leading Tooling
Meet LangSmith
Conclusions, Q&A

Recommended

How Does Generative AI Actually Work? (a quick semi-technical introduction to...
How Does Generative AI Actually Work? (a quick semi-technical introduction to...How Does Generative AI Actually Work? (a quick semi-technical introduction to...
How Does Generative AI Actually Work? (a quick semi-technical introduction to...ssuser4edc93
ย 
Leveraging Generative AI & Best practices
Leveraging Generative AI & Best practicesLeveraging Generative AI & Best practices
Leveraging Generative AI & Best practicesDianaGray10
ย 
Generative AI for the rest of us
Generative AI for the rest of usGenerative AI for the rest of us
Generative AI for the rest of usMassimo Ferre'
ย 
Prompting is an art / Sztuka promptowania
Prompting is an art / Sztuka promptowaniaPrompting is an art / Sztuka promptowania
Prompting is an art / Sztuka promptowaniaMichal Jaskolski
ย 
Exploring Opportunities in the Generative AI Value Chain.pdf
Exploring Opportunities in the Generative AI Value Chain.pdfExploring Opportunities in the Generative AI Value Chain.pdf
Exploring Opportunities in the Generative AI Value Chain.pdfDung Hoang
ย 
Using the power of Generative AI at scale
Using the power of Generative AI at scaleUsing the power of Generative AI at scale
Using the power of Generative AI at scaleMaxim Salnikov
ย 
AI and ML Series - Introduction to Generative AI and LLMs - Session 1
AI and ML Series - Introduction to Generative AI and LLMs - Session 1AI and ML Series - Introduction to Generative AI and LLMs - Session 1
AI and ML Series - Introduction to Generative AI and LLMs - Session 1DianaGray10
ย 
The Future of AI is Generative not Discriminative 5/26/2021
The Future of AI is Generative not Discriminative 5/26/2021The Future of AI is Generative not Discriminative 5/26/2021
The Future of AI is Generative not Discriminative 5/26/2021Steve Omohundro
ย 

More Related Content

What's hot

Large Language Models Bootcamp
Large Language Models BootcampLarge Language Models Bootcamp
Large Language Models BootcampData Science Dojo
ย 
Generative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdfGenerative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdfLiming Zhu
ย 
Intro to LLMs
Intro to LLMsIntro to LLMs
Intro to LLMsLoic Merckel
ย 
The Rise of the LLMsโ€Š-โ€ŠHow I Learned to Stop Worrying & Love theย GPT!
The Rise of the LLMsโ€Š-โ€ŠHow I Learned to Stop Worrying & Love theย GPT!The Rise of the LLMsโ€Š-โ€ŠHow I Learned to Stop Worrying & Love theย GPT!
The Rise of the LLMsโ€Š-โ€ŠHow I Learned to Stop Worrying & Love theย GPT!taozen
ย 
Generative AI at the edge.pdf
Generative AI at the edge.pdfGenerative AI at the edge.pdf
Generative AI at the edge.pdfQualcomm Research
ย 
presentation.pdf
presentation.pdfpresentation.pdf
presentation.pdfcaa28steve
ย 
Building and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache AirflowBuilding and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache AirflowKaxil Naik
ย 
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer RoadmapEpisode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer RoadmapAnant Corporation
ย 
Unlocking the Power of Generative AI An Executive's Guide.pdf
Unlocking the Power of Generative AI An Executive's Guide.pdfUnlocking the Power of Generative AI An Executive's Guide.pdf
Unlocking the Power of Generative AI An Executive's Guide.pdfPremNaraindas1
ย 
Let's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchersLet's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchersSteven Van Vaerenbergh
ย 
Using Generative AI
Using Generative AIUsing Generative AI
Using Generative AIMark DeLoura
ย 
Large Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdfLarge Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdfDavid Rostcheck
ย 
Building, Evaluating, and Optimizing your RAG App for Production
Building, Evaluating, and Optimizing your RAG App for ProductionBuilding, Evaluating, and Optimizing your RAG App for Production
Building, Evaluating, and Optimizing your RAG App for ProductionSri Ambati
ย 
Use Case Patterns for LLM Applications (1).pdf
Use Case Patterns for LLM Applications (1).pdfUse Case Patterns for LLM Applications (1).pdf
Use Case Patterns for LLM Applications (1).pdfM Waleed Kadous
ย 
LLMs_talk_March23.pdf
LLMs_talk_March23.pdfLLMs_talk_March23.pdf
LLMs_talk_March23.pdfChaoYang81
ย 
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdfRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdfPo-Chuan Chen
ย 
๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ: ๐‚๐ก๐š๐ง๐ ๐ข๐ง๐  ๐‡๐จ๐ฐ ๐๐ฎ๐ฌ๐ข๐ง๐ž๐ฌ๐ฌ ๐ˆ๐ง๐ง๐จ๐ฏ๐š๐ญ๐ž๐ฌ ๐š๐ง๐ ๐Ž๐ฉ๐ž๐ซ๐š๐ญ๐ž๐ฌ
๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ: ๐‚๐ก๐š๐ง๐ ๐ข๐ง๐  ๐‡๐จ๐ฐ ๐๐ฎ๐ฌ๐ข๐ง๐ž๐ฌ๐ฌ ๐ˆ๐ง๐ง๐จ๐ฏ๐š๐ญ๐ž๐ฌ ๐š๐ง๐ ๐Ž๐ฉ๐ž๐ซ๐š๐ญ๐ž๐ฌ๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ: ๐‚๐ก๐š๐ง๐ ๐ข๐ง๐  ๐‡๐จ๐ฐ ๐๐ฎ๐ฌ๐ข๐ง๐ž๐ฌ๐ฌ ๐ˆ๐ง๐ง๐จ๐ฏ๐š๐ญ๐ž๐ฌ ๐š๐ง๐ ๐Ž๐ฉ๐ž๐ซ๐š๐ญ๐ž๐ฌ
๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ: ๐‚๐ก๐š๐ง๐ ๐ข๐ง๐  ๐‡๐จ๐ฐ ๐๐ฎ๐ฌ๐ข๐ง๐ž๐ฌ๐ฌ ๐ˆ๐ง๐ง๐จ๐ฏ๐š๐ญ๐ž๐ฌ ๐š๐ง๐ ๐Ž๐ฉ๐ž๐ซ๐š๐ญ๐ž๐ฌVINCI Digital - Industrial IoT (IIoT) Strategic Advisory
ย 
The current state of generative AI
The current state of generative AIThe current state of generative AI
The current state of generative AIBenjaminlapid1
ย 
LLMs Bootcamp
LLMs BootcampLLMs Bootcamp
LLMs BootcampFiza987241
ย 

What's hot (20)

Large Language Models Bootcamp
Large Language Models BootcampLarge Language Models Bootcamp
Large Language Models Bootcamp
ย 
Generative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdfGenerative-AI-in-enterprise-20230615.pdf
Generative-AI-in-enterprise-20230615.pdf
ย 
Intro to LLMs
Intro to LLMsIntro to LLMs
Intro to LLMs
ย 
The Rise of the LLMsโ€Š-โ€ŠHow I Learned to Stop Worrying & Love theย GPT!
The Rise of the LLMsโ€Š-โ€ŠHow I Learned to Stop Worrying & Love theย GPT!The Rise of the LLMsโ€Š-โ€ŠHow I Learned to Stop Worrying & Love theย GPT!
The Rise of the LLMsโ€Š-โ€ŠHow I Learned to Stop Worrying & Love theย GPT!
ย 
Generative AI at the edge.pdf
Generative AI at the edge.pdfGenerative AI at the edge.pdf
Generative AI at the edge.pdf
ย 
presentation.pdf
presentation.pdfpresentation.pdf
presentation.pdf
ย 
Building and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache AirflowBuilding and deploying LLM applications with Apache Airflow
Building and deploying LLM applications with Apache Airflow
ย 
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer RoadmapEpisode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
Episode 2: The LLM / GPT / AI Prompt / Data Engineer Roadmap
ย 
Unlocking the Power of Generative AI An Executive's Guide.pdf
Unlocking the Power of Generative AI An Executive's Guide.pdfUnlocking the Power of Generative AI An Executive's Guide.pdf
Unlocking the Power of Generative AI An Executive's Guide.pdf
ย 
Let's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchersLet's talk about GPT: A crash course in Generative AI for researchers
Let's talk about GPT: A crash course in Generative AI for researchers
ย 
Using Generative AI
Using Generative AIUsing Generative AI
Using Generative AI
ย 
Large Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdfLarge Language Models - Chat AI.pdf
Large Language Models - Chat AI.pdf
ย 
Building, Evaluating, and Optimizing your RAG App for Production
Building, Evaluating, and Optimizing your RAG App for ProductionBuilding, Evaluating, and Optimizing your RAG App for Production
Building, Evaluating, and Optimizing your RAG App for Production
ย 
Use Case Patterns for LLM Applications (1).pdf
Use Case Patterns for LLM Applications (1).pdfUse Case Patterns for LLM Applications (1).pdf
Use Case Patterns for LLM Applications (1).pdf
ย 
LLMs_talk_March23.pdf
LLMs_talk_March23.pdfLLMs_talk_March23.pdf
LLMs_talk_March23.pdf
ย 
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdfRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf
ย 
๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ: ๐‚๐ก๐š๐ง๐ ๐ข๐ง๐  ๐‡๐จ๐ฐ ๐๐ฎ๐ฌ๐ข๐ง๐ž๐ฌ๐ฌ ๐ˆ๐ง๐ง๐จ๐ฏ๐š๐ญ๐ž๐ฌ ๐š๐ง๐ ๐Ž๐ฉ๐ž๐ซ๐š๐ญ๐ž๐ฌ
๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ: ๐‚๐ก๐š๐ง๐ ๐ข๐ง๐  ๐‡๐จ๐ฐ ๐๐ฎ๐ฌ๐ข๐ง๐ž๐ฌ๐ฌ ๐ˆ๐ง๐ง๐จ๐ฏ๐š๐ญ๐ž๐ฌ ๐š๐ง๐ ๐Ž๐ฉ๐ž๐ซ๐š๐ญ๐ž๐ฌ๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ: ๐‚๐ก๐š๐ง๐ ๐ข๐ง๐  ๐‡๐จ๐ฐ ๐๐ฎ๐ฌ๐ข๐ง๐ž๐ฌ๐ฌ ๐ˆ๐ง๐ง๐จ๐ฏ๐š๐ญ๐ž๐ฌ ๐š๐ง๐ ๐Ž๐ฉ๐ž๐ซ๐š๐ญ๐ž๐ฌ
๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐€๐ˆ: ๐‚๐ก๐š๐ง๐ ๐ข๐ง๐  ๐‡๐จ๐ฐ ๐๐ฎ๐ฌ๐ข๐ง๐ž๐ฌ๐ฌ ๐ˆ๐ง๐ง๐จ๐ฏ๐š๐ญ๐ž๐ฌ ๐š๐ง๐ ๐Ž๐ฉ๐ž๐ซ๐š๐ญ๐ž๐ฌ
ย 
How will development change with LLMs
How will development change with LLMsHow will development change with LLMs
How will development change with LLMs
ย 
The current state of generative AI
The current state of generative AIThe current state of generative AI
The current state of generative AI
ย 
LLMs Bootcamp
LLMs BootcampLLMs Bootcamp
LLMs Bootcamp
ย 

Similar to LLMs in Production: Tooling, Process, and Team Structure

The Semantic Knowledge Graph
The Semantic Knowledge GraphThe Semantic Knowledge Graph
The Semantic Knowledge GraphTrey Grainger
ย 
How Azure helps to build better business processes and customer experiences w...
How Azure helps to build better business processes and customer experiences w...How Azure helps to build better business processes and customer experiences w...
How Azure helps to build better business processes and customer experiences w...Maxim Salnikov
ย 
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and VocabulariesHaystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and VocabulariesMax Irwin
ย 
Tensors for topic modeling and deep learning on AWS Sagemaker
Tensors for topic modeling and deep learning on AWS SagemakerTensors for topic modeling and deep learning on AWS Sagemaker
Tensors for topic modeling and deep learning on AWS SagemakerAnima Anandkumar
ย 
Fishing Graphs in a Hadoop Data Lake
Fishing Graphs in a Hadoop Data LakeFishing Graphs in a Hadoop Data Lake
Fishing Graphs in a Hadoop Data LakeArangoDB Database
ย 
Overview of Artificial Intelligence at AWS - DevDay Los Angeles 2017
Overview of Artificial Intelligence at AWS - DevDay Los Angeles 2017Overview of Artificial Intelligence at AWS - DevDay Los Angeles 2017
Overview of Artificial Intelligence at AWS - DevDay Los Angeles 2017Amazon Web Services
ย 
An Overview of AI at AWS - Amazon Lex, Amazon Polly, Amazon Rekognition - Dev...
An Overview of AI at AWS - Amazon Lex, Amazon Polly, Amazon Rekognition - Dev...An Overview of AI at AWS - Amazon Lex, Amazon Polly, Amazon Rekognition - Dev...
An Overview of AI at AWS - Amazon Lex, Amazon Polly, Amazon Rekognition - Dev...Amazon Web Services
ย 
Vitalii Bondarenko, Pavlo Shtogryn "Cloud AI Solutions for Enterprises on AWS...
Vitalii Bondarenko, Pavlo Shtogryn "Cloud AI Solutions for Enterprises on AWS...Vitalii Bondarenko, Pavlo Shtogryn "Cloud AI Solutions for Enterprises on AWS...
Vitalii Bondarenko, Pavlo Shtogryn "Cloud AI Solutions for Enterprises on AWS...Lviv Startup Club
ย 
The Future of Data Science
The Future of Data ScienceThe Future of Data Science
The Future of Data ScienceDataWorks Summit
ย 
AI and Python: Developing a Conversational Interface using Python
AI and Python: Developing a Conversational Interface using PythonAI and Python: Developing a Conversational Interface using Python
AI and Python: Developing a Conversational Interface using Pythonamyiris
ย 
The need for sophistication in modern search engine implementations
The need for sophistication in modern search engine implementationsThe need for sophistication in modern search engine implementations
The need for sophistication in modern search engine implementationsBen DeMott
ย 
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and SparkVital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and SparkVital.AI
ย 
Ssas dmx ile kurum iรงi verilerin iฬ‡ลŸlenmesi
Ssas dmx ile kurum iรงi verilerin iฬ‡ลŸlenmesiSsas dmx ile kurum iรงi verilerin iฬ‡ลŸlenmesi
Ssas dmx ile kurum iรงi verilerin iฬ‡ลŸlenmesiKoray Kocabas
ย 
Discovering User's Topics of Interest in Recommender Systems
Discovering User's Topics of Interest in Recommender SystemsDiscovering User's Topics of Interest in Recommender Systems
Discovering User's Topics of Interest in Recommender SystemsGabriel Moreira
ย 
The Relevance of the Apache Solr Semantic Knowledge Graph
The Relevance of the Apache Solr Semantic Knowledge GraphThe Relevance of the Apache Solr Semantic Knowledge Graph
The Relevance of the Apache Solr Semantic Knowledge GraphTrey Grainger
ย 
Neural Models for Information Retrieval
Neural Models for Information RetrievalNeural Models for Information Retrieval
Neural Models for Information RetrievalBhaskar Mitra
ย 
OSMC 2023 | Experiments with OpenSearch and AI by Jochen Kressin & Leanne La...
OSMC 2023 | Experiments with OpenSearch and AI by Jochen Kressin &  Leanne La...OSMC 2023 | Experiments with OpenSearch and AI by Jochen Kressin &  Leanne La...
OSMC 2023 | Experiments with OpenSearch and AI by Jochen Kressin & Leanne La...NETWAYS
ย 
Cis 555 Week 4 Assignment 2 Automated Teller Machine (Atm)...
Cis 555 Week 4 Assignment 2 Automated Teller Machine (Atm)...Cis 555 Week 4 Assignment 2 Automated Teller Machine (Atm)...
Cis 555 Week 4 Assignment 2 Automated Teller Machine (Atm)...Karen Thompson
ย 

Similar to LLMs in Production: Tooling, Process, and Team Structure (20)

The Semantic Knowledge Graph
The Semantic Knowledge GraphThe Semantic Knowledge Graph
The Semantic Knowledge Graph
ย 
How Azure helps to build better business processes and customer experiences w...
How Azure helps to build better business processes and customer experiences w...How Azure helps to build better business processes and customer experiences w...
How Azure helps to build better business processes and customer experiences w...
ย 
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and VocabulariesHaystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
Haystack 2018 - Algorithmic Extraction of Keywords Concepts and Vocabularies
ย 
Tensors for topic modeling and deep learning on AWS Sagemaker
Tensors for topic modeling and deep learning on AWS SagemakerTensors for topic modeling and deep learning on AWS Sagemaker
Tensors for topic modeling and deep learning on AWS Sagemaker
ย 
Fishing Graphs in a Hadoop Data Lake
Fishing Graphs in a Hadoop Data LakeFishing Graphs in a Hadoop Data Lake
Fishing Graphs in a Hadoop Data Lake
ย 
Overview of Artificial Intelligence at AWS - DevDay Los Angeles 2017
Overview of Artificial Intelligence at AWS - DevDay Los Angeles 2017Overview of Artificial Intelligence at AWS - DevDay Los Angeles 2017
Overview of Artificial Intelligence at AWS - DevDay Los Angeles 2017
ย 
An Overview of AI at AWS - Amazon Lex, Amazon Polly, Amazon Rekognition - Dev...
An Overview of AI at AWS - Amazon Lex, Amazon Polly, Amazon Rekognition - Dev...An Overview of AI at AWS - Amazon Lex, Amazon Polly, Amazon Rekognition - Dev...
An Overview of AI at AWS - Amazon Lex, Amazon Polly, Amazon Rekognition - Dev...
ย 
Vitalii Bondarenko, Pavlo Shtogryn "Cloud AI Solutions for Enterprises on AWS...
Vitalii Bondarenko, Pavlo Shtogryn "Cloud AI Solutions for Enterprises on AWS...Vitalii Bondarenko, Pavlo Shtogryn "Cloud AI Solutions for Enterprises on AWS...
Vitalii Bondarenko, Pavlo Shtogryn "Cloud AI Solutions for Enterprises on AWS...
ย 
The Future of Data Science
The Future of Data ScienceThe Future of Data Science
The Future of Data Science
ย 
AI and Python: Developing a Conversational Interface using Python
AI and Python: Developing a Conversational Interface using PythonAI and Python: Developing a Conversational Interface using Python
AI and Python: Developing a Conversational Interface using Python
ย 
The need for sophistication in modern search engine implementations
The need for sophistication in modern search engine implementationsThe need for sophistication in modern search engine implementations
The need for sophistication in modern search engine implementations
ย 
introduction Azure OpenAI by Usama wahab khan
introduction  Azure OpenAI by Usama wahab khanintroduction  Azure OpenAI by Usama wahab khan
introduction Azure OpenAI by Usama wahab khan
ย 
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and SparkVital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
Vital AI MetaQL: Queries Across NoSQL, SQL, Sparql, and Spark
ย 
Ssas dmx ile kurum iรงi verilerin iฬ‡ลŸlenmesi
Ssas dmx ile kurum iรงi verilerin iฬ‡ลŸlenmesiSsas dmx ile kurum iรงi verilerin iฬ‡ลŸlenmesi
Ssas dmx ile kurum iรงi verilerin iฬ‡ลŸlenmesi
ย 
Discovering User's Topics of Interest in Recommender Systems
Discovering User's Topics of Interest in Recommender SystemsDiscovering User's Topics of Interest in Recommender Systems
Discovering User's Topics of Interest in Recommender Systems
ย 
The Relevance of the Apache Solr Semantic Knowledge Graph
The Relevance of the Apache Solr Semantic Knowledge GraphThe Relevance of the Apache Solr Semantic Knowledge Graph
The Relevance of the Apache Solr Semantic Knowledge Graph
ย 
Ad507
Ad507Ad507
Ad507
ย 
Neural Models for Information Retrieval
Neural Models for Information RetrievalNeural Models for Information Retrieval
Neural Models for Information Retrieval
ย 
OSMC 2023 | Experiments with OpenSearch and AI by Jochen Kressin & Leanne La...
OSMC 2023 | Experiments with OpenSearch and AI by Jochen Kressin &  Leanne La...OSMC 2023 | Experiments with OpenSearch and AI by Jochen Kressin &  Leanne La...
OSMC 2023 | Experiments with OpenSearch and AI by Jochen Kressin & Leanne La...
ย 
Cis 555 Week 4 Assignment 2 Automated Teller Machine (Atm)...
Cis 555 Week 4 Assignment 2 Automated Teller Machine (Atm)...Cis 555 Week 4 Assignment 2 Automated Teller Machine (Atm)...
Cis 555 Week 4 Assignment 2 Automated Teller Machine (Atm)...
ย 

More from Aggregage

Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Aggregage
ย 
How Personalized Customer Experiences Drive Retail Growth and Revenue
How Personalized Customer Experiences Drive Retail Growth and RevenueHow Personalized Customer Experiences Drive Retail Growth and Revenue
How Personalized Customer Experiences Drive Retail Growth and RevenueAggregage
ย 
Your Expert Guide to CX Orchestration & Enhancing Customer Journeys
Your Expert Guide to CX Orchestration & Enhancing Customer JourneysYour Expert Guide to CX Orchestration & Enhancing Customer Journeys
Your Expert Guide to CX Orchestration & Enhancing Customer JourneysAggregage
ย 
The Resurgence of Direct Mail as a Growth Marketing Strategy
The Resurgence of Direct Mail as a Growth Marketing StrategyThe Resurgence of Direct Mail as a Growth Marketing Strategy
The Resurgence of Direct Mail as a Growth Marketing StrategyAggregage
ย 
HR Pro Playbook: How to Navigate Global Workforce Expansion with Confidence
HR Pro Playbook: How to Navigate Global Workforce Expansion with ConfidenceHR Pro Playbook: How to Navigate Global Workforce Expansion with Confidence
HR Pro Playbook: How to Navigate Global Workforce Expansion with ConfidenceAggregage
ย 
A Tale of Two Case Studies: Using LLMs in Production
A Tale of Two Case Studies: Using LLMs in ProductionA Tale of Two Case Studies: Using LLMs in Production
A Tale of Two Case Studies: Using LLMs in ProductionAggregage
ย 
SME Relationships: Proven Solutions for Seamless Collaboration and Success
SME Relationships: Proven Solutions for Seamless Collaboration and SuccessSME Relationships: Proven Solutions for Seamless Collaboration and Success
SME Relationships: Proven Solutions for Seamless Collaboration and SuccessAggregage
ย 
LLMOps for Your Data: Best Practices to Ensure Safety, Quality, and Cost
LLMOps for Your Data: Best Practices to Ensure Safety, Quality, and CostLLMOps for Your Data: Best Practices to Ensure Safety, Quality, and Cost
LLMOps for Your Data: Best Practices to Ensure Safety, Quality, and CostAggregage
ย 
The Truth About Customer Journey Mapping
The Truth About Customer Journey MappingThe Truth About Customer Journey Mapping
The Truth About Customer Journey MappingAggregage
ย 
The Future of Customer Loyalty: Rewards, Retention, Appreciation, and More
The Future of Customer Loyalty: Rewards, Retention, Appreciation, and MoreThe Future of Customer Loyalty: Rewards, Retention, Appreciation, and More
The Future of Customer Loyalty: Rewards, Retention, Appreciation, and MoreAggregage
ย 
Predictions You Can Rely On: How Data Drives Successful Financial Forecasting
Predictions You Can Rely On: How Data Drives Successful Financial ForecastingPredictions You Can Rely On: How Data Drives Successful Financial Forecasting
Predictions You Can Rely On: How Data Drives Successful Financial ForecastingAggregage
ย 
Software Composition Analysis: The New Armor for Your Cybersecurity
Software Composition Analysis: The New Armor for Your CybersecuritySoftware Composition Analysis: The New Armor for Your Cybersecurity
Software Composition Analysis: The New Armor for Your CybersecurityAggregage
ย 
The 4 Dysfunctions of Recruiting & How You Can Successfully Overcome Them
The 4 Dysfunctions of Recruiting & How You Can Successfully Overcome ThemThe 4 Dysfunctions of Recruiting & How You Can Successfully Overcome Them
The 4 Dysfunctions of Recruiting & How You Can Successfully Overcome ThemAggregage
ย 
How to Design a Seamless & Personalized Digital Donor Journey
How to Design a Seamless & Personalized Digital Donor JourneyHow to Design a Seamless & Personalized Digital Donor Journey
How to Design a Seamless & Personalized Digital Donor JourneyAggregage
ย 
2 Retail Sectors That Are Completely Changing the Game: FMCG & Q-Commerce
2 Retail Sectors That Are Completely Changing the Game: FMCG & Q-Commerce2 Retail Sectors That Are Completely Changing the Game: FMCG & Q-Commerce
2 Retail Sectors That Are Completely Changing the Game: FMCG & Q-CommerceAggregage
ย 
Let's Get Physical: How to Blend Direct Mail Marketing with Your Digital Stra...
Let's Get Physical: How to Blend Direct Mail Marketing with Your Digital Stra...Let's Get Physical: How to Blend Direct Mail Marketing with Your Digital Stra...
Let's Get Physical: How to Blend Direct Mail Marketing with Your Digital Stra...Aggregage
ย 
Banking on Loyalty: Holistic Financial Advice for Unparalleled Business Growth
Banking on Loyalty: Holistic Financial Advice for Unparalleled Business GrowthBanking on Loyalty: Holistic Financial Advice for Unparalleled Business Growth
Banking on Loyalty: Holistic Financial Advice for Unparalleled Business GrowthAggregage
ย 
Keep the Competitive Edge and Reduce Churn
Keep the Competitive Edge and Reduce ChurnKeep the Competitive Edge and Reduce Churn
Keep the Competitive Edge and Reduce ChurnAggregage
ย 
Mastering Supply Chain Sustainability: Where Does ESG Fit In?
Mastering Supply Chain Sustainability: Where Does ESG Fit In?Mastering Supply Chain Sustainability: Where Does ESG Fit In?
Mastering Supply Chain Sustainability: Where Does ESG Fit In?Aggregage
ย 
From Complexity to Clarity: Strategies for Effective Compliance and Security ...
From Complexity to Clarity: Strategies for Effective Compliance and Security ...From Complexity to Clarity: Strategies for Effective Compliance and Security ...
From Complexity to Clarity: Strategies for Effective Compliance and Security ...Aggregage
ย 

More from Aggregage (20)

Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
Can Brain Science Actually Help Make Your Training & Teaching "Stick"?
ย 
How Personalized Customer Experiences Drive Retail Growth and Revenue
How Personalized Customer Experiences Drive Retail Growth and RevenueHow Personalized Customer Experiences Drive Retail Growth and Revenue
How Personalized Customer Experiences Drive Retail Growth and Revenue
ย 
Your Expert Guide to CX Orchestration & Enhancing Customer Journeys
Your Expert Guide to CX Orchestration & Enhancing Customer JourneysYour Expert Guide to CX Orchestration & Enhancing Customer Journeys
Your Expert Guide to CX Orchestration & Enhancing Customer Journeys
ย 
The Resurgence of Direct Mail as a Growth Marketing Strategy
The Resurgence of Direct Mail as a Growth Marketing StrategyThe Resurgence of Direct Mail as a Growth Marketing Strategy
The Resurgence of Direct Mail as a Growth Marketing Strategy
ย 
HR Pro Playbook: How to Navigate Global Workforce Expansion with Confidence
HR Pro Playbook: How to Navigate Global Workforce Expansion with ConfidenceHR Pro Playbook: How to Navigate Global Workforce Expansion with Confidence
HR Pro Playbook: How to Navigate Global Workforce Expansion with Confidence
ย 
A Tale of Two Case Studies: Using LLMs in Production
A Tale of Two Case Studies: Using LLMs in ProductionA Tale of Two Case Studies: Using LLMs in Production
A Tale of Two Case Studies: Using LLMs in Production
ย 
SME Relationships: Proven Solutions for Seamless Collaboration and Success
SME Relationships: Proven Solutions for Seamless Collaboration and SuccessSME Relationships: Proven Solutions for Seamless Collaboration and Success
SME Relationships: Proven Solutions for Seamless Collaboration and Success
ย 
LLMOps for Your Data: Best Practices to Ensure Safety, Quality, and Cost
LLMOps for Your Data: Best Practices to Ensure Safety, Quality, and CostLLMOps for Your Data: Best Practices to Ensure Safety, Quality, and Cost
LLMOps for Your Data: Best Practices to Ensure Safety, Quality, and Cost
ย 
The Truth About Customer Journey Mapping
The Truth About Customer Journey MappingThe Truth About Customer Journey Mapping
The Truth About Customer Journey Mapping
ย 
The Future of Customer Loyalty: Rewards, Retention, Appreciation, and More
The Future of Customer Loyalty: Rewards, Retention, Appreciation, and MoreThe Future of Customer Loyalty: Rewards, Retention, Appreciation, and More
The Future of Customer Loyalty: Rewards, Retention, Appreciation, and More
ย 
Predictions You Can Rely On: How Data Drives Successful Financial Forecasting
Predictions You Can Rely On: How Data Drives Successful Financial ForecastingPredictions You Can Rely On: How Data Drives Successful Financial Forecasting
Predictions You Can Rely On: How Data Drives Successful Financial Forecasting
ย 
Software Composition Analysis: The New Armor for Your Cybersecurity
Software Composition Analysis: The New Armor for Your CybersecuritySoftware Composition Analysis: The New Armor for Your Cybersecurity
Software Composition Analysis: The New Armor for Your Cybersecurity
ย 
The 4 Dysfunctions of Recruiting & How You Can Successfully Overcome Them
The 4 Dysfunctions of Recruiting & How You Can Successfully Overcome ThemThe 4 Dysfunctions of Recruiting & How You Can Successfully Overcome Them
The 4 Dysfunctions of Recruiting & How You Can Successfully Overcome Them
ย 
How to Design a Seamless & Personalized Digital Donor Journey
How to Design a Seamless & Personalized Digital Donor JourneyHow to Design a Seamless & Personalized Digital Donor Journey
How to Design a Seamless & Personalized Digital Donor Journey
ย 
2 Retail Sectors That Are Completely Changing the Game: FMCG & Q-Commerce
2 Retail Sectors That Are Completely Changing the Game: FMCG & Q-Commerce2 Retail Sectors That Are Completely Changing the Game: FMCG & Q-Commerce
2 Retail Sectors That Are Completely Changing the Game: FMCG & Q-Commerce
ย 
Let's Get Physical: How to Blend Direct Mail Marketing with Your Digital Stra...
Let's Get Physical: How to Blend Direct Mail Marketing with Your Digital Stra...Let's Get Physical: How to Blend Direct Mail Marketing with Your Digital Stra...
Let's Get Physical: How to Blend Direct Mail Marketing with Your Digital Stra...
ย 
Banking on Loyalty: Holistic Financial Advice for Unparalleled Business Growth
Banking on Loyalty: Holistic Financial Advice for Unparalleled Business GrowthBanking on Loyalty: Holistic Financial Advice for Unparalleled Business Growth
Banking on Loyalty: Holistic Financial Advice for Unparalleled Business Growth
ย 
Keep the Competitive Edge and Reduce Churn
Keep the Competitive Edge and Reduce ChurnKeep the Competitive Edge and Reduce Churn
Keep the Competitive Edge and Reduce Churn
ย 
Mastering Supply Chain Sustainability: Where Does ESG Fit In?
Mastering Supply Chain Sustainability: Where Does ESG Fit In?Mastering Supply Chain Sustainability: Where Does ESG Fit In?
Mastering Supply Chain Sustainability: Where Does ESG Fit In?
ย 
From Complexity to Clarity: Strategies for Effective Compliance and Security ...
From Complexity to Clarity: Strategies for Effective Compliance and Security ...From Complexity to Clarity: Strategies for Effective Compliance and Security ...
From Complexity to Clarity: Strategies for Effective Compliance and Security ...
ย 

Recently uploaded

Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...2toLead Limited
ย 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceVijayananda Mohire
ย 
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...BookNet Canada
ย 
How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxInfosec
ย 
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerCentralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerSaiLinnThu2
ย 
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...htrindia
ย 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceSusan Ibach
ย 
Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfMostafa Higazy
ย 
Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Product School
ย 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeJosh Gellers
ย 
PrismCRM-RealEstate-SalesCRM_byCode5Company
PrismCRM-RealEstate-SalesCRM_byCode5CompanyPrismCRM-RealEstate-SalesCRM_byCode5Company
PrismCRM-RealEstate-SalesCRM_byCode5CompanyMustafa KuฤŸu
ย 
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxGraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxNeo4j
ย 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Product School
ย 
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Jay Zhao
ย 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoProduct School
ย 
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner,  Challenge Like a VC by former CPO, TripadvisorAct Like an Owner,  Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner, Challenge Like a VC by former CPO, TripadvisorProduct School
ย 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...Neo4j
ย 
AMER Introduction to ThousandEyes Webinar
AMER Introduction to ThousandEyes WebinarAMER Introduction to ThousandEyes Webinar
AMER Introduction to ThousandEyes WebinarThousandEyes
ย 
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024BookNet Canada
ย 

Recently uploaded (20)

Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
ย 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial Intelligence
ย 
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
ย 
How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptx
ย 
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerCentralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
ย 
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
HBR SERIES METAL HOUSED RESISTORS POWER ELECTRICAL ABSORBS HIGH CURRENT DURIN...
ย 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data science
ย 
Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdf
ย 
Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...
ย 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human Justice
ย 
PrismCRM-RealEstate-SalesCRM_byCode5Company
PrismCRM-RealEstate-SalesCRM_byCode5CompanyPrismCRM-RealEstate-SalesCRM_byCode5Company
PrismCRM-RealEstate-SalesCRM_byCode5Company
ย 
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxGraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
ย 
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
ย 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
ย 
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
ย 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
ย 
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner,  Challenge Like a VC by former CPO, TripadvisorAct Like an Owner,  Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
ย 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ย 
AMER Introduction to ThousandEyes Webinar
AMER Introduction to ThousandEyes WebinarAMER Introduction to ThousandEyes Webinar
AMER Introduction to ThousandEyes Webinar
ย 
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
ย 

LLMs in Production: Tooling, Process, and Team Structure

  • 1. LLMs in Production: Tooling, Process, and Team Structure December 6th, 2023 10:30am PST
  • 2. Have a question or comment for our panelists? Use this QR code to engage with our speakers, or visit the link in the chat! Having an audio issue? Try dialing in by phone! Dial: +1 312 626 6799 Webinar ID: 819 1469 6007 Passcode: 385318 Closed Captioning is available for this webinar!
  • 3. Our Panelists Tony Karrer Founder & CEO TechEmpower, Founder & CTO Aggregage Greg Loughnane Founder & CEO of AI Makerspace Chris Alexiuk Co-Founder & CTO at AI Makerspace
  • 5. BY THE END OF TODAY... Understand processes for building and improving production LLM applications Overview of industry-standard tooling How to leverage LangSmith
  • 6. OVERVIEW LLM Ops, LLM OS, and โ€œThe New Stackโ€ Leading Tooling Meet LangSmith Conclusions, Q&A
  • 8. What is LLM Ops?
  • 15. ENTERPRISE BUILDS Synthetic data, closed-source models 1. Baseline performance a. Open-source models 2. Add your private data a. Iterate 3. Optimize models, data, metrics, inference, efficiency a.
  • 16. NEXT-LEVEL โ“Some additional questions: On premise hardware? What scale and speed? Training proprietary LLM?
  • 17. Small Language Models (SLMs) Efficiency Transparency Accuracy Security
  • 19. ๐ŸงฉPROTOTYPING LLM APPS Prompt Engineering 1. Question Answering Systems 2. Fine-Tuning Models 3.
  • 23. โ€œTHE NEW STACK AND OPS FOR AIโ€ #LLM OPS
  • 24. USER EXPERIENCE Control for uncertainty (PE) Build guardrails for steerability and safety (Harm/Help)
  • 25. MODEL CONSISTENCY Constrain model behavior (FT) Ground the model (RAG)
  • 26. EVALUATING PERF Create evaluation suites (RAGAS) Use model-graded evals (GPT-4)
  • 27. LATENCY AND COST Use semantic caching (Prompts) Route to cheaper models (FT)
  • 32. OUR LLM OPS CURRICULUM ๐Ÿง‘โ€๐Ÿ’ปBuilding LLM Applications in Pure Python 1. ๐Ÿ”—LangChain Powered RAG and Advanced Retrieval 2. ๐Ÿฆ™Open-Source Production RAG with LlamaIndex 3. ๐Ÿ•ด๏ธAgents, ๐Ÿง‘โ€๐Ÿ’ปHackathon, and ๐Ÿง‘โ€๐ŸซDemo Day! 4.
  • 33. OUR LLM OPS CURRICULUM
  • 35. LLMs: OpenAI GPT-4-Turbo, Mistral 7B Evaluation: RAGAS, Built-in metrics Visibility: Weights and Biases Infrastructure: LangChain LlamaIndex Vector Database: Pinecone FAISS Embedding Models: OpenAI Ada MTEB Leaderboard User Interface: Chainlit Deployment: Hugging Face Amazon Bedrock
  • 37. BASIC RAG + ADVANCED RAG W/ LANGSMITH
  • 40. RETREIVAL AUGMENTED GENERATION Retrieval Finds references in your documents Augmented Adds references to your prompts Generation Improves answers to questions!
  • 41. SPECIALIZED DOMAINS Jargon, e.g.; Legal, healthcare, financial, insurance, government, research Alignment With common language that humans use
  • 44. ๐Ÿงฉ3 EASY PIECES TO RETRIEVAL Ask a question 1. Search database for stuff similar to question 2. Return the stuff 3.
  • 45. ๐Ÿ“‡INDEX (THE DATABASE) Split docs into chunks 1. Create embeddings for each chunk 2. Store embeddings in vector store index 3. Embeddings Vector Store Index Documents Raw Source Documents Chunked Documents [0.1,0.4,-0.6,...] [0.2,0.3,-0.4,...] [0.8,0.3,-0.1,...]
  • 46. ๐Ÿ•RETRIEVERS Embeddings Vector Store Index Documents Raw Source Documents Chunked Documents [0.1,0.4,-0.6,...] Query INPUT [0.1,0.4,-0.6,...] Find Nearest Neighbors Context: From source 1 Context: From source 2 Context: From source ๐Ÿ• [0.2,0.3,-0.4,...] [0.8,0.3,-0.1,...]
  • 47. [0.1, 0.4, -0.6, ...] Ryan was ... Query Find Nearest Neighbours (cosine similarity) Vector Database App Logic INPUT โ€œQuery...โ€ Embedding Model
  • 48. [0.1, 0.4, -0.6, ...] Use the provided context to answer the user's query. You may not answer the user's query unless there is specific context in the following text. If you do not know the answer, or cannot answer, please respond with "I don't know". Context: {context} User Query: {user_query} Query Embedding Model Chat Model Prompt Templates INPUT โ€œQuery...โ€ Find Nearest Neighbours (cosine similarity) Vector Database App Logic
  • 49. Embedding Model Chat Model Vector Store Find Nearest Neighbours (cosine similarity) Return document(s) from Nearest Neighbours [0.1, 0.4, -0.6, ...] Prompt Templates Vector Database App Logic App Logic Use the provided context to answer the user's query. You may not answer the user's query unless there is specific context in the following text. If you do not know the answer, or cannot answer, please respond with "I don't know". Context: {context} User Query: {user_query} Context: ref 1 Context: ref 2 Context: ref 3 Context: ref 4 Ryan was ... Query INPUT โ€œQueryโ€
  • 50. Embedding Model Chat Model Vector Store Find Nearest Neighbours (cosine similarity) Return document(s) from Nearest Neighbours [0.1, 0.4, -0.6, ...] Prompt Templates Vector Database App Logic App Logic Use the provided context to answer the user's query. You may not answer the user's query unless there is specific context in the following text. If you do not know the answer, or cannot answer, please respond with "I don't know". Context: {context} User Query: {user_query} Context: ref 1 Context: ref 2 Context: ref 3 Context: ref 4 Answer Query INPUT OUTPUT โ€œQueryโ€
  • 51. Embedding Model Chat Model Vector Store Find Nearest Neighbours (cosine similarity) Return document(s) from Nearest Neighbours [0.1, 0.4, -0.6, ...] Prompt Templates Vector Database App Logic App Logic Use the provided context to answer the user's query. You may not answer the user's query unless there is specific context in the following text. If you do not know the answer, or cannot answer, please respond with "I don't know". Context: {context} User Query: {user_query} Context: ref 1 Context: ref 2 Context: ref 3 Context: ref 4 Answer Query INPUT OUTPUT โ€œQueryโ€ Dense Vector Retrieval In-Context Learning
  • 52. Embedding Model Chat Model Vector Store Find Nearest Neighbours (cosine similarity) Return document(s) from Nearest Neighbours [0.1, 0.4, -0.6, ...] Prompt Templates Vector Database App Logic App Logic Use the provided context to answer the user's query. You may not answer the user's query unless there is specific context in the following text. If you do not know the answer, or cannot answer, please respond with "I don't know". Context: {context} User Query: {user_query} Context: ref 1 Context: ref 2 Context: ref 3 Context: ref 4 Answer Query INPUT OUTPUT โ€œQueryโ€ Dense Vector Retrieval In-Context Learning
  • 56. EMBEDDING & CHAT (LLM) MODEL Chat Model (e.g.๏ปฟ , LLM) e.g,. OpenAI GPT-4 Embeddi๏ปฟ ng Model e.g., Cohere V3 Embedding Model Chat Model
  • 58. Vector Database Qdrant Infrasturucture & Evaluation LangChain LangSmith DATABASE AND INFRASTRUCTURE Vector Database
  • 60. โ€œThe real power comes when you can combine them with other sources of computation or knowledge.โ€ ~ Harrison Chase Creator of LangChain
  • 65. Search OpenAI blog for top k resources, rerank 1. Ask specific questions related to content 2. Return answers to questions with sources 3. OpenAI RAG Flow
  • 66. OPENAI RAG ๐Ÿ‘จโ€๐Ÿ’ป Presented By Chris Alexiuk, LLM Wizard ๐Ÿช„
  • 67. HOW DO I IMPROVE?
  • 69. PROMPT ENGINEERING Check system-level prompting and one-shot/few-shot examples for alignment with your task e.g., varies
  • 70. RAG โ€œAre you pulling the right references? e.g., context recall
  • 71. FINE-TUNING EMBEDDINGS Is your model understanding domain-specific language? e.g., hit rate
  • 72. AGENTS Is your model reasoning the way a human would? e.g., ???
  • 73. THE RISE OF THE AI ENGINEER
  • 74. THE AGE OF THE AI ENGINEER โ€œA wide range of AI tasks that used to take 5 years and a research team to accomplish in 2013, now just require API docs and a spare afternoon in 2023.โ€ It is now possible to build what used to take months in a single day!
  • 75. DATA SCIENTISTS! Enhance Retrieval (and thus Generation!) Fine-Tuning Embeddings Chat Models Evaluation
  • 76. CONCLUSIONS Best-practice tools are out there! LangSmith-like tooling is the most comprehensive Building Prompt Engineering, RAG, Fine-Tuning Improvement Depends on Building! Eval varies Lots of work for data scientist and AI Engineers!
  • 77. Q&A Tony Karrer Founder & CEO TechEmpower, Founder & CTO Aggregage Dr. Greg Loughnane Founder & CEO of AI Makerspace Chris Alexiuk Co-Founder & CTO at AI Makerspace Tara Dwyer Webinar Manager /in/tonykarrer/ aggregage.com /in/gregloughnane/ aimakerspace.io /in/csalexiuk/ aimakerspace.io /in/taradwyer/ artificialintelligencezone.com JOIN THE GENERATIVE AI FOR TECHNOLOGY LEADERS LINKEDIN GROUP FOR THOUGHTFUL DISCUSSION AND Q&A! VISIT THE LINK OR SCAN THE QR CODE! bit.ly/genaitechleaders