[DSC Europe 23] Ivan Petrovic - Approach to Architecting Generative AI Solutions
•Download as PPTX, PDF•
0 likes•46 views
This document discusses HTEC's approach to architecting generative AI solutions. It covers topics such as large language models (LLMs), common use cases for AI like classification and generation, and HTEC's framework which includes cognitive design, responsible AI practices, and a six step process from problem assessment to deployment. It also discusses prompts and prompt engineering to influence LLM output, and shows an architecture for a dynamic agent that can authenticate, upload data, perform inference, and notify users.
Recommended
Recommended
More Related Content
Similar to [DSC Europe 23] Ivan Petrovic - Approach to Architecting Generative AI Solutions
Similar to [DSC Europe 23] Ivan Petrovic - Approach to Architecting Generative AI Solutions (20)
More from DataScienceConferenc1
More from DataScienceConferenc1 (20)
Recently uploaded
Recently uploaded (20)
[DSC Europe 23] Ivan Petrovic - Approach to Architecting Generative AI Solutions
- 1. © HTEC Group, 2023 Approach To Architecting GenerativeAI Solutions Ivan Petrović Senior Technology Lead @ HTEC Group Empowering your digital tomorrow
- 2. © HTEC Group, 2023 2 Topics Approach To Architecting Generative AI Solutions Architecture Dive – Dynamic Agent Intro To Non-AI Folks HTEC’s Approach 01 02 03
- 3. © HTEC Group, 2023 Approach To Architecting Generative AI Solutions Intro To Non-AI Folks
- 4. © HTEC Group, 2023 4 What Are LLMs? Approach To Architecting Generative AI Solutions Large Language Model Paris What is the capital of France? Output Input
- 5. © HTEC Group, 2023 5 Hosting LLMs Approach To Architecting Generative AI Solutions Large Language Model + +
- 6. © HTEC Group, 2023 6 Multimodal Models Approach To Architecting Generative AI Solutions Multimodal Model IMU
- 7. © HTEC Group, 2023 7 Most Common Use-Cases Approach To Architecting Generative AI Solutions Classify Generate Rewrite Cluster Extract Search Summarize
- 8. © HTEC Group, 2023 HTEC’s Approach Approach To Architecting Generative AI Solutions
- 9. © HTEC Group, 2023 9 Throughout our consultative process we apply Cognitive Design principals and Responsible AI practices to ensure human integrity and ethical guidance Framework COGNITIVE DESIGN & RESPONSIBLE AI ACCELERATORS Analytics Intelligent assist 2 1 3 Data AI/ML ENGINEERING DATA ENGINEERING DATA SCIENCE PRODUCT DESIGN Cognitive Design CX SERVICE DESIGN BUSINESS DESIGN Generative AI Automation & productivity Autonomous systems Decision intelligence Data-driven innovation Computer vision Knowledge management Predictive analytics Intelligent process automation Anomaly detection AI-powered security & compliance Predictive maintenance STEP 1 Problem assessment STEP 2 Cognitive design STEP 3 Secure data onboarding & security STEP 4 AI/ML model development STEP 5 Continuous solution optimization STEP 6 Deployment & scaling Approach To Architecting Generative AI Solutions
- 10. © HTEC Group, 2023 10 Prompts and Prompt Engineering • Prompts are inputs sent to LLM to elicit a certain response • Natural language instructions • Prompt Enginering • Influence the creativity of LLM • Halucinations • Training vs Prompting Approach To Architecting Generative AI Solutions Prompt content Instruction Context Output Indicator Input Data
- 11. © HTEC Group, 2023 11 Points of Interest • Types of learning (LLM) • Zero shot • One shot • Few shot • Fine tunning • Vector Databases • Considerations • Ethics • Licencing • Knowledge cutoff • Token limitations • Token modalities Approach To Architecting Generative AI Solutions Input Zero Shot Example 1 One Shot Input Example 1 Few Shot … Example N Input
- 12. © HTEC Group, 2023 12 Complexity Hierarchy Approach To Architecting Generative AI Solutions Complexity level No Context Simple Context Tool Use Multi-Agent Important points No persistent memory, can be used hirerachaly Retaining conv. history, conv. buffers, summary, window Persisten memory with vector storage Multiple agents with different prompts, spawning agents trough interanl main agent API Use-case example Document summarisation Chatbot conversation memory ChatGPT with plugins AutoGPT, BabyAGI
- 13. © HTEC Group, 2023 Architecture Dive - Dynamic Agent Approach To Architecting Generative AI Solutions
- 14. © HTEC Group, 2023 14 Agent Reusability Configuration Agent type (container) Agent instance Approach To Architecting Generative AI Solutions
- 15. © HTEC Group, 2023 15 Auth Token Stream data Make API calls Send requests in natural language Architecture Approach To Architecting Generative AI Solutions Validate Auth Token Send message Consume message Notify on start Stream response Upload data with token Send prompt Send prompt to LLM Send progress Consume progress status Read/write chat history
- 16. © HTEC Group, 2023 16 Send prompt Send progress Auth Token Stream data Make API calls Send requests in natural language Authentication Approach To Architecting Generative AI Solutions Validate Auth Token Send message Consume message Notify on start Stream response Upload data with token Send prompt to LLM Consume progress status Read/write chat history
- 17. © HTEC Group, 2023 17 Auth Token Stream data Make API calls Send requests in natural language Data Upload Approach To Architecting Generative AI Solutions Validate Auth Token Send message Consume message Notify on start Stream response Upload data with token Send prompt to LLM Send progress Consume progress status Read/write chat history Send prompt
- 18. © HTEC Group, 2023 18 Send progress Auth Token Stream data Make API calls Send requests in natural language Inference Flow Approach To Architecting Generative AI Solutions Validate Auth Token Send message Consume message Notify on start Stream response Upload data with token Send prompt Send prompt to LLM Consume progress status Read/write chat history
- 19. © HTEC Group, 2023 19 Auth Token Stream data Make API calls Send requests in natural language Notification Flow Approach To Architecting Generative AI Solutions Validate Auth Token Send message Consume message Notify on start Stream response Upload data with token Send prompt to LLM Send progress Consume progress status Read/write chat history Send prompt
- 20. © HTEC Group, 2023 20 Chatbot components Approach To Architecting Generative AI Solutions Make API calls Make API calls LLM request Prompt rendering Multiple techniques Conversation history Memory configuration Input data Strategy config Prompt templates Consume message - metadata - agent configuration - input data Progress message Stream response data Send prompt to LLM
- 21. © HTEC Group, 2023 21 Configuration Approach To Architecting Generative AI Solutions Make API calls Make API calls LLM request Prompt rendering Multiple techniques Conversation history Memory configuration Input data Strategy config Prompt templates Consume message - metadata - agent configuration - input data Progress message Stream response data Send prompt to LLM Notification
- 22. © HTEC Group, 2023 22 Agent Type Approach To Architecting Generative AI Solutions Make API calls Make API calls LLM request Prompt rendering Multiple techniques Conversation history Memory configuration Strategy config Prompt templates Consume message - metadata - agent configuration - input data Progress message Stream response data Send prompt to LLM Notification
- 23. © HTEC Group, 2023 23 Agent Data Flow Approach To Architecting Generative AI Solutions Make API calls Make API calls LLM request Prompt rendering Multiple techniques Conversation history Memory configuration Input data Consume message - input data Progress message Stream response data Send prompt to LLM Notification
- 24. © HTEC Group, 2023 24 Transformation Data Sources Business Logic End User Ingestion Analytics Service Question Processing Context Building Query Generation Fund Monitoring Platform Azure Data Lake (Raw Storage) Azure Data Lake (Analytics Storage) Data Processing (Spark Pool) Ingestion Pipeline UI Argo CD Azure DevOps Azure Container Registry Azure Key Vault Azure OpenAI Service Azure Synapse Analytics Real-world AI and Generative AI AI-Assisted Fund Performance Analytics Authentication & Authorisation (KeyCloak)
- 25. © HTEC Group, 2023 Architecting Generative AI Solutions QnA Ivan Petrović Senior Technology Lead @ HTEC Group
Editor's Notes
- Greet the audience, greet the oportunity to speak and share on DSC Provide an short intro if not done before the presentation
- Glance over topics that we will cover - First one is something that will help Non AI folks to get an understanding what the heck is this genAI everyone talks about (if there are any) - Overview of how we @ HTEC approaches about thinking and solving these problems and what to consider when building - Go into one of the accelerators we have and an example of the system we have built
- This can be an overview for someone who is non AI engineer @ HTEC we have identified that there is a path engineer can take to get into the field of Generative AI and we will touch on that - To enable them to catch up - Learning Plan for engineers Why is it important to emphasise this path to learning and intro to Generative AI
- What are LLMs? Number of parameters, RLHF
- How LLMs can be used/deployed - on prem, deploying custom LLM on cloud enviorment - Managed by your company - ON EDGE - using 3rd party LLM, e.g. OpenAI via OpenAI API - Hybrid; combination of first two
- What modalities are there that LLMs are going if not to be using? - Text - Image - Audio - Sensor data (Meta) Etc. Naglasiti da postoje posebne implementacije koje to omogućavaju - možda reći malo o tome šta je tokenizacija -
- What are some of the use cases when LLMs can be used Focus is only on text right now, but we are seeing that image is being used also but not that often.
- Before diving into the concreete examples - describe to audience how we at HTEC are approaching the problem of identifying HTEC's
- Go trough the steps but from the engineering and problem perspective Emphasis PREDICTIVE WALK BETWEEN RND AND Productization This is a fork of existing workflow and augmented by the needs we have seen to emerge in GenAI Whitepaper was released by our colleagues that targets most of the first 2 steps. We will focus mostly on the middle part right now because it is important to see how much it differs from the traditional AI workflows - What we are observing as new steps emerging, or being more emphasized - Cognitive design - More emphasis on security and monitoring for potential rogue behavior (Agents can require access sensitive data, like query databases, generate queries, etc.) - Accelerators are something you build to help you jump-start the implementation
- - What are Prompts? - What is their modality? Considering only text now - What is prompt engineering? - How to handle creativity of the LLMs? - From which parts it consists of? What are tokens?
- Postaviti se iz pozicije da smo mi ovo pomagali našim arhitektama... - treniram trenere - šta smo mi pomogli i kako našim kolegama - What and how models can learn and be trained? - Model training – adapting the weights to accommodate new data - Model prompting – selecting model input to achieve desired output - Types of learning - slightly different than one in DL - Ethical consideration - Licensing and knowledge cutoff - Vector Databasees - How the semantic search is done - Token Limitation - Prompt length is limited - prompt contains task specification, additional information, and context information - Token Modalities - Text modality, a token is approx 3 characteds (3 tokens are used per word, on average) - Other modalities use different tokenization schemes - Tendencija da 'e modeli da budu skoro near real time... - Promena u poslednjih mesec dva koliko je - GROK je near real time - Recent update to openAI
- Talk about the complexity - Increasing complexity provides more capabitilities to be achieved - Stateless API – Two approaches - chat completition - text completition - Hint kako utiče kontekst i da li veličlina konteksta je bitna - AutoGPT malo bolje objasniti - šta su obećali a šta je moguće
- Dive into an example how an Dynamic Agent would be implemented Implementation depends on the use-case Focus is on reusability and modularity of the system that is being implemented
- Building the solution, modularity and configurability is the key. This is one of the ways how to create an accelearator
- Sinhroni i asinhrona komunikacija Ideja za regione, multiregion za lokalne limite Crtica ka spolja C4 L2 nivo dijagrama - Da postoje kontejneri Naglasiti kao simplified Vehicle ka arhitektama da oni budu svesni šta sve postoji ovde o čemu moraju voditi rešenja
- Sinhroni i asinhrona komunikacija Ideja za regione, multiregion za lokalne limite Crtica ka spolja C4 L2 nivo dijagrama - Da postoje kontejneri Naglasiti kao simplified Vehicle ka arhitektama da oni budu svesni šta sve postoji ovde o čemu moraju voditi rešenja
- Sinhroni i asinhrona komunikacija Ideja za regione, multiregion za lokalne limite Crtica ka spolja C4 L2 nivo dijagrama - Da postoje kontejneri Naglasiti kao simplified Vehicle ka arhitektama da oni budu svesni šta sve postoji ovde o čemu moraju voditi rešenja Get SAS Token
- Sinhroni i asinhrona komunikacija Ideja za regione, multiregion za lokalne limite Crtica ka spolja C4 L2 nivo dijagrama - Da postoje kontejneri Naglasiti kao simplified Vehicle ka arhitektama da oni budu svesni šta sve postoji ovde o čemu moraju voditi rešenja RPC - job based sistem za obradu nekih duga;kih zadataka - sistem koji ne zavisi od request response timeouta Progress queue Da bi korisnik znao gde se trenutno nalazi sistem 100 documents to process
- Sinhroni i asinhrona komunikacija Ideja za regione, multiregion za lokalne limite Crtica ka spolja C4 L2 nivo dijagrama - Da postoje kontejneri Naglasiti kao simplified Vehicle ka arhitektama da oni budu svesni šta sve postoji ovde o čemu moraju voditi rešenja
- Not constrained to function calling for certain tasks This enables the system to be interchangable