Downloaded 10 times
Uploaded byFrancesco Corti
Supercharge Your AI Development with Local LLMs
In today's AI development landscape, developers face significant challenges when building applications that leverage powerful large language models (LLMs) through SaaS platforms like ChatGPT, Gemini, and others. While these services offer impressive capabilities, they come with substantial costs that can quickly escalate especially during the development lifecycle. Additionally, the inherent latency of web-based APIs creates frustrating bottlenecks during the critical testing and iteration phases of development, slowing down innovation and frustrating developers. This talk will introduce the transformative approach of integrating local LLMs directly into their development environments. By bringing these models closer to where the code lives, developers can dramatically accelerate development lifecycles while maintaining complete control over model selection and configuration. This methodology effectively reduces costs to zero by eliminating dependency on pay-per-use SaaS services, while opening new possibilities for comprehensive integration testing, rapid prototyping, and specialized use cases.
More Related Content
Similar to Supercharge Your AI Development with Local LLMs
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Supercharge Your AI Development with Local LLMs
- 1. Supercharge Your AIDevelopment with Local LLMs Speaker Francesco Corti Principal Product Manager, Docker
- 2. This slide deck is developed by DataCouch About Speaker Speaker Francesco Corti PrincipalProduct Manager, Docker Principal Product Manager @ Open Source enthusiast Developer at heart (ex-DevRel) Speaker & Author
- 3. This slide deck is developed by DataCouch What you shouldexpect The Current AI Development Landscape (Hands-On) The Core Challenges What Are Local LLMs? Development Guide For Local LLMs (Hands-On) Integration Patterns And Tips for Developers Questions & Answers Demo... Demo… Demo…
- 4.
- 5. This slide deck is developed by DataCouch Disclaimer I’m not hereto sell We are going to focus on GenAI (subset of AI) What you are going to see today is evolving fast* (*) Insanely fast Call To Action: Let me know your experience!
- 6. This slide deck is developed by DataCouch Developing GenAI Applicationsis not necessarily hard User Application Submits prompt LLM API SaaS Service Large Language Model API Send request to LLM OpenAI API
- 7. Demo Create a simpleweb application in python running at the port 8080. The web application develops a chat with the end user. The requirements are: Flask==2.3.3, requests==2.31.0 python-dotenv==1.0.0 Use the OpenAI client because I'm going to use it. The ChatGPT base url available at https://api.openai.com/v1/ to interact with the OpenAI API. The OpenAI API key is sk-proj-XXX-XXX. Then execute: python3 -m venv venv source venv/bin/activate pip3 install -r requirements.txt python3 app.py Asking Cursor to create a GenAI Application
- 9. This slide deck is developed by DataCouch Developing GenAI Applicationsis not necessarily hard easy Example: Modular Monolith LLM Approach With Composability https://medium.com/data-science/generative-ai-design-patterns-a-comprehensive-guide-41425a40d7d0 Real life scenario are never easy (as they should)
- 10. This slide deck is developed by DataCouch Developing GenAI ApplicationsUsing RAG User Application RAG (Retrieval Augmented Generation) Retrieval model Vector database Create vector (embedding) for prompt Find relevant information Submits prompt LLM API Send request to LLM SaaS Service Large Language Model API Ingestion is not trivial The Business Logic is more complex (nothing crazy)
- 11. This slide deck is developed by DataCouch Developing GenAI ApplicationsUsing Tools (MCP) User Application Submits prompt LLM API Send request to LLM SaaS Service Large Language Model API Tool/Function calling Tool (either bundled or remote) Invoke tools and get responses RAG (Retrieval Augmented Generation)
- 15. This slide deck is developed by DataCouch Developing GenAI ApplicationsUsing A Custom Model User Application Submits prompt LLM API Send request to LLM Inference Engine RAG (Retrieval Augmented Generation) Tools LLM LLM LLM Custom LLM It can be complex Cannot be SaaS
- 16. This slide deck is developed by DataCouch The Increasing Complexityof Developing GenAI Applications GenAI App Leveraging SaaS Services GenAI App Leveraging RAG and SaaS Services (Eventually Using Tools) GenAI App Using Custom Models Easy Complex … Major Concerns Security & Privacy Latency Cost At Scale Business Developer
- 17. This slide deck is developed by DataCouch Introducing Local InferenceEngines + LLMs A local Large Language Model (LLM) is an AI model deployed and executed entirely on your own hardware — such as a personal computer, workstation, or server — without relying on external cloud services User Application Submits prompt Large Language Model (LLM) Inference Engine Large Language Model API Send request to LLM OpenAI API Not as SaaS Large Language Model (LLM) Large Language Model (LLM)
- 18. This slide deck is developed by DataCouch Are Local InferenceEngines New? No, they exists since some time* (*) But in these days they are getting more attention
- 19. This slide deck is developed by DataCouch Are LLMs HardTo Find And Use? No, almost every product has its own registry… but not always the same format. GGUF GPTQ AWQ EXL2 HQQ ONNX Safetensors PyTorch (.pt/.bin) TensorFlow (.pb/.ckpt)
- 20. Demo ● Local LLMin Action ● Managing Local LLMs ● Same Power But Local
- 22. Demo ● Local LLMin Action ● Managing Local LLMs ● Same Power But Local
- 25. Demo ● Local LLMin Action ● Managing Local LLMs ● Same Power But Local
- 27. This slide deck is developed by DataCouch Strengths And Weaknesses + ○Data Are Managed Locally (Security & Privacy) ○ Zero Latency ○ Zero Cost (For Execution) - ○ Local Resources (Need of Bigger Machines) ○ Production-like Environments (Despite OpenAI API) ○ Security of Models (Emerging) and MCP Tools
- 28. This slide deck is developed by DataCouch Why Do TheyHave More Attention In These Days? Models Are Getting “Smaller” Emerging Use Cases Testing of GenAI Applications “Light” Chatbots Models Experimentation Models Comparison … Call To Action: Are you Aware About More?
- 29. This slide deck is developed by DataCouch The Current AIDevelopment Landscape (Hands-On) The Core Challenges What Are Local LLMs? Development Guide For Local LLMs (Hands-On) Integration Patterns And Tips for Developers Questions & Answers
- 30. This slide deck is developed by DataCouch AI is goingto be disruptive A lot of developers are using AI for development A lot of developers are playing with AI development Very few are developing AI applications (for production) Now it’s time to learn! Now it’s time to build the DevEx of the future! Francesco Corti Principal Product Manager at Docker Inc. https://www.linkedin.com/in/fcorti/