Sandeep Singh, Head of Applied AI Computer Vision, Beans.ai
H2O Open Source GenAI World SF 2023
In the modern era of machine learning, leveraging both open-source and closed-source solutions has become paramount for achieving cutting-edge results. This talk delves into the intricacies of seamlessly integrating open-source Large Language Model (LLM) solutions like Vicuna, Falcon, and Llama with industry giants such as ChatGPT and Google's Palm. As the demand for fine-tuned and specialized datasets grows, it is imperative to understand the synergy between these tools. Attendees will gain insights into best practices for building and enriching datasets tailored for fine-tuning tasks, ensuring that their LLM projects are both robust and efficient. Through real-world examples and hands-on demonstrations, this talk will equip attendees with the knowledge to harness the power of both open and closed-source tools in a coherent and effective manner.
3. H2O.ai Confidential
Agenda
Introduction
Ecosystem of Open Source LLM Tools
The Significance of blending Open and Closed
Source LLMs
Open Source LLMs: Features and Benefits
Closed Source LLMs: Advantages and Use-Cases
Integration Strategies: Methodologies
Case Study Highlights and Learning from the
Industry Dataset Enrichment.
Low to No Code Fine-Tuning Techniques
Conclusion
4. v
H2O.ai Confidential
About Me
• Head of Applied AI/Computer Vision, Beans AI
• Beans AI based out of Palo Alto, CA
• We are Location Intelligence Platform.
• Hyper-Accurate Maps, Much more accurate than
Google, Apple and Bing for apartments.
• Computer Vision and Image based Synthesis is
inherent part of Innovation at Beans Maps
• I deal with Satellite Imagery, Location Data,
Convexity Optimization domains at my day to day
job.
• Holds Masters from Georgia Tech.
5. v
H2O.ai Confidential
Ecosystem of LLMs
GPT-4
PaLM
SageMaker
Neo
IBM
Watson
Salesforce
Einstein
Wu Dao 2.0
Clarifai
Cohere
Anthropic
Claude
MT-NLG
LLaMA 2
Falcon-
40B/180B
Vicuna 33B
MPT-30B
GPT-NeoX-
20B
CodeGen
GPT-J
OPT-175B
BLOOM
Baichuan-
13B
Proprietary/Closed Source Open Source
6. v
H2O.ai Confidential
Open Source LLMs: Benefits
● Enhanced data security and privacy: Self-hosted deployment
● Cost savings: No licensing/subscription fees and no API calls expenditure.
● No External Dependency: No reliance on select few vendor avoiding lock-in.
● Code transparency and Constructive Collaboration/Validation: Underlying code and methodologies are
vetted for functionality by community.
● Language Model Customization: Domain Adaptation is more manageable with open-source LLMs by Fine-
tuning.
● Active Community Support: Often thriving communities , Quicker issue resolution, access to resources and
collaborative.
● Fosters innovation: Open-source LLMs encourage innovation by enabling organizations to experiment and
build upon existing models.
● Boon for Startups: Leverage models as a foundation for creative and unique applications.
7. v
H2O.ai Confidential
Closed Source LLMs: Advantages
● Support and Reliability: Vendor Support, Professional assistance, Maintenance, Troubleshooting, SLA Requirements
● Customization for Specific Business Needs: Accommodate Unique requirements of a business
● Security and Data Privacy: May offer more robust security features and privacy assurances critical for industries with
sensitive data.
● Performance: Regular optimizations and enhancements for better performance for specific tasks or industry
● Integration with Proprietary Systems: Dedicated tooling support to use existing proprietary software stacks within an
organization to avoid extensive re-engineering.
● Compliance and Liability: For regulated industries greater assurance of compliance.
● Guardrails Ownership: Responsibility for compliance often falls on the vendor, reducing the legal and financial risks for
the user.
● Continuous Development and Updates: Dedicated teams to keep up-to-date cutting edge
● Commercial Viability: Better viability for businesses with limited resource and investments, Enable quicker feature
developments.
8. v
H2O.ai Confidential
Best of Both Worlds
● Number of options available.
● Possibility of exploiting more than one solution.
● In-house Performance comparison for “your” task, Not just a benchmark.
● Different baselines for particular domain adaptation.
● Amount of fine tuning needs are not same for similar tasks.
● Possibility of using specific LLM solution for specific task in pipeline.
● Combinations available to hyper ensemble these LLM solutions.
● Ability to pick and choose LLMs without affecting other LLMs in play.
9. v
H2O.ai Confidential
Integration Strategies
● Grunualize the task at hand:
Break LLM “initiative” into LLM “tasks”
● Categorize the tasks by Stochasticity Tolerance and Criticality:
Different LLM solutions pose varying degree of temperature sensitivity.
● Less tolerant tasks are candidate for Proprietary Off-the-shelf solutions.
● More tolerant tasks are candidate for Open Source with no or less fine tuning needs.
10. v
H2O.ai Confidential
Case Study Highlights
● At Beans.AI, we use combination of approaches like:
Few tasks are achieved using Prompt Engineering/RAG based approach.
Few tasks are achieved using Limited to moderate Fine Tuning.
● Both Closed source and Open Source LLMs are used.
● Responses from Closed Source LLMs are used by Open Source LLMs and vice versa in pipeline.
● Used for automated support, Insights from dashboard, automated email order etc.
11. v
H2O.ai Confidential
Dataset Enrichment
● No, NOT THAT data enrichment!
● Most of the time:
For “your” purpose, you need “your” data.
● “Your” data is limited by:
Quantity, Quality and Variety
● LLMs are used to overcome:
Quantity: By creating more samples of data
Quality: By working with humans in the loop type setup
Variety: By revising and rewriting intents in many different possible ways.
12. v
H2O.ai Confidential
Dataset Enrichment(cond.)
Example:
● Task: Question Answering Bot for your particular app. Say: Delivery Support App.
● Interaction: Delivery Driver asks a question in the app and expects “how-to” type response.
Question: How do I mark an address not deliverable in the app?
Candidate Answer: Explains the steps to do the same.
● Current Training Data: Set of Questions and Answers in knowledge article.
Enrichment Step:
Prompt engineered app to create variations of your domain specific questions as:
“Ask the above question in 20 different ways”
All these new 20 ways of asking the “same” question, create new training examples for you.
13. v
H2O.ai Confidential
Dataset Enrichment(cond.)
All of these questions below ask the EXACT same thing!
● What's the process for labeling an address as undeliverable within the application?
● Can you guide me through the steps to indicate that an address is non-deliverable in the app?
● How can I flag an address as undeliverable when using the app?
● What is the method for setting an address to 'not deliverable' status in the application?
● Is there a way to mark an address as 'cannot be delivered to' in the app interface?
● Could you explain how to designate an address as not deliverable on the app?
● I'm looking to mark an address as non-deliverable in the app; how do I do that?
● How does one go about indicating that an address is not serviceable in the app?
● In the app, what are the steps to mark an address as one that can't be delivered to?
● What’s the procedure to flag an address as 'not deliverable' in the app's system?
14. v
H2O.ai Confidential
Case Optimization
● We at Beans.AI use LLMs o analyze the pipeline to be used on the fly.
● E.g.
Super Set of Tasks For the Jobs:
Task 1, Task 2, Task 3, Task 4, Task 5, Task 6
Set of Tasks actually need for “job” instance:
Task 2, Task 4 and Task 6 Only.
● Proprietary LLMs with higher reasoning and guardrails is used to find the Tasks needed to be run.
● Short-listed Tasks can be run with actual sensitive data uses locally deployed Open Source LLM.
15. v
H2O.ai Confidential
Low to No Code Fine-Tuning Techniques
● H2O LLM Studio: Equivalent to Stable Diffusion’s Automatic1111 or ComfyUI.
● Fine-tune Open Source LLMs without any coding, While can extensible with code.
● GUI specially for LLMs.
● Support for hyperparameters specific to finetune of LLMs.
● Support Low-Rank Adaptation (LoRA) and lower quantization to achieve lean memory footprint.
● Model Performance Tracking in UI.
● Test the fine-tuned model by testing it to get instant feedback.
● Most Important Enabler: Almost touch-less export to Hugging Face Hub.
16. v
H2O.ai Confidential
Low to No Code Fine-Tuning Techniques(cond.)
My first fine-tuning using LLM Studio took almost same time as this presentation!