The document discusses the integration of open source and closed source large language models (LLMs) in building AI solutions, highlighting their features, benefits, and unique use cases. It emphasizes the importance of dataset enrichment and fine-tuning techniques while showcasing case study examples from Beans.ai that leverage both types of LLMs for various applications like customer support. Strategies for effective integration are outlined, promoting a balanced approach to utilizing LLMs to achieve optimal task performance.

1. H2O.ai Confidential
SANDEEP SINGH
Head of Applied AI, Beans.AI

2. H2O.ai Confidential
Building LLM Solutions using Open Source and Closed Source Solutions
in Coherent Manner

3. H2O.ai Confidential
Agenda
Introduction
Ecosystem of Open Source LLM Tools
The Significance of blending Open and Closed
Source LLMs
Open Source LLMs: Features and Benefits
Closed Source LLMs: Advantages and Use-Cases
Integration Strategies: Methodologies
Case Study Highlights and Learning from the
Industry Dataset Enrichment.
Low to No Code Fine-Tuning Techniques
Conclusion

4. v
H2O.ai Confidential
About Me
• Head of Applied AI/Computer Vision, Beans AI
• Beans AI based out of Palo Alto, CA
• We are Location Intelligence Platform.
• Hyper-Accurate Maps, Much more accurate than
Google, Apple and Bing for apartments.
• Computer Vision and Image based Synthesis is
inherent part of Innovation at Beans Maps
• I deal with Satellite Imagery, Location Data,
Convexity Optimization domains at my day to day
job.
• Holds Masters from Georgia Tech.

5. v
H2O.ai Confidential
Ecosystem of LLMs
GPT-4
PaLM
SageMaker
Neo
IBM
Watson
Salesforce
Einstein
Wu Dao 2.0
Clarifai
Cohere
Anthropic
Claude
MT-NLG
LLaMA 2
Falcon-
40B/180B
Vicuna 33B
MPT-30B
GPT-NeoX-
20B
CodeGen
GPT-J
OPT-175B
BLOOM
Baichuan-
13B
Proprietary/Closed Source Open Source

6. v
H2O.ai Confidential
Open Source LLMs: Benefits
● Enhanced data security and privacy: Self-hosted deployment
● Cost savings: No licensing/subscription fees and no API calls expenditure.
● No External Dependency: No reliance on select few vendor avoiding lock-in.
● Code transparency and Constructive Collaboration/Validation: Underlying code and methodologies are
vetted for functionality by community.
● Language Model Customization: Domain Adaptation is more manageable with open-source LLMs by Fine-
tuning.
● Active Community Support: Often thriving communities , Quicker issue resolution, access to resources and
collaborative.
● Fosters innovation: Open-source LLMs encourage innovation by enabling organizations to experiment and
build upon existing models.
● Boon for Startups: Leverage models as a foundation for creative and unique applications.

7. v
H2O.ai Confidential
Closed Source LLMs: Advantages
● Support and Reliability: Vendor Support, Professional assistance, Maintenance, Troubleshooting, SLA Requirements
● Customization for Specific Business Needs: Accommodate Unique requirements of a business
● Security and Data Privacy: May offer more robust security features and privacy assurances critical for industries with
sensitive data.
● Performance: Regular optimizations and enhancements for better performance for specific tasks or industry
● Integration with Proprietary Systems: Dedicated tooling support to use existing proprietary software stacks within an
organization to avoid extensive re-engineering.
● Compliance and Liability: For regulated industries greater assurance of compliance.
● Guardrails Ownership: Responsibility for compliance often falls on the vendor, reducing the legal and financial risks for
the user.
● Continuous Development and Updates: Dedicated teams to keep up-to-date cutting edge
● Commercial Viability: Better viability for businesses with limited resource and investments, Enable quicker feature
developments.

8. v
H2O.ai Confidential
Best of Both Worlds
● Number of options available.
● Possibility of exploiting more than one solution.
● In-house Performance comparison for “your” task, Not just a benchmark.
● Different baselines for particular domain adaptation.
● Amount of fine tuning needs are not same for similar tasks.
● Possibility of using specific LLM solution for specific task in pipeline.
● Combinations available to hyper ensemble these LLM solutions.
● Ability to pick and choose LLMs without affecting other LLMs in play.

9. v
H2O.ai Confidential
Integration Strategies
● Grunualize the task at hand:
Break LLM “initiative” into LLM “tasks”
● Categorize the tasks by Stochasticity Tolerance and Criticality:
Different LLM solutions pose varying degree of temperature sensitivity.
● Less tolerant tasks are candidate for Proprietary Off-the-shelf solutions.
● More tolerant tasks are candidate for Open Source with no or less fine tuning needs.

10. v
H2O.ai Confidential
Case Study Highlights
● At Beans.AI, we use combination of approaches like:
Few tasks are achieved using Prompt Engineering/RAG based approach.
Few tasks are achieved using Limited to moderate Fine Tuning.
● Both Closed source and Open Source LLMs are used.
● Responses from Closed Source LLMs are used by Open Source LLMs and vice versa in pipeline.
● Used for automated support, Insights from dashboard, automated email order etc.

11. v
H2O.ai Confidential
Dataset Enrichment
● No, NOT THAT data enrichment!
● Most of the time:
For “your” purpose, you need “your” data.
● “Your” data is limited by:
Quantity, Quality and Variety
● LLMs are used to overcome:
Quantity: By creating more samples of data
Quality: By working with humans in the loop type setup
Variety: By revising and rewriting intents in many different possible ways.

12. v
H2O.ai Confidential
Dataset Enrichment(cond.)
Example:
● Task: Question Answering Bot for your particular app. Say: Delivery Support App.
● Interaction: Delivery Driver asks a question in the app and expects “how-to” type response.
Question: How do I mark an address not deliverable in the app?
Candidate Answer: Explains the steps to do the same.
● Current Training Data: Set of Questions and Answers in knowledge article.
Enrichment Step:
Prompt engineered app to create variations of your domain specific questions as:
“Ask the above question in 20 different ways”
All these new 20 ways of asking the “same” question, create new training examples for you.

13. v
H2O.ai Confidential
Dataset Enrichment(cond.)
All of these questions below ask the EXACT same thing!
● What's the process for labeling an address as undeliverable within the application?
● Can you guide me through the steps to indicate that an address is non-deliverable in the app?
● How can I flag an address as undeliverable when using the app?
● What is the method for setting an address to 'not deliverable' status in the application?
● Is there a way to mark an address as 'cannot be delivered to' in the app interface?
● Could you explain how to designate an address as not deliverable on the app?
● I'm looking to mark an address as non-deliverable in the app; how do I do that?
● How does one go about indicating that an address is not serviceable in the app?
● In the app, what are the steps to mark an address as one that can't be delivered to?
● What’s the procedure to flag an address as 'not deliverable' in the app's system?

14. v
H2O.ai Confidential
Case Optimization
● We at Beans.AI use LLMs o analyze the pipeline to be used on the fly.
● E.g.
Super Set of Tasks For the Jobs:
Task 1, Task 2, Task 3, Task 4, Task 5, Task 6
Set of Tasks actually need for “job” instance:
Task 2, Task 4 and Task 6 Only.
● Proprietary LLMs with higher reasoning and guardrails is used to find the Tasks needed to be run.
● Short-listed Tasks can be run with actual sensitive data uses locally deployed Open Source LLM.

15. v
H2O.ai Confidential
Low to No Code Fine-Tuning Techniques
● H2O LLM Studio: Equivalent to Stable Diffusion’s Automatic1111 or ComfyUI.
● Fine-tune Open Source LLMs without any coding, While can extensible with code.
● GUI specially for LLMs.
● Support for hyperparameters specific to finetune of LLMs.
● Support Low-Rank Adaptation (LoRA) and lower quantization to achieve lean memory footprint.
● Model Performance Tracking in UI.
● Test the fine-tuned model by testing it to get instant feedback.
● Most Important Enabler: Almost touch-less export to Hugging Face Hub.

16. v
H2O.ai Confidential
Low to No Code Fine-Tuning Techniques(cond.)
My first fine-tuning using LLM Studio took almost same time as this presentation!

17. H2O.ai Confidential