Philipp Singer, Senior Principal Data Scientist, H2O.ai H2O Open Source GenAI World SF 2023

1. v
H2O.ai Confidential
Senior Principal Data Scientist at H2O.ai
● Vienna / Austria
● Kaggle GM
● All things deep learning
● I love training models
● H2O Hydrogen Torch
● H2O LLM Studio
Philipp Singer

2. H2O.ai Confidential
Mastering Classification and Regression with LLMs:
Insights from Kaggle Competitions

3. v
H2O.ai Confidential
The typical LLM use case
Text generation Q&A / RAG Labeling Agents

4. v
H2O.ai Confidential
What about classification?
Common business use-case
Classify text into two or more categories
Sentiment classification
Document categorization
Spam detection
Language detection
Topic classification

5. v
H2O.ai Confidential
The common way
Supervised training
Train a model on labeled data, predict on unlabeled data
Bag of word approach
Vectorize into fixed vocabulary, train gradient boosting models
Transformer approach
Train transformer models like BERT, Roberta, Deberta

6. v
H2O.ai Confidential
The LLM way
Zero-shot classification
Ask a LLM model for the prediction without training
Zero-shot engineering
Can we improve zero-shot prediction quality?
Fine-tuning LLMs
Fine-tune a LLM model for task-specific classification

7. H2O.ai Confidential
Use case

8. v
H2O.ai Confidential
Financial sentiment data
huggingface.co/datasets/financial_phrasebank
● Data from FiQA and Financial PhraseBank
● Subset with >=75% annotator agreement
● Data statistics
○ 3,453 rows
○ Train: 2,589 rows
○ Val: 864 rows
● Labels
○ ~62% neutral
○ ~26% positive
○ ~12% negative
● Simplest majority baseline
○ 0.622 Accuracy
Cramo and Peab
have signed
exclusive five-year
rental agreements
in Finland and have
extended their
existing rental
agreements in the
Swedish market for
another five years.
Construction work
on the Helsinki
Music Centre is to
start this autumn,
with the total cost of
the project
estimated at 140
million euros.
The company's
profit totaled
578,100 in H1 2007,
down 30.9% year-
on-year.

9. v
H2O.ai Confidential
Let’s ask h2oGPT

10. v
H2O.ai Confidential
Zero-shot streamlining is difficult
No training needed
No labels needed
Easy to get started
Prompt engineering tricky
Difficult to automate into business processes
Runtime expensive
Often interpretable results Evaluation still required → labels

11. v
H2O.ai Confidential
Zero-shot logits approach
GPT
Your task is to analyze the message below and predict whether it has negative, neutral or positive
sentiment.
Return on investment was 16.6% compared to 15.8% in 2004.
The sentiment is

12. v
H2O.ai Confidential
Zero-shot logits approach

13. v
H2O.ai Confidential
Zero-shot logits approach

14. v
H2O.ai Confidential
Classification fine-tuning
GPT Negative
Neutral
Positive
Classification
Head
Finetune

15. v
H2O.ai Confidential
H2O LLM Studio
github.com/h2oai/h2o-llmstudio
● Easily and effectively fine-tune LLMs
● CLI & No-Code GUI
● SOTA techniques
○ LoRA
○ Quantization
○ Hyperparameter tuning
○ Experiment tracking and advanced evaluation
○ SFT & RLHF
● Newest Problem Type: Causal Classification
● Fully open-sourced

16. v
H2O.ai Confidential
Fine-tuning approach

17. v
H2O.ai Confidential
Fine-tuning approach

18. H2O.ai Confidential
Kaggle success stories

19. v
H2O.ai Confidential
H2O.ai Predict the LLM

20. v
H2O.ai Confidential
CommonLit - Evaluate Student Summaries Competition

21. v
H2O.ai Confidential
LLM Science Exam Competition

22. H2O.ai Confidential
philipp.singer@h2o.ai
@ph_singer
Contact