Modern NLP models can be daunting: No more bag-of-words but complex neural network architectures, with billions of parameters. Engineers, financial analysts, entrepreneurs, and mere tinkerers, fear not! You can get started with as little as 10 lines of code. Presentation prepared for the Abu Dhabi Machine Learning Meetup Season 3 Episode 3 hosted at ADGM in Abu Dhabi.

1. G
a
utier M
a
rti (ADML S3E3) - 21 M
a
rch 2023
UsingLargeLanguageModelsin
10LinesofCode

2. blog and colab links
https://marti.ai/quant/2023/03/11/set
fi
t-llm-10-lines-of-code.html

3. Transformers
Modern NLP models can be daunting:
No more bag-of-words but complex neural
network architectures (transformers), with
billions of parameters.
Engineers,
fi
nancial analysts,
entrepreneurs, and mere tinkerers, fear not!
You can get started with as little as 10 lines
of code.

4. TheHuggingFaceLibrary
2 Lines of Code for Sentiment An
a
lysis
https://huggingface.co/

5. Doesitworkforspecializedlanguage?
Validation sentences:
• Sentence 1: In a decently risk-on session (CDX IG -2.8 CDX HY -8.9 SPX @ 2,900), the CDS
of Anadarko Petroleum Corp. (APC) outperformed the broader market, tightening by
c65bp. (Expected sentiment: POSITIVE)
• Sentence 2: Bonds are also 75-100bp tighter. (Expected sentiment: POSITIVE)
• Sentence 3: That is because the oil giant Chevron Corp. (CVX) agreed to buy APC.
(Expected sentiment: NEUTRAL)
• Sentence 4: The equity is valued $33B, which will be paid in stocks and cash (75/25: 0.3869
CVX shares and $16.25 in cash per APC share). (Expected sentiment: NEUTRAL)
Use c
a
se: Fin
a
nci
a
l j
a
rgon speci
f
ic to credit m
a
rket comments

6. Doesitworkforspecializedlanguage?
No! But you c
a
n get g
a
rb
a
ge in 2 lines of code only.
The pretrained o
ff
-the-shelf model gets Sentences 1, 3, 4 wrong... Remember, we expect: POSITIVE, POSITIVE, NEUTRAL, NEUTRAL.

7. Potentialsolution:TheHuggingFaceHub
• In the Hub, there are models pretrained with
fi
nancial sentiment in mind:
• https://huggingface.co/ahmedrachid/FinancialBERT-Sentiment-Analysis
• https://huggingface.co/yiyanghkust/
fi
nbert-tone
The Hugging F
a
ce Hub is
a
pl
a
tform with over 120k models, 20k d
a
t
a
sets, ...

8. Thesemodelswere
f
ine-tuned...
• They were designed to analyze corporate announcements from the perspective of
an equity fundamental analyst.
• Both models yield: POSITIVE, NEUTRAL, NEUTRAL, NEUTRAL,
with Sentence 2 "Bonds are also 75-100bp tighter" being wrong.
... on the wrong "sentiment" t
a
sk.

9. Game Over?

10. HuggingFaceSetFit-Ef
f
icientFine-Tuning
• SetFit means Sentence Transformer Fine-tuning
• Based on a couple of examples {xi} for each class {c(xi)}, a sentence transformer is
fi
ne-tuned on a dataset of positive, and negative, triplets (xi, xj, I(c(xi) = c(xj)))
• Given a
fi
ne-tuned sentence transformer ST, one can
fi
t a logistic regression (the
classi
fi
cation head CH) on {(ST(xi), c(xi))}.
• At inference, apply CH(ST(xi))
P
a
per: Ef
f
icient Few-Shot Le
a
rning Without Prompts

11. SetFitinpractice
• Hand-labeled a dozen examples per class:
1: positive, 0: neutral, -1: negative
• Import a generic pre-trained sentence transformer
Two for one trainer:
1. Fine-tune ST, and
2.
fi
t CH.

12. SetFitResults
The model h
a
s successfully
a
d
a
pted to credit portfolio m
a
n
a
gers' j
a
rgon.

13. Questions?