Dictionary-assisted supervised contrastive learning (DASCL) is a method that leverages specialized dictionaries when fine-tuning pretrained language models. It combines cross-entropy loss with a supervised contrastive learning objective to improve classification performance, particularly in few-shot learning settings. Evaluations on tasks like sentiment analysis and abuse detection found that DASCL outperforms cross-entropy alone or supervised contrastive learning without dictionaries. Interpretability techniques like contrastive explanations can provide insights into why models make predictions by comparing predictions to alternative options.

1. 2023 EMNLP Day
San Kim
2023.01.10

2. Dictionary-Assisted Supervised Contrastive Learning
Dictionary-assisted supervised contrastive learning(DASCL)
• A way to leverage specialized dictionaries when finetuning pretrained language models
Dictionary Examples
• Opinion Lexicon
• Lexicoder Sentiment Dictionary
Findings
• Finetuning using the DASCL objective combined with cross-entropy improves classification
performance metrics across various applications
• Comparisons to cross-entropy alone and Supervised contrastive learning(SCL) – Gunel et al.
2021
• Find greatest improvements in few-shot learning settings
• Also find improvements when using the entire training set

3. Dictionary-Assisted Supervised Contrastive Learning
Keyword Simplification

4. Dictionary-Assisted Supervised Contrastive Learning
Keyword Simplification

5. Dictionary-Assisted Supervised Contrastive Learning
Contrastive Learning
• The two RoBERTa networks share the same weights.
• The dimension of the projection layer is arbitrary.

6. Dictionary-Assisted Supervised Contrastive Learning

7. Dictionary-Assisted Supervised Contrastive Learning
AbuseEval, using OLID(Offensive Language Identification Dataset)-OffensEval(SemEval 2019 shared
task)
• Explicit (abuse)
• #ThursdayThoughts- FUCK liberals. Forever.
• @USER @USER SHE IS A FUCKING MESS!! I HATE HER SO MUCH
• @USER Gotohell ! This is NOT Queen for a Day. I believe you less and less with every bit of
bullsh*t you pull. You’re nothing but a lying Demonrat! #MAGA #Trump2020
• Implicit (abuse)
• 4 out of 10 British people are basically full-on racists. 4 out of 10 voters vote for the
Conservatives. Coincidence!???!???
• @USER@USER Oh you are in England? Your views on gun control stopped mattering in 1776.
• @USER@USER Wonder how many children he molested
• @USER Isn’t the coalition for gun control headed up by the lady who was turned down for a
job because she was a bully?
• Not (abusive)
• @USER I miss you bitch!!
• @USER Nigga we’re going next week

8. Dictionary-Assisted Supervised Contrastive Learning

9. Interpreting Language Models with Contrastive Explanations (BP session)
Common Approch
• Why did the LM predict [something]?
• Why did the LM predict “barking”?
• Input: Can you stop the dog from
• Output: barking
Contrastive explanations are more intuitive
• Why did the LM predict [target] instead of [foil]?
• Why did the LM predict “barking” instead of “crying”?
• Input: Can you stop the dog from
• Output: barking

10. Interpreting Language Models with Contrastive Explanations (BP session)
Contrastive explanations for language models

11. Interpreting Language Models with Contrastive Explanations (BP session)
• Gradient Norm
• Contrastive Gradient Norm
• Gradient X Input
• Contrastive Gradient X Input
• Input Erasure
• Contrastive Input Erasure

12. Interpreting Language Models with Contrastive Explanations (BP session)
• BLiMP

13. Interpreting Language Models with Contrastive Explanations (BP session)
BLiMP
• ANAPHOR AGREEMENT: the requirement that reflexive pronouns like himself (a.k.a. anaphora)
agree with their antecedents in person, number, gender, and animacy.
• ARGUMENT STRUCTURE: the ability of different verbs to appear with different types of arguments.
For instance, different verbs can appear with a direct object, participate in the causative alternation,
or take an inanimate argument.
• BINDING: the structural relationship between a pronoun and its antecedent. All paradigms
illustrate aspects of Chomsky’s (1981) Principle A. Because coindexation cannot be annotated in
BLiMP, Principles B and C are not illustrated.
• CONTROL/RAISING: syntactic and semantic differences between various types of predicates that
embed an infinitival VP. This includes control, raising, and toughmovement predicates.
• DETERMINER-NOUN AGREEMENT: number agreement between demonstrative determiners (e.g.,
this/these) and the associated noun.
• ELLIPSIS: the possibility of omitting expressions from a sentence. Because this is difficult to
illustrate with sentences of equal length, our paradigms cover only special cases of noun phrase
ellipsis that meet this constraint.

14. Interpreting Language Models with Contrastive Explanations (BP session)
BLiMP
• FILLER-GAP: dependencies arising from phrasal movement in, for example, whquestions.
• IRREGULAR FORMS: irregular morphology on English past participles (e.g., broken). We are unable
to evaluate models on nonexistent forms like *breaked because such forms are out of the
vocabulary for some LMs.
• ISLAND EFFECTS: restrictions on syntactic environments where the gap in a filler-gap dependency
may occur.
• NPILICENSING: restrictions on the distribution of negative polarity items like any and ever limited
to, for example, the scope of negation and only.
• QUANTIFIERS: restrictions on the distribution of quantifiers. We cover two such restrictions:
superlative quantifiers (e.g., at least) cannot embed under negation, and definite quantifiers and
determiners cannot be subjects in existential-there constructions.
• SUBJECT-VERB AGREEMENT: subjects and present tense verbs must agree in number.

15. Interpreting Language Models with Contrastive Explanations (BP session)

16. Interpreting Language Models with Contrastive Explanations (BP session)
RULE
• Anaphor Agreement: The gender and the number of a pronoun must agree with its antecedent.
We implement the ‘coref’ rule using spaCy and NeuralCoref to extract all input tokens that are
coreferent with the target token
• Argument Structure: Certain arguments can only appear with certain types of verbs. For example,
action verbs must often be used with animate objects. We implement the ‘main_verb’ rule using
spaCy to extract the main verb of the input sentence.
• Determiner-Noun Agreement: Demonstrative determiners and the associated noun must agree. We
implement the ‘det_noun’ rule by generating the dependency tree using spaCy and extracting the
determiner of the target noun.

17. Interpreting Language Models with Contrastive Explanations (BP session)
RULE
• NPI Licensing: Certain negative polarity items (NPI) are only allowed to appear in certain contexts,
e.g. “never” appears on its own in sentences, while the word “ever” generally must be preceded by
“no”. In all of our examples with NPI licensing, the word “even” is an NPI that can appear in the
acceptable example but not in the unacceptable example, so we create the npi rule that extracts
this NPI.
• Subject-Verb Agreement: The number of the subject and its verb in the present tense must agree.
We implement the ‘subj_verb’ rule by generating the dependency tree using spaCy and extracting
the subject of the target verb.

18. Interpreting Language Models with Contrastive Explanations (BP session)
Alignment Metrics
• Probes Needed
• We measure the number of tokens we need to probe, based on the explanation, to find a
token that is in the know evidence.
• Mean Reciprocal Rank (MRR)
• We calculate the average of the inverse of the rank of the first token that is part of the known
evidence if the tokens are sorted in descending saliency.

19. Interpreting Language Models with Contrastive Explanations (BP session)

20. Interpreting Language Models with Contrastive Explanations (BP session)

21. Interpreting Language Models with Contrastive Explanations (BP session)

22. Interpreting Language Models with Contrastive Explanations (BP session)

23. Interpreting Language Models with Contrastive Explanations (BP session)

24. Interpreting Language Models with Contrastive Explanations (BP session)

25. Interpreting Language Models with Contrastive Explanations (BP session)

26. Interpreting Language Models with Contrastive Explanations (BP session)

27. Interpreting Language Models with Contrastive Explanations (BP session)

28. Interpreting Language Models with Contrastive Explanations (BP session)

29. Interpreting Language Models with Contrastive Explanations (BP session)

30. Interpreting Language Models with Contrastive Explanations (BP session)