Recent progress in language model pre-training has led to important improvements in Named Entity Recognition (NER). Nonetheless, this progress has been mainly tested in well-formatted documents such as news, Wikipedia, or scientific articles. In social media the landscape is different, in which it adds another layer of complexity due to its noisy and dynamic nature. In this paper, we focus on NER in Twitter, one of the largest social media platforms, and construct a new NER dataset, TweetNER7, which contains seven entity types annotated over 11,382 tweets from September 2019 to August 2021. The dataset was constructed by carefully distributing the tweets over time and taking representative trends as a basis. Along with the dataset, we provide a set of language model baselines and perform an analysis on the language model performance on the task, especially analyzing the impact of different time periods. In particular, we focus on three important temporal aspects in our analysis: short-term degradation of NER models over time, strategies to fine-tune a language model over different periods, and self-labeling as an alternative to lack of recently-labeled data. TweetNER7 is released publicly (this https URL) along with the models fine-tuned on it (NER models have been integrated into TweetNLP and can be found athttps://github.com/asahi417/tner/tree/master/examples/tweetner7_paper).

1. Named Entity Recognition in Twitter:
A Dataset and Analysis on Short-Term Temporal Shifts
Computer Science & Informatics, Cardiff University, Cardiff NLP
Snap Inc.,
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
https://huggingface.co/datasets/tner/tweetner7

2. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Named-Entity Recognition (NER)
2
Jacob Collier is an English artist.
BERT + Linear Projection
Jacob is English
an artist.
PJacob PCollier Pan PEnglish Partist
Pis
Collier
Tokenization
Location
Person

3. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
NER on Social Media
● Noisy Text
○ Less formal.
○ Different vocabulary.
○ Emoji, urls, etc.
● Temporal Shift
○ The meaning of words is constantly changing or evolving over time.
○ New entities in new period.
3

4. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
TweetNER7
TweetNER7 is a NER dataset on Twitter with 7 entity types.
● Person, Location, Corporation, Creative work, Group, Product.
● Tweets are collected from Sep 2019 to Aug 2021.
○ 2020-set: Sep 2019 ~ Aug 2020 (5,768 tweets)
○ 2021-set: Sep 2020 ~ Aug 2021 (5,612 tweets)
● Temporal shift setup
○ Training/Validation: 2020-set
○ Test: 2021-set
4
https://huggingface.co/datasets/tner/tweetner7

5. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Other Twitter NER Datasets
TweetNER7 is unique for
● Uniform distribution over months
○ 2,000 tweets in each month
● (Relatively) short term temporal shift
○ Over 2 years
● Large annotation (data size & entities)
○ More than 10k annotated tweets with 7 entities
5

6. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Dataset Collection
TweetTopic tweet collection:
● Query 50 tweets every two hours
from September 2019 to October
2021
● Pre-filtering & Near de-duplication
● Trend-filter
6
Tweet collection pipeline.
Trend-filter

7. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Preprocessing Tweets
● Special token for URLs.
● Special representation for
usernames.
7
Raw tweet.
Processed tweet.

8. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Dataset Annotation
● Mechanical Turk:
○ 3 annotators per tweet
● Agreement
○ 1 / 3: Disregard
○ 2 / 3: Manual check
8

9. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Experiment: Temporal Shift Setup
Setup: Train & validate on 2020-set & test
on 2021-set
Metric: Micro F1 / Macro F1
Result:
● RoBERTa is the best in 2021.
● BERTweet is the best in 2020.
● Performance: 2021 > 2020.
9
Model
Micro F1
(2021/2020)
Macro F1
(2021/2020)
BERTweet Base 64.1 / 66.4 59.4 / 62.4
BERTweet Large 64.0 / 65.9 59.5 / 62.6
RoBERTa Base 64.2 / 64.2 59.1 / 60.2
RoBERTa Large 64.8 / 65.7 60.0 / 61.9
TimeLM2019 64.3 / 65.4 59.3 / 61.1
TimeLM2020 62.9 / 64.4 58.3 / 60.3
TimeLM2021 64.2 / 65.4 59.5 / 61.1

10. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Results Breakdown
Easy entities:
Person, product, location
● Large number of entities
● Low diversity
Challenging entities:
Creative work, event
● Small number of entities
● High diversity
10

11. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Experiment: Continuous Fine-tuning
Setup: Use both of 2020 & 2021 training
set by concatenation or continuous
fine-tuning.
Result:
● Continuous fine-tuning achieves
the best results.
● TimeLM19 is better than TimeLM20
and TimeLM21.
11
Model Dataset
Micro F1
(2021/2020)
Macro F1
(2021/2020)
RoBERTa
LARGE
2020-set 64.8 / 65.7 60.0 / 61.9
Continuous 66.0 / 66.3 60.9 / 62.4
TimeLM
2019
2020-set 64.3 / 65.4 59.3 / 61.1
Continuous 65.9 / 64.8 61.1 / 60.6
TimeLM
2020
2020-set 62.9 / 64.4 58.3 / 60.3
Continuous 65.5 / 65.3 60.6 / 61.3
TimeLM
2021
2020-set 64.2 / 65.4 59.5 / 61.1
Continuous 65.1 / 64.9 60.0 / 60.7

12. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Experiment: Self-labeling
Setup: Collect additional 93,594 and
878,80 tweets from the period of
2020-set and 2021-set. Generate pseudo
annotation by fine-tuned RoBERTa Large
model.
Result: No significant improvements.
12
Dataset
Micro F1
(2021)
Macro F1
(2021)
2020-set 64.8 60
2020-self-labeling 64.6 59.3
2021-self-labeling 64.2 59.3

13. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Are the pseudo labels not useful at all?
Setup: For incorrect prediction,
retrieve the pseudo-labels for the
entity within the range of 7 days,
and take the most frequent
prediction as the contextualized
prediction.
13

14. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Contextualized prediction usually
matches the original one, meaning is
wrong 😩
But, the second most frequent
prediction is correct in average 🤔
There is a signal at least, but not easy
way to utilize it.
14
Result of Contextualized Prediction

15. Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts
Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, Jose Camacho-Collados
Summary
● TweetNER7, a NER dataset on Twitter with temporal-shift and 7 entity types
● Report LM fine-tuning results
○ Temporal split vs Random split
○ Continuous fine-tuning
● Self-labeling
○ Contextual prediction
15

16. Thank you!!