Transformer models have achieved state-of-the-art results for news classification tasks, but remain difficult to modify to yield the desired class probabilities in a multi-class setting. Using a neural topic model to create dense topic clusters helps with generating these class probabilities. The presented work uses the BERTopic clustered embeddings model as a preprocessor to eliminate documents that do not belong to any distinct cluster or topic. By combining the resulting embeddings with a Sentence Transformer fine-tuned with SetFit, we obtain a prompt-free framework that demonstrates competitive performance even with few-shot labeled data. Our findings show that incorporating BERTopic in the preprocessing stage leads to a notable improvement in the classification accuracy of news documents. Furthermore, our method outperforms hybrid approaches that combine text and images for news document classification.

1. Mohamad Al Sayed
Adrian M.P. Brașoveanu
Lyndon J.B. Nixon
Arno Scharl
Unsupervised Topic Modeling with BERTopic
for Coarse and Fine-Grained News
Classification

2. Use Case: IPTC
● IPTC classification
● 17 domains with subcategories
● Initial approach: Silver Standard
● Assignments based on URLs
● Multiple categories possible
● Needed for news media projects

3. Challenges with Current Text Classification Pipelines
● Data Quality
● Scalability
● Interpretability
● Imbalanced Classes
● Computational Costs
● Model Overfitting

4. Neural Topic Modeling: BERTopic
● Text Preprocessing
● Document Embedding
● Dimensionality Reduction
● Clustering
● Topic Representation

5. Our Solution - Method
● Extract a coherent topic representation using BERTopic.
● Use the documents to train any classifier (RoBERTa, SetFit, etc.)

6. Experiment - Datasets

7. Experiment - Models
● RoBERTa (Robustly Optimized BERT Pretraining Approach)
● SetFit (Few-shot)

8. Experiment - Embeddings Evaluation
● RoBERTa Embeddings win

9. Experiment - Model Evaluation (with and without BERTopic)

10. Portal Integration

11. Conclusion
● Improved Content Understanding
● Customization
● Improved Analytics
● Mitigation of Information Overload
● Improvements on Downstream Tasks

12. Thank you for your attention!
Acknowledgments
GENTIO project funded by the Austrian Federal Ministry for Climate Action,
Environment, Energy, Mobility and Technology (BMK) via the ICT of the Future
Program (GA No. 873992).
DWBI project funded by Vienna Science and Technology Fund (WWTF)
[10.47379/ICT20096].
Images
Copyright © IPTC, HuggingFace, Mohamad Al Sayed