This document discusses Arabic tokenization and stemming. It presents research on tokenizing and stemming Arabic text. For tokenization, the researchers used a bigram approach that achieved 98.83% accuracy on a dataset of 29,092 tokens. For stemming, they evaluated root-based, light, n-gram, and hybrid approaches. Their hybrid method, which incorporates roots and light stemming with n-grams, performed best, achieving 82.33% accuracy for Arabic text categorization. The document concludes the hybrid approach is effective for Arabic processing tasks.

1. Imam Mohammad Ibn Saud
Islamic University
College of Computing and
Information Science
Computer sciences Department
Prepared by:
Al-Moammar.A., Al-Abdullah.H., and Al-Ajlan.N
Arabic Tokenization and
Stemming
Supervised by:
Dr. Amal Al-Saif.

2. Arabic Tokenization and
Stemming

3. Outline
 Introduction
 Tokenization:
• Arabic Characteristics.
• Methodology.
• Result.
 Stemming:
• Arabic Characteristics.
• Methodology.
• Results.
 Conclusion.

4. Introduction
 Arabic language.
 Tokenization.
 Stemming.

5. Outline
 Introduction
 Tokenization:
• Arabic Characteristics.
• Methodology.
• Result.
 Stemming:
• Arabic Characteristics.
• Methodology.
• Results.
 Conclusion.

6. Arabic Language Characteristics
• Writing the letter in ambiguous case cause orthography problems.
• Encliticization of a word ending with “ ” or “ ” :
• Ambiguity results from decliticization of “ ” “l” “ ” “A” and “ ” “Al” “the”.
word Encliticization of word
“their Friday”
“collect them”
“Your level”

7. Outline
 Introduction
 Tokenization:
• Arabic Characteristics.
• Methodology.
• Result.
 Stemming:
• Arabic Characteristics.
• Methodology.
• Results.
 Conclusion.

8. My Approach
 Sample of Arabic tokenized text:
 The Bigrams equation that used is:
P(wi | sj) is probability of ith word given jth segmentation.
P(sj | si-1)is probability of jth segmentation given previous segmentation.

9. Outline
 Introduction
 Arabic Characteristics.
 Tokenization:
• Arabic Characteristics.
• Methodology.
• Result.
 Stemming:
• Arabic Characteristics.
• Methodology.
• Results.
 Conclusion.

10. Results
The result of My Approach algorithm:
• They used Bigrams on 45 files with size of 29092 tokens.
• The final accuracy was 98.83%.
Recall Accuracy Precision F-measure
Result without statistical
support
0.9877462 0.9802977 0.8617793 0.920473

11. Outline
 Introduction
 Arabic Characteristics.
 Tokenization:
• Methodology.
• Result.
 Stemming:
• Arabic Characteristics.
• Methodology.
• Results.
 Conclusion.

12. Arabic Language Characteristics

13. Outline
 Introduction
 Arabic Characteristics.
 Tokenization:
• Methodology.
• Result.
 Stemming:
• Arabic Characteristics.
• Methodology.
• Results.
 Conclusion.

14. Methodology
 Root-based.
 Light Stemmer.
 N-Gram.
 Hybrid Method.

15. Root-based
 Example of root-based stemmer

16. Light Stemmer
 Removed morphemes by Light stemmers

17. Light Stemmer
 Classification of Light8 stemmer

18. N-gram
 Statistical stemmer based on calculating a measure of
similarity between a pair of words.
 N-gram techniques:
• Digram.
• Trigram.

19. N-gram
N-gram techniques:
• ( )
• Digram (N=2)
“
• Trigram (N=3)

20. N-gram
 The string similarity measures calculated using Dice’s
Coefficient:
S = 2Cwq /(Aw + Bq)
Example :
“
would be:
(2 * 4/(10 +5) = 0.533).

21. Outline
 Introduction
 Arabic Characteristics.
 Tokenization:
• Methodology.
• Result.
 Stemming:
• Arabic Characteristics.
• Methodology.
• Results.
 Conclusion.

22. Hybrid Method
 Incorporates three different techniques for Arabic Stemming.
 The Hybrid algorithm starts with constructing the root file
containing more than 9,000 valid Arabic roots.

23. Results

24. Results
 Hybrid algorithm was found to supersede the other
stemming ones.
 The obtained results illustrate that using the hybrid stemmer
enhances the performance of some Arabic process.
 In Arabic Text Categorization: the averages accuracies are:
74.41% for khoja, 59.71% for light stemming, 48.17% for
n-grams, and 82.33% for Hybrid stemmer.

25. Outline
 Introduction
 Arabic Characteristics.
 Tokenization:
• Methodology.
• Result.
 Stemming:
• Arabic Characteristics.
• Methodology.
• Results.
 Conclusion.

26. Conclusion

27. Thanks