3. Multilingual Collections
There are 6,703 languages listed in the Ethnologue
Digital libraries
OCLC Online Computer Library Center serves more
than 17,000 libraries in 52 countries and contains over 30
million bibliographic records with over 500 million
records ownership attached in more than 370 languages
World Wide Web
Around 40% of Internet users do not speak English,
however, 80% of Web sites are still in English
Hsin-Hsi Chen 10-3
4. The General Problem
Find documents written in any language
Using queries expressed in a single language
Traditional IR identifies relevant documents in the same
language as the query (monolingual IR)
Cross-language information retrieval (CLIR) tries to
identify relevant documents in a language different from
that of the query
This problem is more and more acute for IR on the Web
due to the fact that the Web is a truly multilingual
environment
4
6. Global Internet User Population
6Source: Global Reach
5%
9%
6%
5%
5%
4%
4%
3%
2%
2%
3%
52%
Spanish Japanese German French
Chinese Scandanavian Italian Dutch
Korean Portuguese Other English
8%
8%
5%
3%
21%
2%5%2%
5%
3%
6%
32%
Spanish Japanese German
French Chinese Scandanavian
Italian Dutch Korean
Portuguese Other English
8%
12%
6%
4%
8%
2%
5%
2%6%2%
5%
40%
Spanish Japanese German French
Chinese Scandanavian Italian Dutch
Korean Portuguese Other English
English
English
2000 2005
Chinese
7. Multilingual Collections
There are 6,703 languages listed in the Ethnologue
Digital libraries
OCLC Online Computer Library Center serves more
than 17,000 libraries in 52 countries and contains over 30
million bibliographic records with over 500 million
records ownership attached in more than 370 languages
World Wide Web
Around 40% of Internet users do not speak English,
however, 80% of Web sites are still in English
Hsin-Hsi Chen 10-7
8. Importance of CLIR
CLIR research is becoming more and more important
for global information exchange and knowledge
sharing.
National Security
Foreign Patent Information Access
Medical Information Access for Patients
8
9. CLIR is Multidisciplinary
CLIR involves researchers from the following
fields:
information retrieval, natural language
processing, machine translation and
summarization, speech processing,
document image understanding,
human-computer interaction
9
10. User Needs
Search a monolingual collection in a language that the
user cannot read.
Retrieve information from a multilingual collection
using a query in a single language.
Select images from a collection indexed with free text
captions in an unfamiliar language.
Locate documents in a multilingual collection of
scanned page images.
10
11. Why Do Cross-Language IR?
When users can read several languages
Eliminates multiple queries
Query in most fluent language
Monolingual users can also benefit
If translations can be provided
If it suffices to know that a document exists
If text captions are used to search for images
11
12. CLIR Experimental System
2 systems:
SMART Information retrieval system modified to work with 11
European languages (Danish, Dutch, English, Finnish, French,
German, Italian, Norwegian, Portuguese, Spanish, Swedish)
Generation of restricted bigrams
Pseudo-Relevance feedback
TAPIR is a language model IR system written by M. Srikanth. It
has been adated to work with 12 different European languages
(Danish, Dutch, English, Finnish, French, German, Italian,
Norwegian, Portuguese, Russian, Spanish, Swedish)
12
14. Design Decisions
What to index?
Free text or controlled vocabulary
What to translate?
Queries or documents
Where to get translation knowledge?
Dictionary, ontology, training corpus
14
15. 15
Term-aligned Sentence-aligned Document-aligned Unaligned
Parallel Comparable
Knowledge-based Corpus-based
Controlled Vocabulary Free Text
Cross-Language Text Retrieval
Query Translation Document Translation
Text Translation Vector Translation
Ontology-based Dictionary-based
Thesaurus-based
16. Problems with CLIR
Morphological processing difficult for some languages
(e.g. Arabic)
Many different encodings for Arabic
Windows Arabic (e.g. dictionaries)
Unicode (UTF-8) (e.g. corpus)
Macintosh Arabic (e.g. queries)
Normalization
Remove diacritics
ةَّيِبَرَعال to يةِبالعر Arabic (language)
Standardize spellings for foreign names
آلينتون vs آلنتون “Kleentoon” vs “Klntoon” for Clinton
16
17. Problems with CLIR (contd)
Morphological processing (contd.)
Arabic stemming
Root + patterns+suffixes+prefixes=word
ktb+CiCaC=kitab
All verbs and nouns derived from fewer than 2000 roots
Roots too abstract for information retrieval
ktb → kitab a book kitabi my book
alkitab the book kitabuki your book (f)
kataba to write kitabuka your book (m)
maktab office kitabuhu his book
maktaba library, bookstore ...
Want stem=root+pattern+derivational affixes?
No standard stemmers available,
only morphological (root) analyzers
17
18. Problems with CLIR (contd)
Availability of resources
Names and phrases are very important, most lexicons do
not have good coverage
Difficult to get hold of bilingual dictionaries
can sometimes be found on the Web
e.g. for recent Arabic cross-lingual evaluation we used 3 on-
line Arabic- English dictionaries (including harvesting) and a
small lexicon of country and city names
Parallel corpora are more difficult and require more
formal arrangements
18
19. CLIR better than IR?
How can cross-language beat within-language?
We know there are translation errors
Surely those errors should hurt performance
Hypothesis is that translation process may disambiguate
some query terms
Words that are ambiguous in Arabic may not be ambiguous in
English
Expansion during translation from English to Arabic prevents
the ambiguity from re-appearing
Has been proposed that CLIR is a model for IR
Translate query into one language and then back to original
Given hypothesis, should have an improved query
Should be reasonable to do this across many different languages
19
