Extract linguistic features, hypothesis testing and validating linguistic rules.
So our user interfaces are designed for the users who would prefer a visualized and summarized view of the data in sinmin in a simple way while keeping all the functionalities required
Sinmin Literature Review Presentation
CORPUS FOR SINHALA LANGUAGE
Upeksha W. D.
Wijayarathna D. G. C. D.
Siriwardena M. P.
Lasandun K. H. L.
Dr. Chinthana Wimalasuriya
Prof. Gihan Dias
Mr. N. H. N. D. De Silva
Sinmin is a Corpus for Sinhala language which is
➢Covers wide range of language (Structured and
● Literature Review
● Introduction to corpus linguistics and What is a Corpus
● Usages of a corpus
● Existing Corpus Implementations
● Identifying Sinhala Sources and Crawling
● Data Storage and Information Retrieval from Corpus
● Information Visualization
● Extracting Linguistic Feature
● Current Progress
INTRODUCTION TO CORPUS LINGUISTICS
AND WHAT IS A CORPUS
Handford, M. and McCarthy, M. J. (2004) “Invisible to us” - A preliminary corpus based study of spoken
business english, Discourse In the Profession: Perspectives from Corpus Linguistics 167-201
WHAT IS A CORPUS??
“A corpus is a principled collection of authentic texts
stored electronically that can be used to discover
information about language that may not have been
noticed through intuition alone.” - Bennet (2010)
Bennet, G. R. (2010) Using Corpora in the Language Learning Classroom,Michigan ELT.
● There are mainly 8 kinds of corpora.
● They are generalized corpuses, specialized corpuses,
learner corpuses, pedagogic corpuses, historical
corpuses, parallel corpuses, comparable corpuses,
and monitor corpuses.
● The broadest type of corpus is the genarilezed
be a generalized corpus.
cover all types of Sinhala Language.
USAGES OF A CORPUS
● Implementing translators, spell checkers and grammar
● Identifying lexical and grammatical features of a language.
● Identifying varieties of language of context of usage and
● Retrieving statistical details of a language.
● Providing backend support for tools like OCR, POS Tagger,
● There is a implemented corpus for Sinhala language
which is known as UCSC Text Corpus of
● It consists of about 10 million words, but it covers
very little amount of language and it is not updating.
CORPUS FOR SINHALA LANGUAGE?
COMPOSITION OF THE CORPUS
● Language comprising the corpus cannot be random
but chosen according to specific characteristics.
● It must use authentic texts. The language it contains
is not made up for the sole purpose of creating the
EXAMPLE - COMPOSITION OF COCA
● The COCA contains more than 385 million words
from 1990–2008 (20 million words each year).
● Texts are evenly divided between 5 genres, spoken
(20%), fiction (20%), popular magazines (20%),
newspapers (20%) and academic journals (20%).
COMPOSITION OF UCSC TEXT CORPUS OF
DATA STORAGE AND INFORMATION
RETRIEVAL FROM CORPUS
Existing corpora uses two main technologies for data
● Relational Databases
● Indexed file Systems
INDEXED FILE SYSTEMS AS STORAGE
● BNC uses this mechanism.
● data is stored as XML like files which follows a
scheme known as the Corpus Data Interchange
● This supports to store a great deal of detail about the
structure of each text, such as its division into
sections or chapters, paragraphs, verse lines, etc.
RELATIONAL DATABASE AS STORAGE
● COCA, Corpus del Español use relational databases.
CORPUS DEL ESPAÑOL USES SEPARATE
TABLES FOR BIGRAMS AND TRIGRAMS
RELATIONAL DB VS INDEXED FILE
● Indexed file systems use extensive use of indexes
● Relational Database models are relatively fast.
● In Indexed file systems, difficult to add additional
layers of annotation.
No study has been done on
how NoSQL performs in
Most of the popular corpora like BNC, COCA, Corpus
Del Espanol, Google books corpus use similar kind of
EXTRACTING LINGUISTIC FEATURES
● A main usage of a language corpus is extracting
linguistic features of a language.
● Linguistic features for many languages has been
identified using Corpora.
● Example - A corpus-based linguistics analysis on
written corpus: colligation of “TO” and “FOR.”
INFORMATION RETRIEVAL PERFORMANCE
COMPARISON - PART 1
INFORMATION RETRIEVAL PERFORMANCE
COMPARISON - PART 2
Cassandra performed better than others in
most of the scenarios, and its insertion
time increased linearly.
So we chose it for implementing the
USER INTERFACE DESIGN AND
● Web interface of Sinmin has been designed for users
who would prefer a visualised and summarized view
of statistical data of Sinmin.
● Visual design of the interface has been made in a
way that any user without prior experience of the
interface is able to fulfill his information
requirements with little effort.
CORPUS API DESIGN AND IMPLEMENTATION
• REST API to expose Corpus services
• Much complex and customizable data retrieval and
• Interface for third party applications to consume
● Comparison between performance of various
database systems for implementing a language
corpus - 11th Beyond Databases, Architectures and
Structures conference (Pending)
● Implementing a Corpus for Sinhala Language -
Symposium on Language Technology for South Asia
REMAINING WORK FOR THE NEXT PHASE
• Finish writing crawlers
• Feed data to Cassendra database
• Connecting front end with API calls