Sinmin is a corpus for the Sinhala language that is continuously updating and scalable. It covers a wide range of structured and unstructured Sinhala language data from sources like news, academic writings, fiction, and more. The architecture includes crawlers to fetch web pages, data cleaning mechanisms to handle issues like erroneous characters and short forms, and Cassandra as the main storage system. The API and user interface allow users to perform queries on the corpus to obtain word and ngram frequencies, latest articles, and more. Performance testing was conducted on the storage systems and API.

1. Sinmin - Corpus for Sinhala
Language
1
Upeksha W. D.
Wijayarathna D. G. C. D.
Siriwardena M. P.
Lasandun K. H. L.
Supervisors :
Dr. Chinthana Wimalasuriya
Prof. Gihan Dias
Mr. N. H. N. D. de Silva

2. Outline
● Introduction
● Crawler Implementation and Design
● Data Cleaning and Tokenizing Mechanisms
● Selecting Data Storage Mechanism
● Data Storage Model of SinMin
● User Interface Design and Implementation
● API Design and Implementation
● Unit Testing
● Performance Testing of the API
● Implemented Sample Usages
2

3. What is a Corpus??
“A corpus is a principled collection of
authentic texts stored electronically that
can be used to discover information about
language that may not have been noticed
through intuition alone.” - Bennet (2010)
3

4. Usages of a Corpus
● Implementing translators, spell checkers and grammar
checkers.
● Identifying lexical and grammatical features of a language.
● Identifying varieties of language of context of usage and
time.
● Retrieving statistical details of a language.
● Providing backend support for tools like OCR, POS Tagger,
etc.
4

5. Sinmin is a Corpus for Sinhala
language which is
➢ Continuously updating
➢ Dynamic (Scalable)
➢ Covers wide range of language (Structured and unstructured)
5

6. Architecture of Sinmin
6

7. Identified Sinhala Resources
7
News Academic Creative
Writing
Spoken Gazette
News Paper Text books Fiction Subtitle Gazette
News Items Religious Blogs
Wikipedia Magazine
mahawansa

8. Identified Sinhala Resources
8

9. Crawler Implementation and
Design
9

10. Crawlers are responsible of finding
web pages that contain sinhala
content, fetching, parsing and storing
them in a manageable format.
10

11. Crawler Architecture
11

12. Sample Xml File With One Article Stored In It
12

13. Crawler Controller
Crawler controller monitors and handles
the status of the web crawlers.
13

14. 14

15. Data Cleaning and Tokenizing
Mechanisms used
15

16. Identified Issues
● Erroneous characters of the texts
● Short forms
● Consecutive Sinhala vowel sign problem
fixing
16

17. Erroneous Characters Of The
Texts
● Invalid Unicode characters
Eg: Characters in a private user area, Replacement
character
● Symbols
Eg: “,”, “.”, “{“, “(“, “?”
17

18. Erroneous Characters Of The
Texts
● Unwanted non-Sinhala characters
Eg: ‘u+200C’, Á, À, ®, ¡, ª, º
● Non-symbolic characters which were terminating
words
18

19. Short Forms
● Short forms consists of full stops.
● But those full stop marks aren’t separating
sentences nor words.
E.g.: පෙ. ව. (pm), රු. (Rupees)
19

20. Identified Common Short
Forms
"ඒ.", "බී.", "සී.", "ඩී.", "ඊ.", "එෆ්."
"පෙ.", "ව.", "ෙ.", "රු."
"0.", "1.", "2.", "3.", "4.", "5.", "6.", "7.", "8.", "9."
20

21. Consecutive Sinhala Vowel Sign
Problem
21

22. Consecutive Sinhala Vowel Sign
Problem
● Solution: Mapping them into one format
● Convention: Only one vowel sign to a Sinhala
letter
22

23. Consecutive Sinhala Vowel Sign
Problem
23

24. Selecting Data Storage
Mechanism for Sinmin
24

25. The performance of data insertion and
retrieval mainly depend on the Data
Storage Mechanism used for the
Corpus.
25

26. We tested performance of several
database systems to determine what
should we use to store data.
26

27. We Considered Following Data
Storage Systems
27

28. We considered performance for
inserting data and for retrieving 12
different information needs.
Data set and source code
https://github.com/madurangasiriwardena/performance-test
28

29. Data Insertion Time Comparison
29

30. Information Retrieval Performance
Comparison - Part 1
30

31. Information Retrieval Performance
Comparison - Part 2
31

32. Cassandra performed better than others
in most of the scenarios, and its
insertion time increased linearly.
So we chose it for implementing
corpus.
32

33. Data Storage Model of Sinmin
33

34. ● We Used Cassandra as the Main Storage System of
Sinmin
● Apache Cassandra version 2.1.2 used.
● cqlsh version 5.0.1 used
34
Cassandra

35. ● Most queries of API are retrieved from Cassandra
Database.
● Cassandra Database consist of more than 50 Column
Families where each of them provides a specific
information need
35
Cassandra

36. ● Oracle used as a backup storage server.
36
Oracle

37. Oracle
Schema
37

38. Wildcard Search Feature
Wildcard search feature enables users to run wild-
card queries on the corpus
Eg: පෙ?
ෙහ*
38

39. Wildcard Search Feature
● Implemented using Apache Solr
● More than 1.2 million distinct words
● Supports at most 10 asterisks and atmost 10
question marks
39

40. Sinhala Vowel Sign Problem At Wildcard
Search
In Sinhala Unicode, Sinhala vowel signs are separate
Unicode characters
40

41. Sinhala Vowel Sign Problem At Wildcard
Search
Solution: Represent Sinhala letter and vowel sign as one
entity
41

42. User Interface Design and
Implementation
42

43. ● Web interface of Sinmin has been designed for users
who would prefer a visualised and summarized view
of statistical data of Sinmin.
● Visual design of the interface has been made in a
way that any user without prior experience of the
interface is able to fulfill his information
requirements with little effort.
43

44. Sinmin user interface allows to,
● Find the probability of an n-gram
● Find the most probable word comes after an n-gram
● Compare the usage of n-grams
● Find statistics of words, bigrams and trigrams
● Wildcard search
● Find latest articles for an n-gram
44

45. 45

46. 46

47. API Design and Implementation
47

48. REST API
● REST API to expose Corpus services
● Much complex and customizable data retrieval and
filtering
● Interface for third party applications to consume
48

49. REST API
● Depends on backend databases (Cassandra,
Oracle, Solr)
● Cassandra acts as main storage system
● Oracle is used as a backup database
● Solr is used for wildcard search functions
49

50. Architecture
50

51. API Functions
● wordFrequency
● bigramFrequency
● trigramFrequency
● frequentWords
● frequentBigrams
● frequentTrigrams
● latestArticlesForWord
● latestArticlesForBigram
● latestArticlesForTrigram
51
● frequentWordsAroundWord
● frequentWordsInPosition
● frequentWordsInPositionReverse
● frequentWordsAfterWordTimeRange
● frequentWordsAfterBigramTimeRange
● wordCount
● bigramCount
● trigramCount

52. Performance Testing of the API
52

53. Throughput Under Different Load
Conditions
53

54. Time Taken To Process Requests Under
Different Load Conditions
54

55. Full Stop Predictor For OCR
● One challenge in OCR development is identifying
fullstops.
● This tool is a consumer application of Sinmin that
predicts the full stop marks of Sinhala texts.
55

56. Publications
● Implementing a Corpus for Sinhala Language -
Symposium on Language Technology for South Asia
(Presented)
● Comparison between performance of various
database systems for implementing a language
corpus – 11th International Beyond Databases,
Architectures and Structures conference (Accepted)
56

57. Future Works
● Annotate Words with POS Taggers and lemmas.
● Implement tools and applications that make use of
the corpus
57

58. Q & A
58

59. Thank You!
59