This project was awarded 2nd best project at IASNLP 2017 held at IIIT Hyderabad.

1. Hello!

2. Language
Identification system
for
code-mixed Social Media Text
Analysis

3. Multilingualspeakersoftenswitchbetweenlanguages.
Mixingmultiplelanguagestogether(codemixing)isapopulartrend
insocialmediausers.
Thiscomplicatesautomaticlanguageidentificationasitisshifted
fromdocumentleveltowordlevel.
Challengesproblems

4.  Enhancesocialmediaanalysisinlanguage-denseareas.
Challengesmotivation

5. Creationof newlexicalandsyntacticstructures(e.g.code-mixingon
morphemelevel).
Interflowofdissimilargrammarwhencombininglanguages.
Classificationof particularwords/phrasesthathavebeenassimilatedinto
onelanguagefromanother.(e.g..Bottle)
Classificationof particularwordsthatexistinmorethanonelanguage.
(e.g..Hum)
Definingcanonicalformsforwordnormalization.
ChallengesChallenges
Hi

6.  Classofstatisticalmodelingmethodoftenappliedinpatternrecognitionandmachine
learningandusedforstructuredprediction.
 Featurefunctionisafunctionthattakesininputasthesentence, positionofwordandlabels
asdefined,outputsareal-valuednumber(usuallyeither0or1).
 Conversionoffeaturestoprobabilitiesbyassigningthemweightsand exponentiating,
normalizingthesummationofweightedfeatures.
ChallengesConditional random fields (crf)

7. @buttmona098 @HamzaIdrees Accha Topic Change karo
ChallengesDataset format
Univ Univ Hi HiEn En
@imp13196 ATM me Cash nhi hai
Univ Acro Hi En Hi Hi

8.  CRF++isasimple,customizable,andopensourceimplementationofCRFfor
segmenting/labelingsequentialdata.CRF++isdesignedforgenericpurposeandwillbe
appliedtoavarietyofNLPtasks,suchasNamedEntityRecognition,InformationExtraction
andTextChunking.
ChallengesCrf++ tool
% crf_learn template_file train_file model_file
% crf_test -m model_file test_files
https://taku910.github.io/crfpp/#format

9. ChallengesExample of template file

10. ChallengesExample of input file

11. Precision Recall F1-Score Support
En 0.77 0.81 0.79 2015
Ne 0.39 0.46 0.42 304
Hi 0.87 0.83 0.85 2817
Univ 0.98 0.98 0.98 1011
Mixed 0.00 0.00 0.00 3
Acro 0.81 0.55 0.66 78
Avg/Total 0.83 0.83 0.83 6228
initial approach
Precision-RecallTable:
ConfusionMatrix:

12. Affixes(Prefix,Suffix)
Lengthof Word
FirstCharacterinUppercase(BinaryFeature)
AllCharactersinUppercase(BinaryFeature)
Ifcontainsanysymbol(BinaryFeature)
Stop-WordsforEnglish
PreviousWord/NextWord Etc.
ChallengesFeatures
Total Features : 10

13. 3232 17 127 4 0 1
68 141 28 0 0 1
138 18 1315 0 0 5
4 0 0 970 0 0
0 0 0 0 0 0
12 0 4 5 0 37
Precision Recall F1-Score Support
En 0.94 0.96 0.95 3381
Ne 0.80 0.59 0.68 238
Hi 0.89 0.89 0.89 1476
Univ 0.99 1.00 0.99 974
Mixed 0.00 0.00 0.00 0
Acro 0.84 0.64 0.73 58
Avg/Total 0.93 0.93 0.93 6127
Final approach
Precision-RecallTable:
ConfusionMatrix:

14. Affixes(Prefix,Suffix)
LengthofWord
FirstCharacterinUppercase(BinaryFeature)
AllCharactersinUppercase(BinaryFeature)
Ifcontainsanysymbol(BinaryFeature)
Stop-WordsforEnglish
PreviousWord/NextWord
ListoftrainedHindiWords(BinaryFeature)
NumericalDataFeature Etc.
ChallengesFeatures (final)
Total Features : 17

15.  Probabilisticsequencemodel:givenasequenceofunits(words,letters,morphemes,
sentences),itcomputesaprobabilitydistributionoverpossiblesequencesoflabels
andchoosesthebestlabelsequence.
ChallengesHidden markov model (hmm)

16. Trigrams‘n’Tags,isaveryefficientstatisticalPOStaggerthatistrainableondifferent
languagesandvirtuallyany tagset.Thecomponentforparametergenerationtrainsontagged
corpora.Thesystemincorporatesseveralmethodsofsmoothingandofhandlingunknown
words.
ChallengesTnt tool
http://www.coli.uni-saarland.de/~thorsten/tnt/
% ./tnt-para train_file
% ./tnt train_file test_file

17. ChallengesExample of tnt

18. ChallengesExample of tnt output file

19. CRF hmm
Vs.

20. CRF : Accuracy 93 %
HMM : Accuracy 67%

21. Challenges
Thank You!
Parth Desai
Shreshta Bhat
Harish B. Manish Shrivastava
CreatedBy:
Underthesupervisionof: