investigate the problem of automatically predicting topic boundaries in texts

1. Bank of America is to sue ABN Amro after it was blocked from buying ABN's US
bank, complicating bids for ABN from Barclays and Royal Bank of Scotland.
Bank of America launched the action after a Dutch court froze its planned
purchase of LaSalle, ruling the deal was illegal without investor approval.
Pakistan have included Mohammad Asif in a 16-man squad for a three-match
series with Sri Lanka in Abu Dhabi this month.
RAW TEXT – collection of news from the site http://news.bbc.co.uk
Bank/NNP of/IN America/NNP is/VBZ to/TO sue/VB ABN/NNP Amro/NNP
after/IN it/PRP was/VBD blocked/VBN from/IN buying/VBG ABN/NNP 's/POS
US/NNP bank/NN ,/, complicating/VBG bids/NNS for/IN ABN/NNP from/IN
Barclays/NNP and/CC Royal/NNP Bank/NNP of/IN Scotland/NNP ./.
Bank/NNP of/IN America/NNP launched/VBD the/DT action/NN after/IN a/DT
Dutch/JJ court/NN froze/VB its/PRP$ planned/JJ purchase/NN of/IN
LaSalle/NNP ,/, ruling/VBG the/DT deal/NN was/VBD illegal/JJ without/IN
investor/NN approval/NN ./.
Pakistan/NNP have/VBP included/VBN Mohammad/NNP Asif/NNP in/IN a/DT
16-man/JJ squad/NN for/IN a/DT three-match/JJ series/NN with/IN Sri/NNP
Lanka/NNP in/IN Abu/NNP Dhabi/NNP this/DT month/NN ./.
TAGGING MODULE – using LPOST (a Perl program that uses a
variant of the Brown/Penn-style tag-sets) or POStagger (a program
designed at Tokyo University)

2. Bank/NNP America/NNP sue/VB ABN/NNP Amro/NNP blocked/VBN buying/VBG
ABN/NNP 's/POS US/NNP bank/NN ,/, complicating/VBG bids/NNS ABN/NNP
Barclays/NNP Royal/NNP Bank/NNP Scotland/NNP ./.
Bank/NNP America/NNP launched/VBD action/NN Dutch/JJ court/NN froze/VB
planned/JJ purchase/NN LaSalle/NNP ,/, ruling/VBG deal/NN illegal/JJ
investor/NN approval/NN ./.
Pakistan/NNP included/VBN Mohammad/NNP Asif/NNP 16-man/JJ squad/NN
three-match/JJ series/NN Sri/NNP Lanka/NNP Abu/NNP Dhabi/NNP month/NN ./.
STOPWORDS ELIMINATION

3. MODULE 1
UNIGRAMS –
keep only the important
words from texts
(substantives and verbs) and
group the texts if possible
N-GRAMS –
extract the n-grams (we
used bigrams) from texts
and group the texts if
possible
COLLOCA
TION –
extract the
collocation
from texts
Bank/NNP America/NNP,
America/NNP sue/VB, sue/VB
ABN/NNP …
Bank/NNP America/NNP,
America/NNP launched/VBD,
launched/VBD action/NN …
Pakistan/NNP included/VBN,
included/VBN
Mohammad/NNP,
Mohammad/NNP Asif/NNP …
Verbs sue, block, buy,
complicate
Nouns bank, bid
Verbs launch, rule
Nouns action, court, deal,
purchase, investor,
approval
Verbs include
Nouns squad, series,
month
-
-
-

4. MODULE 2
Bank/NNP America/NNP sue/VB ABN/NNP Amro/NNP blocked/VBN
buying/VBG ABN/NNP 's/POS US/NNP bank/NN ,/, complicating/VBG bids/NNS
ABN/NNP Barclays/NNP Royal/NNP Bank/NNP Scotland/NNP ./.
Bank/NNP America/NNP launched/VBD action/NN Dutch/JJ court/NN froze/VB
planned/JJ purchase/NN LaSalle/NNP ,/, ruling/VBG deal/NN illegal/JJ
investor/NN approval/NN ./.
Pakistan/NNP included/VBN Mohammad/NNP Asif/NNP 16-man/JJ squad/NN
three-match/JJ series/NN Sri/NNP Lanka/NNP Abu/NNP Dhabi/NNP
month/NN ./.
The words from the text are grouped into lexical chains and after all the
words have been classified, the topic shifts can be identified by
assuming that a high density of chain starts and ends means that the
topic has changed.

5. MODULE 3
The words from the text are grouped into lexical chains as described in
Module 2 and after that, every word is replaced by the chain that is part
of. Next, every paragraph is assigned to the chain that contains the
majority of the words from that paragraph. If two consecutive
paragraphs are assigned to different chains, it means that a topic
changed occurred.
Bank/NNP America/NNP c1 ABN/NNP Amro/NNP c2 c3 ABN/NNP
's/POS US/NNP c3 ,/, c2 c3 ABN/NNP Barclays/NNP Royal/NNP
Bank/NNP Scotland/NNP ./.
Bank/NNP America/NNP c4 c1 c5 c1 c2 c6 c3 LaSalle/NNP ,/, c1 c3
c1 c3 c3 ./.
Pakistan/NNP c7 Mohammad/NNP Asif/NNP c8 c8 c8 c9 Sri/NNP
Lanka/NNP Abu/NNP Dhabi/NNP c10 ./.
 c3
 c1/c3
 c8

6. VOTING MODULE
The results obtained from the three modules are combined. After
each paragraph, the outputs of the modules are investigated in order
to find out whether it should be a topic limit or not.
Paragraph 1
Paragraph 2
Paragraph 3
MODULE 1
Paragraph 1
Paragraph 2
Paragraph 3
MODULE 2
Paragraph 1
Paragraph 2
Paragraph 3
MODULE 3
Paragraph 1
Paragraph 2
Paragraph 3
FINAL RESULTS

7. Raw
text
Voting
Module
Final
result
Tagging
Module
Figure 1. Application architecture
Stopwords
elimination
Module 3Module 2Module 1
Result 3Result 2Result 1

8. Text
neadnotat
Modul
de votare
Rezultate
finale
Modul de
adnotare
Figura 3. Arhitectura Aplicatiei
Eliminarea
cuvintelor de
stop
Modulul n
Rezultatul n
Modulul 1
Rezultatul 1
Modulul 2
Rezultatul 2