Text mining and natural language processing techniques can be used to extract useful information from text data. Common text mining tasks include text categorization to classify documents into predefined categories, document clustering to group similar documents without predefined categories, and keyword-based association analysis to find frequent patterns and relationships between keywords in a collection of documents. Text classification algorithms such as support vector machines, k-nearest neighbors, naive Bayes, and neural networks can be applied to categorize documents based on their contents.
Applications of Word Vectors in Text Retrieval and Classificationshakimov
Applications of word vectors (word2vec, BERT, etc.) on problems such as text retrieval, classification of textual documents for tasks such as sentiment analysis, spam detection.
Cross-language information retrieval (CLIR) is a technique to locate documents written in one natural language by queries expressed in another language. This project investigates the feasibility of CLIR based on domain-specific bilingual corpus databases.
Applications of Word Vectors in Text Retrieval and Classificationshakimov
Applications of word vectors (word2vec, BERT, etc.) on problems such as text retrieval, classification of textual documents for tasks such as sentiment analysis, spam detection.
Cross-language information retrieval (CLIR) is a technique to locate documents written in one natural language by queries expressed in another language. This project investigates the feasibility of CLIR based on domain-specific bilingual corpus databases.
Survey of Generative Clustering Models 2008Roman Stanchak
Survey of Generative Clustering Models "Probabilistic Topic Models" circa 2008. Class presentation by Roman Stanchak and Prithviraj Sen for University of Maryland College Park cmsc828g, Link Mining and Dynamic Graph Analysis. Spring 2008. Instructor: Prof. Lise Getoor
A college level presentation covering the following topics:-
Introduction
Text mining Comparison with other mining
Text Mining Process
How Algorithm is derived for Text Mining
Text Analysis For Google Sheet
Conclusion
Over the last years, the Semantic Web has been growing steadily. Today, we count more than 10,000 datasets made available online following Semantic Web standards. Nevertheless, many applications, such as data integration, search, and interlinking, may not take the full advantage of the data without having a priori statistical information about its internal structure and coverage. In fact, there are already a number of tools, which offer such statistics, providing basic information about RDF datasets and vocabularies. However, those usually show severe deficiencies in terms of performance once the dataset size grows beyond the capabilities of a single machine. In this paper, we introduce a software component for statistical calculations of large RDF datasets, which scales out to clusters of machines. More specifically, we describe the first distributed inmemory approach for computing 32 different statistical criteria for RDF datasets using Apache Spark. The preliminary results show that our distributed approach improves upon a previous centralized approach we compare against and provides approximately linear horizontal scale-up. The criteria are extensible beyond the 32 default criteria, is integrated into the larger SANSA framework and employed in at least four major usage scenarios beyond the SANSA community.
Survey of Generative Clustering Models 2008Roman Stanchak
Survey of Generative Clustering Models "Probabilistic Topic Models" circa 2008. Class presentation by Roman Stanchak and Prithviraj Sen for University of Maryland College Park cmsc828g, Link Mining and Dynamic Graph Analysis. Spring 2008. Instructor: Prof. Lise Getoor
A college level presentation covering the following topics:-
Introduction
Text mining Comparison with other mining
Text Mining Process
How Algorithm is derived for Text Mining
Text Analysis For Google Sheet
Conclusion
Over the last years, the Semantic Web has been growing steadily. Today, we count more than 10,000 datasets made available online following Semantic Web standards. Nevertheless, many applications, such as data integration, search, and interlinking, may not take the full advantage of the data without having a priori statistical information about its internal structure and coverage. In fact, there are already a number of tools, which offer such statistics, providing basic information about RDF datasets and vocabularies. However, those usually show severe deficiencies in terms of performance once the dataset size grows beyond the capabilities of a single machine. In this paper, we introduce a software component for statistical calculations of large RDF datasets, which scales out to clusters of machines. More specifically, we describe the first distributed inmemory approach for computing 32 different statistical criteria for RDF datasets using Apache Spark. The preliminary results show that our distributed approach improves upon a previous centralized approach we compare against and provides approximately linear horizontal scale-up. The criteria are extensible beyond the 32 default criteria, is integrated into the larger SANSA framework and employed in at least four major usage scenarios beyond the SANSA community.
What is data mining?
Why data mining is required?
Data mining Applications
Data mining in Retail Industry
Marketing
Risk Management
Fraud Detection
Customer Acquisition and Retention
Abstract: Knowledge has played a significant role on human activities since his development. Data mining is the process of
knowledge discovery where knowledge is gained by analyzing the data store in very large repositories, which are analyzed
from various perspectives and the result is summarized it into useful information. Due to the importance of extracting
knowledge/information from the large data repositories, data mining has become a very important and guaranteed branch of
engineering affecting human life in various spheres directly or indirectly. The purpose of this paper is to survey many of the
future trends in the field of data mining, with a focus on those which are thought to have the most promise and applicability
to future data mining applications.
Keywords: Current and Future of Data Mining, Data Mining, Data Mining Trends, Data mining Applications.
Concept and example of a semantic solution implemented with SQL views to cooperate with users on queries over structured data with independence from database schema knowledge and technology.
Information Extraction, Named Entity Recognition, NER, text analytics, text mining, e-discovery, unstructured data, structured data, calendaring, standard evaluation per entity, standard evaluation per token, sequence classifier, sequence labeling, word shapes, semantic analysis in language technology
Presentation of the main IR models
Presentation of our submission to TREC KBA 2014 (Entity oriented information retrieval), in partnership with Kware company (V. Bouvier, M. Benoit)
Slides from my lecture for the Information Retrieval and Data Mining course at University College London
The slides cover introductory concepts on topic models, vector semantics and basic end applications
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Macroeconomics- Movie Location
This will be used as part of your Personal Professional Portfolio once graded.
Objective:
Prepare a presentation or a paper using research, basic comparative analysis, data organization and application of economic information. You will make an informed assessment of an economic climate outside of the United States to accomplish an entertainment industry objective.
Francesca Gottschalk - How can education support child empowerment.pptxEduSkills OECD
Francesca Gottschalk from the OECD’s Centre for Educational Research and Innovation presents at the Ask an Expert Webinar: How can education support child empowerment?
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
Acetabularia Information For Class 9 .docxvaibhavrinwa19
Acetabularia acetabulum is a single-celled green alga that in its vegetative state is morphologically differentiated into a basal rhizoid and an axially elongated stalk, which bears whorls of branching hairs. The single diploid nucleus resides in the rhizoid.
The Roman Empire A Historical Colossus.pdfkaushalkr1407
The Roman Empire, a vast and enduring power, stands as one of history's most remarkable civilizations, leaving an indelible imprint on the world. It emerged from the Roman Republic, transitioning into an imperial powerhouse under the leadership of Augustus Caesar in 27 BCE. This transformation marked the beginning of an era defined by unprecedented territorial expansion, architectural marvels, and profound cultural influence.
The empire's roots lie in the city of Rome, founded, according to legend, by Romulus in 753 BCE. Over centuries, Rome evolved from a small settlement to a formidable republic, characterized by a complex political system with elected officials and checks on power. However, internal strife, class conflicts, and military ambitions paved the way for the end of the Republic. Julius Caesar’s dictatorship and subsequent assassination in 44 BCE created a power vacuum, leading to a civil war. Octavian, later Augustus, emerged victorious, heralding the Roman Empire’s birth.
Under Augustus, the empire experienced the Pax Romana, a 200-year period of relative peace and stability. Augustus reformed the military, established efficient administrative systems, and initiated grand construction projects. The empire's borders expanded, encompassing territories from Britain to Egypt and from Spain to the Euphrates. Roman legions, renowned for their discipline and engineering prowess, secured and maintained these vast territories, building roads, fortifications, and cities that facilitated control and integration.
The Roman Empire’s society was hierarchical, with a rigid class system. At the top were the patricians, wealthy elites who held significant political power. Below them were the plebeians, free citizens with limited political influence, and the vast numbers of slaves who formed the backbone of the economy. The family unit was central, governed by the paterfamilias, the male head who held absolute authority.
Culturally, the Romans were eclectic, absorbing and adapting elements from the civilizations they encountered, particularly the Greeks. Roman art, literature, and philosophy reflected this synthesis, creating a rich cultural tapestry. Latin, the Roman language, became the lingua franca of the Western world, influencing numerous modern languages.
Roman architecture and engineering achievements were monumental. They perfected the arch, vault, and dome, constructing enduring structures like the Colosseum, Pantheon, and aqueducts. These engineering marvels not only showcased Roman ingenuity but also served practical purposes, from public entertainment to water supply.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
2. Mining Text and Web Data
Text mining, natural language processing and
information extraction: An Introduction
Text categorization methods
3. Mining Text Data: An Introduction
Data Mining / Knowledge Discovery
Structured Data
omeLoan (
oanee: Frank Rizzo
ender: MWF
gency: Lake View
mount: $200,000
erm: 15 years
Multimedia
Free Text
Hypertext
Frank Rizzo bought
his home from Lake
View Real Estate in
1992.
He paid $200,000
under a15-year loan
Loans($200K,[map],...) from MW Financial.
<a href>Frank Rizzo
</a> Bought
<a hef>this home</a>
from <a href>Lake
View Real Estate</a>
In <b>1992</b>.
<p>...
4. Bag-of-Tokens Approaches
Documents
Four score and seven years
ago our fathers brought forth
on this continent, a new
nation, conceived in Liberty,
and dedicated to the
proposition that all men are
created equal.
Now we are engaged in a
great civil war, testing
whether that nation, or …
Token Sets
Feature
Extraction
nation – 5
civil - 1
war – 2
men – 2
died – 4
people – 5
Liberty – 1
God – 1
…
Loses all order-specific information!
Severely limits context!
5. Natural Language Processing
A dog is chasing a boy on the playground
Det
Noun Aux
Noun Phrase
Verb
Complex Verb
Semantic analysis
Dog(d1).
Boy(b1).
Playground(p1).
Chasing(d1,b1,p1).
+
Det Noun Prep Det
Noun Phrase
Noun
Noun Phrase
Lexical
analysis
(part-of-speech
tagging)
Prep Phrase
Verb Phrase
Syntactic analysis
(Parsing)
Verb Phrase
Sentence
Scared(x) if Chasing(_,x,_).
Scared(b1)
Inference
(Taken from ChengXiang Zhai, CS 397cxz – Fall 2003)
A person saying this may
be reminding another person to
get the dog back…
Pragmatic analysis
(speech act)
6. General NLP—Too Difficult!
Word-level ambiguity
“design” can be a noun or a verb (Ambiguous POS)
“root” has multiple meanings (Ambiguous sense)
Syntactic ambiguity
“natural language processing” (Modification)
“A man saw a boy with a telescope .” (PP Attachment)
Anaphora resolution
“John persuaded Bill to buy a TV for himself .”
(himself = John or Bill?)
Presupposition
“He has quit smoking.” implies that he smoked before.
Humans rely on context to interpret (when possible).
This context may extend beyond a given document!
(Taken from ChengXiang Zhai, CS 397cxz – Fall 2003)
7. Shallow Linguistics
Progress on Useful Sub-Goals:
• English Lexicon
• Part-of-Speech Tagging
• Word Sense Disambiguation
• Phrase Detection / Parsing
8. WordNet
An extensive lexical network for the English language
• Contains over 138,838 words.
• Several graphs, one for each part-of-speech.
• Synsets (synonym sets), each defining a semantic sense.
• Relationship information (antonym, hyponym, meronym …)
• Downloadable for free (UNIX, Windows)
• Expanding to other languages (Global WordNet Association)
• Funded >$3 million, mainly government (translation interest)
• Founder George Miller, National Medal of Science, 1991.
watery
moist
parched
wet
dry
damp
anhydrous
arid
synonym
antonym
9. Part-of-Speech Tagging
This
Det
sentence
N
Training data (Annotated
text)
serves as an example of
V1
“This is a new sentence.”
P
Det
N
POS Tagger
P
annotated
V2
text…
N
This is a new
Det Aux Det Adj
sentence.
N
Pick the most1 likely ttag tsequence.
p ( w ,..., wk , 1 ,..., k )
p (t1 | w1 )... p(tk | wk ) p( w1 )... p( wk )
p ( w1 ,..., wk , t1 ,..., t k ) = k
Independent assignment
∏ p ( wi | ti ) p (ti | ti −1 )
Most common tag
=1
p (t1 | w1 )... p(tk | wk ) p(iw1 )... p ( wk )
= k
∏ p ( wi | ti ) p (ti | ti −1 )
Partial dependency
i =1
(HMM)
10. Word Sense Disambiguation
?
“The difficulties of computational linguistics are rooted in ambiguity.”
N
Aux V
P
N
Supervised Learning
Features:
• Neighboring POS tags (N Aux V P N)
• Neighboring words (linguistics are rooted in ambiguity)
• Stemmed form (root)
• Dictionary/Thesaurus entries of neighboring words
• High co-occurrence words (plant, tree, origin,…)
• Other senses of word within discourse
Algorithms:
• Rule-based Learning (e.g. IG guided)
• Statistical Learning (i.e. Naïve Bayes)
• Unsupervised Learning (i.e. Nearest Neighbor)
11. Parsing
Choose most likely parse tree…
Grammar
Lexicon
V → chasing
Aux→ is
N → dog
N → boy
N→ playground
Det→ the
Det→ a
P → on
Probability of this tree=0.000015
NP
Probabilistic CFG
S→ NP VP
NP → Det BNP
NP → BNP
NP→ NP PP
BNP→ N
VP → V
VP → Aux V NP
VP → VP PP
PP → P NP
S
Det
BNP
A
1.0
0.3
0.4
0.3
N
.
.
.
V
NP
is chasing
P
NP
on
a boy
Probability of this tree=0.000011
S
NP
0.01
Det
0.003
A
…
PP
the playground
1.0
…
VP
Aux
dog
…
…
VP
VP
BNP
N
dog
Aux
is
NP
V
chasing NP
a boy
PP
P
NP
on
the playground
12. Mining Text and Web Data
Text mining, natural language processing and
information extraction: An Introduction
Text information system and information
retrieval
Text categorization methods
Mining Web linkage structures
Summary
13. Text Databases and IR
Text databases (document databases)
Large collections of documents from various sources:
news articles, research papers, books, digital libraries,
e-mail messages, and Web pages, library database, etc.
Data stored is usually semi-structured
Traditional information retrieval techniques become
inadequate for the increasingly vast amounts of text
data
Information retrieval
A field developed in parallel with database systems
Information is organized into (a large number of)
documents
Information retrieval problem: locating relevant
documents based on user input, such as keywords or
example documents
14. Information Retrieval
Typical IR systems
Online library catalogs
Online document management systems
Information retrieval vs. database systems
Some DB problems are not present in IR, e.g., update,
transaction management, complex objects
Some IR problems are not addressed well in DBMS,
e.g., unstructured documents, approximate search
using keywords and relevance
15. Basic Measures for Text Retrieval
Relevant
Relevant &
Retrieved
Retrieved
All Documents
Precision: the percentage of retrieved documents that are in fact relevant
to the query (i.e., “correct” responses)
| {Relevant} ∩ {Retrieved } |
precision =
| {Retrieved } |
Recall: the percentage of documents that are relevant to the query and
were, in fact, retrieved
Recall
∩
= | {Relevant} {Retrieved } |
| {Relevant} |
16. Information Retrieval Techniques
Basic Concepts
A document can be described by a set of
representative keywords called index terms.
Different index terms have varying relevance when
used to describe document contents.
This effect is captured through the assignment of
numerical weights to each index term of a document.
(e.g.: frequency, tf-idf)
DBMS Analogy
Index Terms Attributes
Weights Attribute Values
17. Information Retrieval Techniques
Index Terms (Attribute) Selection:
Stop list
Word stem
Index terms weighting methods
Terms Documents Frequency Matrices
Information Retrieval Models:
Boolean Model
Vector Model
Probabilistic Model
18. Boolean Model
Consider that index terms are either present or
absent in a document
As a result, the index term weights are assumed to
be all binaries
A query is composed of index terms linked by three
connectives: not, and, and or
e.g.: car and repair, plane or airplane
The Boolean model predicts that each document is
either relevant or non-relevant based on the match of
a document to the query
19. Keyword-Based Retrieval
A document is represented by a string, which can be
identified by a set of keywords
Queries may use expressions of keywords
E.g., car and repair shop, tea or coffee, DBMS but not
Oracle
Queries and retrieval should consider synonyms, e.g.,
repair and maintenance
Major difficulties of the model
Synonymy: A keyword T does not appear anywhere in
the document, even though the document is closely
related to T, e.g., data mining
Polysemy: The same keyword may mean different
things in different contexts, e.g., mining
20. Similarity-Based Retrieval in Text
Data
Finds similar documents based on a set of common
keywords
Answer should be based on the degree of relevance
based on the nearness of the keywords, relative frequency
of the keywords, etc.
Basic techniques
Stop list
Set of words that are deemed “irrelevant”, even
though they may appear frequently
E.g., a, the, of, for, to, with , etc.
Stop lists may vary when document set varies
21. Similarity-Based Retrieval in Text
Data
Word stem
Several words are small syntactic variants of each
other since they share a common word stem
E.g., drug, drugs, drugged
A term frequency table
Each entry frequent_table(i, j) = # of occurrences of
the word ti in document di
Usually, the ratio instead of the absolute number of
occurrences is used
Similarity metrics: measure the closeness of a
document to a query (a set of keywords)
v1 ⋅ v2
Relative term occurrences
sim(v1 , v2 ) =
| v1 || v2 |
Cosine distance:
22. Indexing Techniques
Inverted index
Maintains two hash- or B+-tree indexed tables:
document_table: a set of document records <doc_id,
postings_list>
term_table: a set of term records, <term, postings_list>
Answer query: Find all docs associated with one or a set of terms
+ easy to implement
– do not handle well synonymy and polysemy, and posting lists could
be too long (storage could be very large)
Signature file
Associate a signature with each document
A signature is a representation of an ordered list of terms that
describe the document
Order is obtained by frequency analysis, stemming and stop lists
23. Types of Text Data
Mining
Keyword-based association analysis
Automatic document classification
Similarity detection
Cluster documents by a common author
Cluster documents containing information from a
common source
Link analysis: unusual correlation between entities
Sequence analysis: predicting a recurring event
Anomaly detection: find information that violates usual
patterns
Hypertext analysis
Patterns in anchors/links
Anchor text correlations with linked objects
24. Keyword-Based Association
Analysis
Motivation
Collect sets of keywords or terms that occur frequently together and
then find the association or correlation relationships among them
Association Analysis Process
Preprocess the text data by parsing, stemming, removing stop
words, etc.
Evoke association mining algorithms
Consider each document as a transaction
View a set of keywords in the document as a set of items in the
transaction
Term level association mining
No need for human effort in tagging documents
The number of meaningless results and the execution time is greatly
reduced
25. Text Classification
Motivation
Automatic classification for the large number of on-line text
documents (Web pages, e-mails, corporate intranets, etc.)
Classification Process
Data preprocessing
Definition of training set and test sets
Creation of the classification model using the selected
classification algorithm
Classification model validation
Classification of new/unknown text documents
Text document classification differs from the classification of
relational data
Document databases are not structured according to attributevalue pairs
27. Document Clustering
Motivation
Automatically group related documents based on their
contents
No predetermined training sets or taxonomies
Generate a taxonomy at runtime
Clustering Process
Data preprocessing: remove stop words, stem, feature
extraction, lexical analysis, etc.
Hierarchical clustering: compute similarities applying
clustering algorithms.
Model-Based clustering (Neural Network Approach):
clusters are represented by “exemplars”. (e.g.: SOM)
28. Text Categorization
Pre-given categories and labeled document
examples (Categories may form hierarchy)
Classify new documents
A standard classification (supervised learning )
problem
Sports
Categorization
System
Business
Education
Sports
Business
Education
…
…
Science
30. Categorization Methods
Manual: Typically rule-based
Does not scale up (labor-intensive, rule inconsistency)
May be appropriate for special data on a particular
domain
Automatic: Typically exploiting machine learning techniques
Vector space model based
Prototype-based (Rocchio)
K-nearest neighbor (KNN)
Decision-tree (learn rules)
Neural Networks (learn non-linear classifier)
Support Vector Machines (SVM)
Probabilistic or generative model based
Naïve Bayes classifier
31. Vector Space Model
Represent a doc by a term vector
Each term defines one dimension
N terms define a N-dimensional space
Element of vector corresponds to term weight
Term: basic concept, e.g., word or phrase
E.g., d = (x1,…,xN), xi is “importance” of term i
New document is assigned to the most likely category
based on vector similarity.
33. What VS Model Does Not
Specify
How to select terms to capture “basic concepts”
Word stopping
e.g. “a”, “the”, “always”, “along”
Word stemming
e.g. “computer”, “computing”, “computerize” =>
“compute”
Latent semantic indexing
How to assign weights
Not all words are equally important: Some are more
indicative than others
e.g. “algebra” vs. “science”
How to measure the similarity
34. How to Assign Weights
Two-fold heuristics based on frequency
TF (Term frequency)
More frequent within a document more relevant
to semantics
e.g., “query” vs. “commercial”
IDF (Inverse document frequency)
Less frequent among documents more
discriminative
e.g. “algebra” vs. “science”
35. TF Weighting
Weighting:
More frequent => more relevant to topic
e.g. “query” vs. “commercial”
Raw TF= f(t,d): how many times term t appears in
doc d
Normalization:
Document length varies => relative frequency preferred
e.g., Maximum frequency normalization
36. How to Measure Similarity?
Given two document
Similarity definition
dot product
normalized dot product (or cosine)
<number>
Throughout this course we have been discussing Data Mining over a variety of data types.
Two former types we covered were Structured Data (relational) and multimedia data.
Today and in the last class we have been discussing Data Mining over free text,
and our next section will cover hypertext, such as web pages.
Text mining is well motivated, due to the fact that much of the world’s data can be
found in free text form (newspaper articles, emails, literature, etc.). There is a
lot of information available to mine.
While mining free text has the same goals as data mining in general
(extracting useful knowledge/stats/trends), text mining must overcome
a major difficulty – there is no explicit structure.
Machines can reason will relational data well since schemas are explicitly available.
Free text, however, encodes all semantic information within natural language. Our
text mining algorithms, then, must make some sense out of this natural language
representation. Humans are great at doing this, but this has proved to be a problem for machines.
<number>
The previous text mining presentations “made sense” out of free text by
viewing text as a bag-of-tokens (words, n-grams). This is the same approach as IR.
Under that model we can already summarize, classify, cluster, and compute co-occurrence stats over free text.
These are quite useful for mining and managing large volumes of free text.
However, there is a potential to do much more. The BOT approach loses a LOT of information
contained in text, such as word order, sentence structure, and context. These are precisely the
features that humans use to interpret text.
Thus the natural question is can we do better?
<number>
NLP, or Computational Linguistics, is an entire field dedicated to the study
of automatically understanding free text. This field has been active since the 50’s.
General NLP attempts to understand document completely (at the level of a human reader).
There are several steps involved in NLP.
…Blah…
<number>
General NLP has proven to be too difficult. It is dubbed “AI-Complete”, meaning that such a
program would basically have to have near-human intelligence (i.e. solve AI).
The reason NLP in general is so difficult is that text is highly ambiguous. NL is meant for
human consumption and often contains ambiguities under the assumption that humans
will be able to develop context and interpret the intended meaning.
For instance [rewind], in this example the sentence could mean that either the dog, or the boy, or both
are on the playground. As a human we know that it is probably both, but that is due to our
knowledge that a dog is probably chasing close behind the boy, playgrounds are large,
they are probably playing, and a playground is a place to play. This background knowledge
probably is not contained in a document containing this sentence.
Despite such obstacles, computational linguists have made great progress on several subproblems.
We will now talk about four of these subproblems.
<number>
Several subgoals to NLP have been addressed to derive more info than just bag-of-tokens view.
English lexicon, POS Tagging, WSD, Parsing
Even with imperfect performance, these solutions already open the door for more intelligent text processing.
<number>
WordNet is an extensive lexical network for the human language.
Consists of a graph of synsets for each part of speech. Contains synonym and antonym relationships.
(hyponym=isa/subset, maple is a tree -> maple is a hyponym of tree)
(meronym=hasa, tree has a leaf -> leaf is a meronym of tree)
As will see, this is useful throughout NLP/ShallowLinguistics.
This encodes some of the lexicon that humans carry with them when interpreting text.
<number>
POS Tagging attempts to label each word with the appropriate part of speech.
Past approaches were rule-based (manual, then learned). Current trend, however, is toward statistical approaches (HMM).
This shift is common throughout NLP, due to the ability of statistical approaches to robustly handle noise and/or unexpected events.
Conceptually statistical approaches are more fitting due to the fact that uncertainty is sometimes unavoidable (ambiguity).
Current algorithms (Brill’s Tagger, CLAWS taggers) report accuracy in the 97% range.
<number>
WSD attempts to resolve ambiguity by labeling each word with a precise sense, as intended in the document.
This is typically performed after POS tagging, since POS tags are quite useful for WSD.
Current approaches address this as a supervised learning problem.
Features include neighboring words w/ POS tags, stemmed form of word, high co-occurrence words (with stem).
Quite a few supervised learning algorithms have been applied (rule-lists, bayesian, NN).
Performance depends heavily upon the particular text, but from what I’ve read 90%+ accuracy is common.
<number>
Parsing attempts to infer the precise grammatical relationships between different words in a given sentence.
For example, POS are grouped into phrases and phrases are combined into sentences.
Approaches include parsing with probabilistic CFG’s, “link dictionaries”, and tree adjoining techniques (super-tagging).
Current techniques can only parse at the sentence level, in some cases reporting accuracy in the 90% range.
Again, the performance heavily depends upon the grammatical correctness and the degree of ambiguity of the text.