Text Analytics for Semantic Computing


Published on

Text Analytics for Semantic Computing - the good, the bad and the ugly, tutorial presentation at ICSC 2008

Published in: Education, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • One of the primary goals of AI has been natural language understanding. Understanding language required not only lexical and grammatical information but also semantic pragmatic and general world knowledge. It’s a complex task and involves many levels of processing and a variety of subtasks. Typical components of a language understanding processing system Understanding language deals with (what is said) syntax and structure of language+ (what does the thing being said say/ask/inform of the world) understanding (semantics, pragmatics, discourse)
  • In the 1970’s AI systems were developed that demostrates interesting aspects of language understanding by developing nl understanding systems that used hand coded symbolic grammars and knowledge bases. Although there was some corpus based language learnng in the 1950’s post Shannon’s Information theory etc, Chomsky’s argument that learnability of language is more an innate property than learned was instrumental in redefining goals of linguistics in the 1950’s. Emphasis on symbolic grammars and representing innate linguistic knowledge (universal grammar)
  • Develpoing such systems however were very human intensive requiring intensive knowledge engineering, typically ran on toy examples and were rather brittle. Partially in response to these uissues there was a paradigm shift in nl understanding. Approaches moved from rationalist methods based on hand coded rules to systems that derived these rules through introspection or empirical or courpus based methods. Development ismore data driven and atleast partially automated thru the use of statistical or machine learning methods.
  • One of the primary goals of AI has been natural language understanding. Understanding language required not only lexical and grammatical information but also semantic pragmatic and general world knowledge. It’s a complex task and involves many levels of processing and a variety of subtasks. Typical components of a language understanding processing system Understanding language deals with (what is said) syntax and structure of language+ (what does the thing being said say/ask/inform of the world) understanding (semantics, pragmatics, discourse)
  • When we are analyzing text for semantic comp, we are doing one of two things - Finding more about wat we know. Often termed as the finding needle in a haystack paradigm, this search / browse method contrasts the other goal of text analysis.. Finding wat we do not know or discovering undisc knowledge
  • Assertional, simple, easier to parse and understand Biomedical literature, however, contains text that describes complex scientific investigations which do not always contain explicit factual assertions. Instead, there is often a series of arguments, opinions and experiments supported by evidence that collectively corroborate or refute a hypothesis that may not be explicitly stated in a simple sentence. Sentences tend to be rather long and convoluted. Furthermore domain specific terms, abbreviations, number ranges and symbols often make sentences hard for the human reader to parse, further complicating automated information extraction. These factors make the task of mining biomedical text substantially more complex than Wikipedia like text. Casual, goal is largely interactive as opposed to informative grammatical errors, misspellings, entity variations not uncommon
  • One of the primary goals of AI has been natural language understanding. Understanding language required not only lexical and grammatical information but also semantic pragmatic and general world knowledge. It’s a complex task and involves many levels of processing and a variety of subtasks. Typical components of a language understanding processing system Understanding language deals with (what is said) syntax and structure of language+ (what does the thing being said say/ask/inform of the world) understanding (semantics, pragmatics, discourse)
  • Using Text to Form Hypotheses about Disease For more than a decade, Don Swanson has eloquently argued why it is plausible to expect new information to be derivable from text collections: experts can only read a small subset of what is published in their fields and are often unaware of developments in related fields. Thus it should be possible to find useful linkages between information in related literatures, if the authors of those literatures rarely refer to one another's work. Swanson has shown how chains of causal implication within the medical literature can lead to hypotheses for causes of rare diseases, some of which have received supporting experimental evidence [Swanson1987,Swanson1991,Swanson and Smalheiser1994,Swanson and Smalheiser1997]. For example, when investigating causes of migraine headaches, he extracted various pieces of evidence from titles of articles in the biomedical literature. Some of these clues can be paraphrased as follows: stress is associated with migraines stress can lead to loss of magnesium calcium channel blockers prevent some migraines magnesium is a natural calcium channel blocker spreading cortical depression (SCD) is implicated in some migraines high leveles of magnesium inhibit SCD migraine patients have high platelet aggregability magnesium can suppress platelet aggregability These clues suggest that magnesium deficiency may play a role in some kinds of migraine headache; a hypothesis which did not exist in the literature at the time Swanson found these links. The hypothesis has to be tested via non-textual means, but the important point is that a new, potentially plausible medical hypothesis was derived from a combination of text fragments and the explorer's medical expertise. (According to swanson91, subsequent study found support for the magnesium-migraine hypothesis [Ramadan et al.1989].) This approach has been only partially automated. There is, of course, a potential for combinatorial explosion of potentially valid links. beeferman98 has developed a flexible interface and analysis tool for exploring certain kinds of chains of links among lexical relations within WordNet.2 However, sophisticated new algorithms are needed for helping in the pruning process, since a good pruning algorithm will want to take into account various kinds of semantic constraints. This may be an interesting area of investigation for computational linguists.
  • Beyond search Analytical operations over text to answer complex questions Requiring aggregation of information across a corpus Context specific Domain specific Application specific Text mining, also known as intelligent text analysis, text data mining or knowledge-discovery in text (KDT), refers generally to the process of extracting interesting and non-trivial information and knowledge from unstructured text. Text mining is a young interdisciplinary field which draws on information retrieval, data mining, machine learning, statistics and computational linguistics. As most information (over 80%) is stored as text, text mining is believed to have a high commercial potential value.
  • The first query is “Flu Epidemic.” In Table 1, we see that the first storyline contains information about flu (identified by terms like ‘vaccines’, ‘strains’), the second contains seasonal news (identified by terms like ‘deaths’, ‘reported’), the third is about bird flu (identified by terms like ‘avian’, ‘bird’), and the fourth is about Spanish flu epidemic from 1918 (identified by terms like ‘spanish’, ‘ 1918’).
  • http://turing.cs.washington.edu/papers/ijcai07.pdf
  • Select well-distributed terms from the collection Eliminate stopwords Retain only those terms with a distribution higher than a threshold (default: top 10%) Build a “backbone” Create paths from unambiguous terms only Bias the structure towards appropriate senses of words Get hypernym path if term: - has only one sense, or - matches a pre-selected WordNet domain Adding a new term increases a count at each node on its path by # of docs with the term.
  • We will cover some fundamentals that are a core part of most TM systems.
  • Tabulate Task, Tools/Methods, Resource, Frameworks Eg. NEI, Syntactic parse based, POS taggers, UIMA,Gate NEI, Lexicon based, lexicons available for dload, UIMA,Gate
  • Syntactic analysis involves determingng the grammatical structure of a sentence. One subtask is pos tagging or
  • Local word characteristics *s not always plural nouns (he works) (his works)
  • Headed by nouns and provide information about the noun in the sentence Headed by prepositions, contains noun phrases and express spatial, temporal and other attributes Oraganizes all elements on a sentence that syntactically depend on the verb
  • a grammar describes which of the possible sequences of symbols (strings) in a language constitute valid words or statements in that language, but it does not describe their semantics (i.e. what they mean).
  • Typical sources of knowledge Meanings of words Meanings of grammatical constructs Knowledge about structure of discourse Common sense knowledge about topic Knowledge about state of affairs in which discourse is occurring
  • Forming new words from old words (derivational) Suffixes (inflections)
  • Synonym test for words/lemmas - Propositional meaning When one word can be substituted for another without changing meaning of sentence Car and automobile substitutable but not identical in meaning
  • pathlen(c1,c2) = number of edges in the shortest path in the thesaurus graph between the sense nodes c1 and c2 simpath(c1,c2) = -log pathlen(c1,c2) wordsim(w1,w2) = max c1  senses(w1),c2  senses(w2) sim(c1,c2)
  • if "Adam Kluver Ltd" had already been recognised as an organisation by the sure-fire rule, in this second step any occurrences of "Kluver Ltd", "Adam Ltd" and "Adam Kluver" are also tagged as possible organizations. This assignment, however, is not definite since some of these words (such as "Adam") could refer to a different entity. This information goes to a pre-trained maximum entropy model (see Mikheev (1998) for more details on this aproach).
  • SRV - Two trends are evident in the recent evolution of the field of information extraction: a preference for simple, often corpus-driven techniques over linguistically sophisticated ones; and a broadening of the central problem definition to include many non-traditional text domains. This development calls for information extraction systems which are as retctrgetable and general as possible. Here, we describe SRV, a learning architecture for information extraction which is designed for maximum generality and flexibility.SRV can exploit domain-specific information,including linguistic syntax and lexical information, in the form of features provided to the system explicitly as input for training. This process is illustrated using a domain created from Reuters corporate acquisitions articles. Features are derived from two general-purpose NLP systems, Sleator and Temperly's link grammar parser and Wordnet. Experiments compare the learner's performance with and without such linguistic information. Surprisingly, in many cases, the system performs as well without this information as with it.
  • The label bias problem represents a simple finite-state model designed to distinguish between the two words rib and rob. In the first time step, r matches both transitions from the start state, so the probability mass gets distributed roughly equally among those two transitions. Next we observe i. Both states 1 and 4 have only one outgoing transition. State 1 has seen this observation often in training, state 4 has almost never seen this observation; but like state 1, state 4 has no choice but to pass all its mass to its single outgoing transition, since it is not generating the observation, only conditioning on it. Thus, states with a single outgoing transition effectively ignore their observations. The top path and the bottom path will be about equally likely, independently of the observation sequence. If one of the two words is slightly more common in the training set, the transitions out of the start state will slightly prefer its corresponding transition, and that word’s state sequence will always win.
  • i. Starting with a few seed entities, it is possible to induce high-precision context patterns by exploiting entity context redundancy. ii. New entity instances of the same category can be extracted from unlabeled data with the induced patterns to create high-precision extensions of the seed lists. iii. Features derived from token membership in the extended lists improve the accuracy of learned named-entity taggers.
  • Text Analytics for Semantic Computing

    2. 2. We have used material from several popular books, papers, course notes and presentations made by experts in this area. We have provided all references to the best of our knowledge. This list however, serves only as a pointer to work in this area and is by no means a comprehensive resource.
    3. 3. <ul><li>KNO.E.SIS </li></ul><ul><ul><li>knoesis.org </li></ul></ul><ul><li>Director: Amit Sheth </li></ul><ul><ul><li>knoesis.wright.edu/amit/ </li></ul></ul><ul><li>Graduate Students: </li></ul><ul><ul><li>Meena Nagarajan </li></ul></ul><ul><ul><ul><li>knoesis.wright.edu/students/meena/ </li></ul></ul></ul><ul><ul><ul><li>[email_address] </li></ul></ul></ul><ul><ul><li>Cartic Ramakrishnan </li></ul></ul><ul><ul><ul><li>knoesis.wright.edu/students/cartic/ </li></ul></ul></ul><ul><ul><ul><li>[email_address] </li></ul></ul></ul>
    4. 4. An Overview of Empirical Natural Language Processing, Eric Brill, Raymond J. Mooney Word Sequence Syntactic Parser Parse Tree Semantic Analyzer Literal Meaning Discourse Analyzer Meaning
    5. 5. <ul><li>Traditional (Rationalist) Natural Language Processing </li></ul><ul><ul><li>Main insight: Using rule-based representations of knowledge and grammar (hand-coded) for language study </li></ul></ul>KB Text NLP System Analysis
    6. 6. <ul><li>Empirical Natural Language Processing </li></ul><ul><ul><li>Main insight: Using distributional environment of a word as a tool for language study </li></ul></ul>KB Text NLP System Analysis Corpus Learning System
    7. 7. <ul><li>Two approaches not incompatible. Several systems use both. </li></ul><ul><li>Many empirical systems make use of manually created domain knowledge. </li></ul><ul><li>Many empirical systems use representations of rationalist methods replacing hand-coded rules with rules acquired from data. </li></ul>
    8. 8. <ul><li>Several algorithms, methods in each task, rationalist and empiricist approaches </li></ul><ul><li>What does a NL processing task typically entail? </li></ul><ul><li>How do systems, applications and tasks perform these tasks? </li></ul><ul><li>Syntax : POS Tagging, Parser </li></ul><ul><li>Semantics : Meaning of words, using context/domain knowledge to enhance tasks </li></ul>
    9. 9. <ul><li>Finding more about what we already know </li></ul><ul><ul><li>Ex. patterns that characterize known information </li></ul></ul><ul><ul><li>The search/browse OR ‘finding a needle in a haystack’ paradigm </li></ul></ul><ul><li>Discovering what we did not know </li></ul><ul><ul><li>Deriving new information from data </li></ul></ul><ul><ul><ul><li>Ex. Relationships between known entities previously unknown </li></ul></ul></ul><ul><ul><li>The ‘extracting ore from rock’ paradigm </li></ul></ul>
    10. 10. <ul><li>Information Extraction - those that operate directly on the text input </li></ul><ul><ul><ul><li>this includes entity, relationship and event detection  </li></ul></ul></ul><ul><li>Inferring new links and paths between key entities </li></ul><ul><ul><ul><li>sophisticated representations for information content, beyond the &quot;bag-of-words&quot; representations used by IR systems </li></ul></ul></ul><ul><li>Scenario detection techniques </li></ul><ul><ul><ul><li>discover patterns of relationships between entities that signify some larger event, e.g. money laundering activities.  </li></ul></ul></ul>
    11. 11. <ul><li>They all make use of knowledge of language (exploiting syntax and structure, different extents) </li></ul><ul><ul><li>Named entities begin with capital letters </li></ul></ul><ul><ul><li>Morphology and meanings of words </li></ul></ul><ul><li>They all use some fundamental text analysis operations </li></ul><ul><ul><li>Pre-processing, Parsing, chunking, part-of-speech, lemmatization, tokenization </li></ul></ul><ul><li>To some extent, they all deal with some language understanding challenges </li></ul><ul><ul><li>Ambiguity, co-reference resolution, entity variations etc. </li></ul></ul><ul><li>Use of a core subset of theoretical models and algorithms </li></ul><ul><ul><li>State machines, rule systems, probabilistic models, vector-space models, classifiers, EM etc. </li></ul></ul>
    12. 12. <ul><li>Analysis for both these goals have many similarities </li></ul><ul><li>Finding entities </li></ul><ul><ul><li>What are we interested in knowing more about? (the known) </li></ul></ul><ul><ul><li>Is what we found something of interest? (the unknown) </li></ul></ul><ul><li>Text is structured (to some extent) </li></ul><ul><ul><li>Syntax, Structure </li></ul></ul><ul><ul><li>Semantics, Pragmatics, Discourse </li></ul></ul><ul><li>Text is noisy </li></ul><ul><ul><li>Pre-processing is not an option in many cases </li></ul></ul><ul><ul><li>Variations not uncommon </li></ul></ul>
    13. 13. <ul><li>Wikipedia like text (GOOD) </li></ul><ul><ul><li>“ Thomas Edison invented the light bulb.” </li></ul></ul><ul><li>Scientific literature (BAD) </li></ul><ul><ul><li>“ This MEK dependency was observed in BRAF mutant cells regardless of tissue lineage, and correlated with both downregulation of cyclin D1 protein expression and the induction of G1 arrest.” </li></ul></ul><ul><li>Text from Social Media (UGLY) </li></ul><ul><ul><li>&quot;heylooo..ano u must hear it loadsss bu your propa faabbb!!&quot; </li></ul></ul>
    14. 14. <ul><li>Illustrate analysis of and challenges posed by these three text types throughout the tutorial </li></ul>
    15. 15. <ul><li>WHAT CAN TM DO FOR HARRY PORTER? </li></ul>A bag of words
    16. 16. Discovering connections hidden in text UNDISCOVERED PUBLIC KNOWLEDGE
    17. 17. <ul><li>Undiscovered Public Knowledge [Swanson] – as mentioned in [Hearst99] </li></ul><ul><li>Search no longer enough </li></ul><ul><ul><ul><li>Information overload – prohibitively large number of hits </li></ul></ul></ul><ul><ul><ul><li>UPK increases with increasing corpus size </li></ul></ul></ul><ul><li>Manual analysis very tedious </li></ul><ul><li>Examples [Hearst99] </li></ul><ul><ul><ul><li>Example 1 – Using Text to Form Hypotheses about Disease </li></ul></ul></ul><ul><ul><ul><li>Example 2 – Using Text to Uncover Social Impact </li></ul></ul></ul>
    18. 18. <ul><ul><li>Swanson’s discoveries </li></ul></ul><ul><ul><ul><li>Associations between Migraine and Magnesium [Hearst99] </li></ul></ul></ul><ul><ul><ul><ul><li>stress is associated with migraines </li></ul></ul></ul></ul><ul><ul><ul><ul><li>stress can lead to loss of magnesium </li></ul></ul></ul></ul><ul><ul><ul><ul><li>calcium channel blockers prevent some migraines </li></ul></ul></ul></ul><ul><ul><ul><ul><li>magnesium is a natural calcium channel blocker </li></ul></ul></ul></ul><ul><ul><ul><ul><li>spreading cortical depression (SCD) is implicated in some migraines </li></ul></ul></ul></ul><ul><ul><ul><ul><li>high levels of magnesium inhibit SCD </li></ul></ul></ul></ul><ul><ul><ul><ul><li>migraine patients have high platelet aggregability </li></ul></ul></ul></ul><ul><ul><ul><ul><li>magnesium can suppress platelet aggregability </li></ul></ul></ul></ul>
    19. 19. <ul><li>Mining popularity from Social Media </li></ul><ul><ul><li>Goal: Top X artists from MySpace artist comment pages </li></ul></ul><ul><ul><li>Traditional Top X lists got from radio plays, cd sales. An attempt at creating a list closer to listeners preferences </li></ul></ul><ul><li>Mining positive, negative affect / sentiment </li></ul><ul><ul><li>Slang, casual text necessitates transliteration </li></ul></ul><ul><ul><ul><li>‘ you are so bad’ == ‘you are good’ </li></ul></ul></ul>
    20. 20. <ul><li>Mining text to improve existing information access mechanisms </li></ul><ul><ul><li>Search [Storylines] </li></ul></ul><ul><ul><li>IR [QA systems] </li></ul></ul><ul><ul><li>Browsing [Flamenco] </li></ul></ul><ul><li>Mining text for </li></ul><ul><ul><li>Discovery & insight [Relationship Extraction] </li></ul></ul><ul><ul><li>Creation of new knowledge </li></ul></ul><ul><ul><ul><li>Ontology instance-base population </li></ul></ul></ul><ul><ul><ul><li>Ontology schema learning </li></ul></ul></ul>
    21. 21. <ul><li>Web search – aims at optimizing for top k (~10) hits </li></ul><ul><li>Beyond top 10 </li></ul><ul><ul><li>Pages expressing related latent views on topic </li></ul></ul><ul><ul><li>Possible reliable sources of additional information </li></ul></ul><ul><li>Storylines in search results [3] </li></ul>
    22. 23. <ul><li>TextRunner[4] </li></ul><ul><ul><li>A system that uses the result of dependency parses of sentences to train a Naïve Bayes classifier for Web-scale extraction of relationships </li></ul></ul><ul><ul><li>Does not require parsing for extraction – only required for training </li></ul></ul><ul><ul><li>Training on features – POS tag sequences, if object is proper noun, number of tokens to right or left etc. </li></ul></ul><ul><ul><li>This system is able to respond to queries like &quot;What did Thomas Edison invent?&quot; </li></ul></ul>
    23. 24. <ul><li>Castanet [1] </li></ul><ul><ul><li>Semi-automatically builds faceted hierarchical metadata structures from text </li></ul></ul><ul><ul><li>This is combined with Flamenco [2] to support faceted browsing of content </li></ul></ul>
    24. 25. Documents Select terms WordNet Build core tree Augment core tree Remove top level categories Compress Tree Divide into facets
    25. 26. Domains used to prune applicable senses in Wordnet (e.g. “dip”) frozen dessert sundae entity substance,matter nutriment dessert ice cream sundae frozen dessert entity substance,matter nutriment dessert sherbet,sorbet sherbet sundae sherbet substance,matter nutriment dessert sherbet,sorbet frozen dessert entity ice cream sundae
    26. 27. Biologically active substance Lipid Disease or Syndrome affects causes affects causes complicates Fish Oils Raynaud’s Disease ??????? instance_of instance_of UMLS Semantic Network MeSH PubMed 9284 documents 4733 documents 5 documents
    27. 28. [Hearst92] Finding class instances [Ramakrishnan et. al. 08] [Nguyen07] Finding attribute “like” relation instances
    28. 29. <ul><li>Automatic acquisition of </li></ul><ul><ul><li>Class Labels </li></ul></ul><ul><ul><li>Class hierarchies </li></ul></ul><ul><ul><li>Attributes </li></ul></ul><ul><ul><li>Relationships </li></ul></ul><ul><ul><li>Constraints </li></ul></ul><ul><ul><li>Rules </li></ul></ul>
    30. 32. <ul><li>[hearst 97] Abstract concepts are difficult to represent </li></ul><ul><li>“ Countless” combinations of subtle, abstract relationships among concepts </li></ul><ul><li>Many ways to represent similar concepts </li></ul><ul><ul><li>E.g. space ship, flying saucer, UFO </li></ul></ul><ul><li>Concepts are difficult to visualize </li></ul><ul><ul><li>High dimensionality </li></ul></ul><ul><ul><li>Tens or hundreds of thousands of features </li></ul></ul>
    31. 33. <ul><li>Ambiguity (sense) </li></ul><ul><ul><li>Keep that smile playin’ (Smile is a track) </li></ul></ul><ul><ul><li>Keep that smile on! </li></ul></ul><ul><li>Variations (spellings, synonyms, complex forms) </li></ul><ul><ul><li>Illeal Neoplasm vs. Adenomatous lesion of the Illeal wall </li></ul></ul><ul><li>Coreference resolution </li></ul><ul><ul><li>“ John wanted a copy of Netscape to run on his PC on the desk in his den; fortunately, his ISP included it in their startup package,” </li></ul></ul>
    32. 34. <ul><li>[hearst 97] Highly redundant data </li></ul><ul><ul><li>… most of the methods count on this property </li></ul></ul><ul><li>Just about any simple algorithm can get “good” results for simple tasks: </li></ul><ul><ul><li>Pull out “important” phrases </li></ul></ul><ul><ul><li>Find “meaningfully” related words </li></ul></ul><ul><ul><li>Create some sort of summary from documents </li></ul></ul>
    33. 35. <ul><li>Concerned with processing documents in natural language </li></ul><ul><ul><li>Computational Linguistics, Information Retrieval, Machine learning, Statistics, Information Theory, Data Mining etc. </li></ul></ul><ul><li>TM generally concerned with practical applications </li></ul><ul><ul><li>As opposed to lexical acquisition (for ex.)in CL </li></ul></ul>
    34. 36. <ul><li>Computing Resources </li></ul><ul><ul><li>Faster disks, CPUs, Networked Information </li></ul></ul><ul><li>Data Resources </li></ul><ul><ul><li>Large corpora, tree banks, lexical data for training and testing systems </li></ul></ul><ul><li>Tools for analysis </li></ul><ul><ul><li>NL analysis: taggers, parsers, noun-chunkers, tokenizers; Statistical Text Analysis: classifiers, nl model generators </li></ul></ul><ul><li>Emphasis on applications and evaluation </li></ul><ul><ul><li>Practical systems experimentally evaluated on real data </li></ul></ul>
    35. 37. <ul><li>TYPES, WHAT THEY DO AND DON’T </li></ul>
    36. 38. <ul><li>Co-occurrence based </li></ul><ul><li>Rule / knowledge based </li></ul><ul><li>Statistical / machine-learning based </li></ul><ul><li>Systems typically use a combination of two or more </li></ul>
    37. 39. <ul><li>Look for terms and posit relationships based on co-occurrence </li></ul><ul><ul><li>“ A word is known by the company it keeps” </li></ul></ul><ul><li>Non-trivial as they deal with problems of language – expression variability, ambiguity </li></ul><ul><li>Sometimes used as a simple baseline when evaluating more sophisticated systems </li></ul>
    38. 40. <ul><li>Exploit real-world knowledge </li></ul><ul><ul><li>About language </li></ul></ul><ul><ul><li>About terms in the domain </li></ul></ul><ul><ul><li>About relationship between terms in the domain </li></ul></ul><ul><ul><li>About variations of terms we know of etc. </li></ul></ul><ul><li>Spectrum of work from using hard-coded patterns (text or linguistic) for TM to using linguistic + semantic analysis for TM </li></ul><ul><li>Harder to develop, maintain rules; comprehensiveness also an issue </li></ul>
    39. 41. <ul><li>Using classifiers operating on text </li></ul><ul><ul><li>At pos level, n-grams, parse trees </li></ul></ul><ul><ul><li>Classifying documents, words, attributes/properties </li></ul></ul><ul><li>Typically requires hand-tagged/labeled data </li></ul><ul><ul><li>Supervised, semi-supervised approaches </li></ul></ul>http://compbio.uchsc.edu/Hunter_lab/Cohen/Hunter_Cohen_Molecular_Cell.pdf
    40. 42. <ul><li>Computational Linguistics - Syntax </li></ul><ul><ul><li>Parts of speech, morphology, phrase structure, parsing, chunking </li></ul></ul><ul><li>Semantics </li></ul><ul><ul><li>Lexical semantics, Syntax-driven semantic analysis, domain model-assisted semantic analysis (WordNet), </li></ul></ul><ul><li>Getting your hands dirty </li></ul><ul><ul><li>Text encoding, Tokenization, sentence splitting, morphology variants, lemmatization </li></ul></ul><ul><ul><li>Using parsers, understanding outputs </li></ul></ul><ul><ul><li>Tools, resources, frameworks </li></ul></ul>
    41. 43. <ul><li>Statistical NLP </li></ul><ul><ul><li>Mathematical foundations, some information theory </li></ul></ul><ul><ul><li>Words, Statistical Inference using n-grams, language models </li></ul></ul><ul><li>How are these used – some examples </li></ul><ul><ul><li>Collocations, Lexical Acquisition, Word sense disambiguation </li></ul></ul>
    42. 44. <ul><li>Several algorithms, variations for each </li></ul><ul><li>We wont go into each algorithm </li></ul><ul><ul><li>That’s a whole semester course </li></ul></ul><ul><li>We’ll define the problem, point out general approach, show examples </li></ul><ul><li>More importantly, we’ll show how results of these components are used by applications we know of today </li></ul>
    43. 45. <ul><li>POS Tags, Taggers, Ambiguities, Examples </li></ul>Word Sequence Syntactic Parser Parse Tree Semantic Analyzer Literal Meaning Discourse Analyzer Meaning
    44. 46. <ul><li>Word classes, syntactic/grammatical categories, parts-of-speech </li></ul><ul><li>Comprehensive lists used by taggers </li></ul><ul><ul><li>87 in the Brown Corpus tagset </li></ul></ul><ul><ul><li>45 in the Penn Treebank tagset </li></ul></ul><ul><ul><li>146 for the C7 tagset </li></ul></ul>POS Tag Description Example CC coordinating conjunction and CD cardinal number 1, third DT determiner the EX existential there there is FW foreign word d'hoevre IN preposition/subordinating conjunction in, of, like JJ adjective green JJR adjective, comparative greener JJS adjective, superlative greenest LS list marker 1) MD modal could, will NN noun, singular or mass table NNS noun plural tables NNP proper noun, singular John NNPS proper noun, plural Vikings PDT predeterminer both the boys POS possessive ending friend's PRP personal pronoun I, he, it PRP$ possessive pronoun my, his
    45. 47. <ul><li>Assigning a pos or syntactic class marker to a word in a sentence/corpus. </li></ul><ul><ul><li>Word classes, syntactic/grammatical categories </li></ul></ul><ul><li>Usually preceded by tokenization </li></ul><ul><ul><li>delimit sentence boundaries, tag punctuations and words. </li></ul></ul><ul><li>Publicly available tree banks, documents tagged for syntactic structure </li></ul><ul><li>Typical input and output of a tagger </li></ul><ul><ul><ul><li>Cancel that ticket. Cancel /VB that /DT ticket /NN ./. </li></ul></ul></ul>
    46. 48. <ul><li>Lexical ambiguity </li></ul><ul><ul><li>Words have multiple usages and parts-of-speech </li></ul></ul><ul><ul><ul><li>A duck in the pond ; Don’ t duck when I bowl </li></ul></ul></ul><ul><ul><ul><ul><li>Is duck a noun or a verb? </li></ul></ul></ul></ul><ul><ul><ul><li>Yes, we can ; Can of soup ; I can ned this idea </li></ul></ul></ul><ul><ul><ul><ul><li>Is can an auxiliary, a noun or a verb? </li></ul></ul></ul></ul><ul><li>Problem in tagging is resolving such ambiguities </li></ul>
    47. 49. <ul><li>Information about a word and its neighbors </li></ul><ul><ul><li>has implications on language models </li></ul></ul><ul><ul><ul><li>Possessive pronouns (mine, her, its) usually followed a noun </li></ul></ul></ul><ul><ul><li>Understand new words </li></ul></ul><ul><ul><ul><li>Toves did gyre and gimble. </li></ul></ul></ul><ul><ul><li>On IE </li></ul></ul><ul><ul><ul><li>Nouns as cues for named entities </li></ul></ul></ul><ul><ul><ul><li>Adjectives as cues for subjective expressions </li></ul></ul></ul>
    48. 50. <ul><li>Useful in understanding words </li></ul><ul><ul><li>We can guess what a new words means by looking at words around it. </li></ul></ul><ul><ul><li>Toves did gyre and gimble. Toves is something than can perform an action. (from hearst tagging) </li></ul></ul>
    49. 51. <ul><li>Rule-based </li></ul><ul><ul><li>Database of hand-written/learned rules to resolve ambiguity -EngCG </li></ul></ul><ul><li>Probability / Stochastic taggers </li></ul><ul><ul><li>Use a training corpus to compute probability of a word taking a tag in a specific context - HMM Tagger </li></ul></ul><ul><li>Hybrids – transformation-based </li></ul><ul><ul><li>The Brill tagger </li></ul></ul><ul><li>A comprehensive list of available taggers </li></ul><ul><ul><li>http://www-nlp.stanford.edu/links/statnlp.html#Taggers </li></ul></ul>
    50. 52. <ul><li>Not a complete representation </li></ul><ul><li>EngCG based on the Constraint Grammar Approach </li></ul><ul><li>Two step architecture </li></ul><ul><ul><li>Use a lexicon of words and likely pos tags to first tag words </li></ul></ul><ul><ul><li>Use a large list of hand-coded disambiguation rules that assign a single pos tag for each word </li></ul></ul>
    51. 53. <ul><li>Sample lexicon </li></ul><ul><ul><li>Word POS AdditionalPOS features </li></ul></ul><ul><ul><li>Slower ADJ COMPARITIVE </li></ul></ul><ul><ul><li>Show V PRESENT </li></ul></ul><ul><ul><li>Show N NOMINATIVE </li></ul></ul><ul><li>Sample rules </li></ul>
    52. 54. <ul><li>What is the best possible tag given this sequence of words? </li></ul><ul><ul><li>Takes context into account; global </li></ul></ul><ul><li>Example: HMM (hidden Markov models) </li></ul><ul><ul><li>A special case of Bayesian Inference </li></ul></ul><ul><ul><li>likely tag sequence is the one that maximizes the product of two terms: </li></ul></ul><ul><ul><ul><li>probability of sequence of tags and probability of each tag generating a word </li></ul></ul></ul>
    53. 55. <ul><li>Peter /NNP is /VBZ expected /VBN to /TO race /VB tomorrow /NN </li></ul><ul><li>to /TO race /??? t i = argmax j P(t j |t i-1 )P(w i |t j ) </li></ul><ul><ul><li>P(VB|TO) × P(race|VB) </li></ul></ul><ul><li>Based on the Brown Corpus: </li></ul><ul><ul><li>Probability that you will see this POS transition and that the word will take this POS </li></ul></ul><ul><ul><li>P(VB|TO) = .34 × P(race|VB) = .00003 = .00001 </li></ul></ul>
    54. 56. <ul><li>Be aware of possibility of ambiguities </li></ul><ul><li>Possible one has to normalize content before sending it to the tagger </li></ul><ul><li>Pre Post Transliteration </li></ul><ul><ul><ul><li>“ Rhi you were da coolest last eve” </li></ul></ul></ul><ul><ul><ul><li>Rhi/VB you/PRP were/VBD da/VBG coolest/JJ last/JJ eve/NN </li></ul></ul></ul><ul><ul><ul><li>“ Rhi you were the coolest last eve” </li></ul></ul></ul><ul><ul><ul><li>Rhi/VB you/PRP were/VBD the/DT coolest/JJ last/JJ eve/NN </li></ul></ul></ul>
    55. 57. <ul><li>Understanding Phrase Structures, Parsing, Chunking </li></ul>Word Sequence Syntactic Parser Parse Tree Semantic Analyzer Literal Meaning Discourse Analyzer Meaning
    56. 58. <ul><li>Words don’t just occur in some order </li></ul><ul><li>Words are organized in phrases </li></ul><ul><ul><li>groupings of words that clunk together </li></ul></ul><ul><li>Major phrase types </li></ul><ul><ul><li>Noun Phrases </li></ul></ul><ul><ul><li>Prepositional phrases </li></ul></ul><ul><ul><li>Verb phrases </li></ul></ul>
    57. 59. <ul><ul><li>Deriving the syntactic structure of a sentence based on a language model ( grammar ) </li></ul></ul><ul><li>Natural Language Syntax described by a context free grammar </li></ul><ul><ul><li>the Start-Symbol S ≡ sentence </li></ul></ul><ul><ul><li>Non-Terminals NT ≡ syntactic constituents </li></ul></ul><ul><ul><li>Terminals T ≡ lexical entries/ words </li></ul></ul><ul><ul><li>Productions P  NT  (NT  T) + ≡ grammar rules </li></ul></ul>http://www.cs.umanitoba.ca/~comp4190/2006/NLP-Parsing.ppt
    58. 60. <ul><li>S  NT, Part-of-Speech  NT, Constituents  NT, Words  T, Rules: </li></ul><ul><ul><li>S  NP VP statement </li></ul></ul><ul><ul><li>S  Aux NP VP question </li></ul></ul><ul><ul><li>S  VP command </li></ul></ul><ul><ul><li>NP  Det Nominal </li></ul></ul><ul><ul><li>NP  Proper-Noun </li></ul></ul><ul><ul><li>Nominal  Noun | Noun Nominal | Nominal PP </li></ul></ul><ul><ul><li>VP  Verb | Verb NP | Verb PP | Verb NP PP </li></ul></ul><ul><ul><li>PP  Prep NP </li></ul></ul><ul><ul><li>Det  that | this | a </li></ul></ul><ul><ul><li>Noun  book | flight | meal | money </li></ul></ul>
    59. 61. <ul><li>Bottom-up Parsing or data-driven </li></ul><ul><li>Top-down Parsing or goal-driven </li></ul><ul><ul><li> S </li></ul></ul><ul><ul><li>Aux NP VP </li></ul></ul><ul><ul><li> Det Nominal Verb NP </li></ul></ul><ul><ul><li>Noun Det Nominal </li></ul></ul><ul><ul><li> does this flight include a meal </li></ul></ul>
    60. 62. Natural Language Parsers, Peter Hellwig, Heidelberg Constituency Parse - Nested Phrasal Structures Dependency parse - Role Specific Structures
    61. 63. <ul><li>Tagging </li></ul><ul><ul><li>John/NNP bought/VBD a/DT book/NN ./. </li></ul></ul><ul><li>Constituency Parse </li></ul><ul><ul><ul><li>Nested phrasal structure </li></ul></ul></ul><ul><ul><ul><li>(ROOT (S (NP (NNP John)) (VP (VBD bought) (NP (DT a) (NN book))) (. .))) </li></ul></ul></ul><ul><li>Typed dependencies </li></ul><ul><ul><li>Role specific structure </li></ul></ul><ul><ul><ul><li>nsubj(bought-2, John-1) </li></ul></ul></ul><ul><ul><ul><li>det(book-4, a-3) </li></ul></ul></ul><ul><ul><ul><li>dobj(bought-2, book-4) </li></ul></ul></ul>
    62. 64. <ul><li>Grammar checking: sentences that cannot be parsed may have grammatical errors </li></ul><ul><li>Using results of Dependency parse </li></ul><ul><ul><li>Word sense disambiguation (dependencies as features or co-occurrence vectors) </li></ul></ul>
    63. 65. <ul><li>MINIPAR </li></ul><ul><ul><li>http://www.cs.ualberta.ca/~lindek/minipar.htm </li></ul></ul><ul><li>Link Grammar parser: http://www.link.cs.cmu.edu/link/ </li></ul><ul><li>Standard “CFG” parsers like the Stanford parser </li></ul><ul><ul><li>http://www-nlp.stanford.edu/software/lex-parser.shtml </li></ul></ul><ul><li>ENJU’s probabilistic HPSG grammar </li></ul><ul><ul><li>http://www-tsujii.is.s.u-tokyo.ac.jp/enju/ </li></ul></ul>
    64. 66. <ul><li>Some applications don’t need the complex output of a full parse </li></ul><ul><li>Chunking / Shallow Parse / Partial Parse </li></ul><ul><ul><li>Identifying and classifying flat, non-overlapping contiguous units in text </li></ul></ul><ul><ul><ul><li>Segmenting and tagging </li></ul></ul></ul><ul><li>Example of chunking a sentence </li></ul><ul><ul><ul><li>[ NP The morning flight] from [ NP Denver] [ VP has arrived] </li></ul></ul></ul><ul><li>Chunking algos mention </li></ul>
    65. 67. From Hearst 97
    66. 68. <ul><li>Entity recognition </li></ul><ul><ul><ul><li>people, locations, organizations </li></ul></ul></ul><ul><li>Studying linguistic patterns (Hearst 92) </li></ul><ul><ul><ul><li>gave NP </li></ul></ul></ul><ul><ul><ul><li>gave up NP in NP </li></ul></ul></ul><ul><ul><ul><li>gave NP NP </li></ul></ul></ul><ul><ul><ul><li>gave NP to NP </li></ul></ul></ul>
    67. 69. <ul><li>Stanford and Enju parser demos; analyzing results </li></ul><ul><ul><li>http://www-tsujii.is.s.u-tokyo.ac.jp/enju/demo.html </li></ul></ul><ul><ul><li>http://nlp.stanford.edu:8080/parser/ </li></ul></ul><ul><li>If you want to know how to run it stand alone </li></ul><ul><ul><li>Talk to one of us or see their very helpful help pages </li></ul></ul>
    68. 70. <ul><li>COLORLESS GREEN IDEAS SLEEP FURIOUSLY </li></ul>Word Sequence Syntactic Parser Parse Tree Semantic Analyzer Literal Meaning Discourse Analyzer Meaning
    69. 71. <ul><li>When raw linguistic inputs nor any structures derived from them will facilitate required semantic processing </li></ul><ul><li>When we need to link linguistic information to the non-linguistic real-world knowledge </li></ul><ul><li>Typical sources of knowledge </li></ul><ul><ul><li>Meaning of words, grammatical constructs, discourse, topic.. </li></ul></ul>
    70. 72. <ul><li>Typical sources of knowledge </li></ul><ul><ul><li>Meanings of words </li></ul></ul><ul><ul><li>Meanings of grammatical constructs </li></ul></ul><ul><ul><li>Knowledge about structure of discourse </li></ul></ul><ul><ul><li>Common sense knowledge about topic </li></ul></ul><ul><ul><li>Knowledge about state of affairs in which discourse is occurring </li></ul></ul>
    71. 73. <ul><li>Lexical Semantics </li></ul><ul><ul><li>The meanings of individual words </li></ul></ul><ul><li>Formal Semantics (Compositional Semantics or Sentential Semantics) </li></ul><ul><ul><li>How those meanings combine to make meanings for individual sentences or utterances </li></ul></ul><ul><li>Discourse or Pragmatics </li></ul><ul><ul><li>How those meanings combine with each other and with other facts about various kinds of context to make meanings for a text or discourse </li></ul></ul>http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    72. 74. <ul><li>Lexeme: set of forms taken by a single word </li></ul><ul><ul><li>run, runs, ran and running forms of the same lexeme RUN </li></ul></ul><ul><li>Lemma: a particular form of a lexeme that is chosen to represent a canonical form </li></ul><ul><ul><ul><li>Carpet for carpets; Sing for sing, sang, sung </li></ul></ul></ul><ul><li>Lemmatization: Meaning of a word approximated by meaning of its lemma </li></ul><ul><ul><li>Mapping a morphological variant to its root </li></ul></ul><ul><ul><ul><li>Derivational and Inflectional Morphology </li></ul></ul></ul>
    73. 75. <ul><li>Word sense: Meaning of a word (lemma) </li></ul><ul><ul><li>Varies with context </li></ul></ul><ul><li>Significance </li></ul><ul><ul><li>Lexical ambiguity </li></ul></ul><ul><ul><ul><li>consequences on tasks like parsing and tagging </li></ul></ul></ul><ul><ul><ul><li>implications on results of Machine translation, Text classification etc. </li></ul></ul></ul><ul><li>Word Sense Disambiguation </li></ul><ul><ul><li>Selecting the correct sense for a word </li></ul></ul>
    74. 76. <ul><li>The study of the way words are built up from smaller meaning units. </li></ul><ul><li>Derivational and Inflectional morphology </li></ul><ul><ul><li>Forming new words from old words (derivational) </li></ul></ul><ul><ul><li>Suffixes (inflections) </li></ul></ul><ul><li>Knowing root helps us understand new words </li></ul>
    75. 77. <ul><li>Porter Stemming Algorithm </li></ul><ul><ul><li>http://tartarus.org/~martin/PorterStemmer/ </li></ul></ul><ul><li>Catvar </li></ul><ul><ul><li>http://clipdemos.umiacs.umd.edu/catvar/ </li></ul></ul><ul><li>Lingsoft </li></ul><ul><ul><li>http://www2.lingsoft.fi/cgi-bin/engtwol?word=cmputers </li></ul></ul><ul><li>Wordnet </li></ul><ul><ul><li>http://www.shiffman.net/teaching/a2z/wordnet/ </li></ul></ul>
    76. 78. <ul><li>Homonymy </li></ul><ul><li>Polysemy </li></ul><ul><li>Synonymy </li></ul><ul><li>Antonymy </li></ul><ul><li>Hypernomy </li></ul><ul><li>Hyponomy </li></ul><ul><li>Meronomy </li></ul><ul><li>Why do we care? </li></ul>http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    77. 79. <ul><li>Homonymy: share a form, relatively unrelated senses </li></ul><ul><ul><li>Bank (financial institution, a sloping mound) </li></ul></ul><ul><li>Polysemy: semantically related </li></ul><ul><ul><li>Bank as a financial institution, as a blood bank </li></ul></ul><ul><ul><li>Verbs tend more to polysemy </li></ul></ul>
    78. 80. <ul><li>Different words/lemmas that have the same sense </li></ul><ul><ul><li>Couch/chair </li></ul></ul><ul><li>One sense more specific than the other (hyponymy) </li></ul><ul><ul><li>Car is a hyponym of vehicle </li></ul></ul><ul><li>One sense more general than the other (hypernymy) </li></ul><ul><ul><li>Vehicle is a hypernym of car </li></ul></ul>
    79. 81. <ul><li>Meronymy </li></ul><ul><ul><li>Engine part of car; engine meronym of car </li></ul></ul><ul><li>Holonymy </li></ul><ul><ul><li>Car is a holonym of engine </li></ul></ul>
    80. 82. <ul><li>Semantic fields </li></ul><ul><ul><li>Cohesive chunks of knowledge </li></ul></ul><ul><ul><li>Air travel: </li></ul></ul><ul><ul><ul><li>Flight, travel, reservation, ticket, departure… </li></ul></ul></ul>
    81. 83. <ul><li>Models these sense relations </li></ul><ul><li>A hierarchically organized lexical database </li></ul><ul><li>On-line thesaurus + aspects of a dictionary </li></ul><ul><ul><ul><li>Versions for other languages are under development </li></ul></ul></ul>http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    82. 84. http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    83. 85. <ul><li>Verbs and Nouns in separate hierarchies </li></ul>http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    84. 86. <ul><li>The set of near-synonyms for a WordNet sense is called a synset ( synonym set ) </li></ul><ul><ul><li>Their version of a sense or a concept </li></ul></ul><ul><li>Duck as a verb to mean </li></ul><ul><ul><ul><li>to move (the head or body) quickly downwards or away </li></ul></ul></ul><ul><ul><ul><li>dip, douse, hedge, fudge, evade, put off, circumvent, parry, elude, skirt, dodge, duck, sidestep </li></ul></ul></ul>
    85. 87. <ul><li>IR and QnA </li></ul><ul><ul><li>Indexing using similar (synonymous) words/query or specific to general words (hyponymy / hypernymy) improves text retrieval </li></ul></ul><ul><li>Machine translation, QnA </li></ul><ul><ul><li>Need to know if two words are similar to know if we can substitute one for another </li></ul></ul>
    86. 88. <ul><li>Most well developed </li></ul><ul><ul><li>Synonymy or similarity </li></ul></ul><ul><li>Synonymy - a binary relationship between words, rather their senses </li></ul><ul><li>Approaches </li></ul><ul><ul><li>Thesaurus based : measuring word/sense similarity in a thesaurus </li></ul></ul><ul><ul><li>Distributional methods: finding other words with similar distributions in a corpus </li></ul></ul>
    87. 89. <ul><li>Thesaurus based </li></ul><ul><ul><li>Path based similarity – two words are similar if they are similar in the thesaurus hierarchy </li></ul></ul>http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    88. 90. <ul><li>We don’t have a thesaurus for every language. Even if we do, many words are missing </li></ul><ul><ul><li>Wordnet: Strong for nouns, but lacking for adjectives and even verbs </li></ul></ul><ul><ul><li>Expensive to build </li></ul></ul><ul><li>They rely on hyponym info for similarity </li></ul><ul><ul><li>car hyponym of vehicle </li></ul></ul><ul><li>Alternative - Distributional methods for word similarity </li></ul>http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    89. 91. <ul><li>Firth (1957): “You shall know a word by the company it keeps!” </li></ul><ul><li>Similar words appear in similar contexts - Nida example noted by Lin: </li></ul><ul><ul><ul><li>A bottle of tezgüino is on the table </li></ul></ul></ul><ul><ul><ul><li>Everybody likes tezgüino </li></ul></ul></ul><ul><ul><ul><li>Tezgüino makes you drunk </li></ul></ul></ul><ul><ul><ul><li>We make tezgüino out of corn. </li></ul></ul></ul>Partial material from http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    90. 92. http://www.stanford.edu/class/cs224u/224u.07.lec2.ppt
    91. 93. <ul><li>So you want to build your own text miner! </li></ul>
    92. 94. <ul><li>Infrastructure intensive </li></ul><ul><li>Luckily, plenty of open source tools, frameworks, resources.. </li></ul><ul><ul><li>http://www-nlp.stanford.edu/links/statnlp.html </li></ul></ul><ul><ul><li>http://www.cedar.buffalo.edu/~rohini/CSE718/References2.html </li></ul></ul>
    93. 95. <ul><li>Mining opinions from casual text </li></ul><ul><li>Data – user comments on artist pages from MySpace </li></ul><ul><ul><li>“ Your musics the shit,…lovve your video you are so bad” </li></ul></ul><ul><ul><li>“ Your music is wicked!!!!” </li></ul></ul><ul><li>Goal </li></ul><ul><ul><li>Popularity lists generated from listener’s comments to complement radio plays/cd sales lists </li></ul></ul>
    94. 96. <ul><ul><li>“ Your musics the shit,…lovve your video you are so bad” </li></ul></ul><ul><li>Pre-processing </li></ul><ul><ul><ul><li>strip html, normalizing text from different sources.. </li></ul></ul></ul><ul><li>Tokenization </li></ul><ul><ul><ul><li>Splitting text into tokens : word tokens, number tokens, domain specific requirements </li></ul></ul></ul><ul><li>Sentence splitting </li></ul><ul><ul><ul><li>! . ? … ; harder in casual text </li></ul></ul></ul><ul><li>Normalizing words </li></ul><ul><ul><ul><li>Stop word removal, lemmatization, stemming, transliterations (da == the) </li></ul></ul></ul>
    95. 97. <ul><li>‘ The smile is so wicked!!’ </li></ul><ul><li>Syntax : Marking sentiment expression from syntax or a dictionary </li></ul><ul><ul><ul><li>The/DT smile/NN is/VBZ so/RB wicked/JJ !/. !/. </li></ul></ul></ul><ul><li>Semantics : Surrounding context </li></ul><ul><ul><ul><li>On Lily Allen’s MySpace page. Cues for Co-ref resolution </li></ul></ul></ul><ul><ul><ul><li>Smile is a track by Lilly Allen. Ambiguity </li></ul></ul></ul><ul><li>Background knowledge / resources </li></ul><ul><ul><ul><li>Using urbandictionary.com for semantic orientation of ‘wicked’ </li></ul></ul></ul>
    96. 98. <ul><li>GATE - General Architecture of Text Engineering, since 1995 at University of Sheffield, UK </li></ul><ul><li>UIMA - Unstructured Information Management Architecture, IBM </li></ul><ul><ul><li>Document processing tools, Components syntactic tools, nlp tools, integrating framework </li></ul></ul>
    97. 99. <ul><li>The GATE (General Architecture for Text Engineering) System: </li></ul><ul><li>http://gate.ac.uk </li></ul><ul><li>http://sourceforge.net/projects/gate </li></ul><ul><li>User’s Guide: http://gate.ac.uk/sale/tao/ </li></ul><ul><li>IBM’s UIMA (Unstructured Information Management </li></ul><ul><li>Architecture): </li></ul><ul><li>http://www.research.ibm.com/UIMA/ </li></ul><ul><li>http://sourceforge.net/projects/uima-framework/ </li></ul><ul><li>Other Resources </li></ul><ul><li>WordNet: http://wordnet.princeton.edu/ </li></ul><ul><li>MuNPEx: http://www.ipd.uka.de/~durm/tm/munpex/ </li></ul>
    100. 102. This MEK dependency was observed in BRAF mutant cells regardless of tissue lineage, and correlated with both downregulation of cyclin D1 protein expression and the induction of G1 arrest. *MEK dependency ISA Dependency_on_an_Organic_chemical *BRAF mutant cells ISA Cell_type *downregulation of cyclin D1 protein expression ISA Biological_process *tissue lineage ISA Biological_concept *induction of G1 arrest ISA Biological_process Information Extraction = segmentation+classification+association+mining Text mining = entity identification+named relationship extraction+discovering association chains…. Segmentation Classification Named Relationship Extraction MEK dependency observed in BRAF mutant cells downregulation of cyclin D1 protein expression correlated with induction of G1 arrest correlated with
    101. 103. MEK dependency observed in BRAF mutant cells downregulation of cyclin D1 protein expression correlated with induction of G1 arrest correlated with
    102. 104. <ul><li>The task of classifying token sequences in text into one or more predefined classes </li></ul><ul><li>Approaches </li></ul><ul><ul><li>Look up a list </li></ul></ul><ul><ul><ul><li>Sliding window </li></ul></ul></ul><ul><ul><li>Use rules </li></ul></ul><ul><ul><li>Machine learning </li></ul></ul><ul><li>Compound entities </li></ul><ul><li>Applied to </li></ul><ul><ul><li>Wikipedia like text </li></ul></ul><ul><ul><li>Biomedical text </li></ul></ul>
    103. 105. <ul><li>The simplest approach </li></ul><ul><ul><li>Proper nouns make up majority of named entities </li></ul></ul><ul><ul><li>Look up a gazetteer </li></ul></ul><ul><ul><ul><li>CIA fact book for organizations, country names etc. </li></ul></ul></ul><ul><ul><li>Poor recall </li></ul></ul><ul><ul><ul><li>coverage problems </li></ul></ul></ul>
    104. 106. <ul><li>Rule based [Mikheev et. Al 1999] </li></ul>Frequency Based &quot;China International Trust and Investment Corp” &quot;Suspended Ceiling Contractors Ltd” &quot;Hughes“ when &quot;Hughes Communications Ltd.“ is already marked as an organization <ul><li>Scalability issues: </li></ul><ul><ul><li>Expensive to create manually </li></ul></ul><ul><ul><li>Leverages domain specific information – domain specific </li></ul></ul><ul><ul><li>Tend to be corpus-specific – due to manual process </li></ul></ul>
    105. 107. <ul><li>Machine learning approaches </li></ul><ul><ul><li>Ability to generalize better than rules </li></ul></ul><ul><ul><li>Can capture complex patterns </li></ul></ul><ul><ul><li>Requires training data </li></ul></ul><ul><ul><ul><li>Often the bottleneck </li></ul></ul></ul><ul><li>Techniques [list taken from Agichtein2007 ] </li></ul><ul><ul><li>Naive Bayes </li></ul></ul><ul><ul><li>SRV [Freitag 1998], Inductive Logic Programming </li></ul></ul><ul><ul><li>Rapier [Califf and Mooney 1997] </li></ul></ul><ul><ul><li>Hidden Markov Models [Leek 1997] </li></ul></ul><ul><ul><li>Maximum Entropy Markov Models [McCallum et al. 2000] </li></ul></ul><ul><ul><li>Conditional Random Fields [Lafferty et al. 2001] </li></ul></ul>
    106. 108. <ul><li>Orthographical Features </li></ul><ul><ul><li>CD28 a protein </li></ul></ul><ul><li>Context Features </li></ul><ul><ul><li>Window of words </li></ul></ul><ul><ul><ul><li>Fixed </li></ul></ul></ul><ul><ul><ul><li>Variable </li></ul></ul></ul><ul><li>Part-of-speech features </li></ul><ul><ul><li>Current word </li></ul></ul><ul><ul><li>Adjacent words – within fixed window </li></ul></ul><ul><li>Word shape features </li></ul><ul><ul><li>Kappa-B replaced with Aaaaa-A </li></ul></ul><ul><li>Dictionary features </li></ul><ul><ul><li>Inexact matches </li></ul></ul><ul><li>Prefixes and Suffixes </li></ul><ul><ul><li>“ ~ase” = protein </li></ul></ul>
    107. 109. <ul><ul><li>HMMs </li></ul></ul><ul><ul><ul><li>a powerful tool for representing sequential data </li></ul></ul></ul><ul><ul><ul><li>are probabilistic finite state models with parameters for state-transition probabilities and state-specific observation probabilities </li></ul></ul></ul><ul><ul><ul><li>the observation probabilities are typically represented as a multinomial distribution over a discrete, finite vocabulary of words </li></ul></ul></ul><ul><ul><ul><li>Training is used to learn parameters that maximize the probability of the observation sequences in the training data </li></ul></ul></ul><ul><ul><li>Generative </li></ul></ul><ul><ul><ul><li>Find parameters to maximize P(X,Y) </li></ul></ul></ul><ul><ul><ul><li>When labeling X i future observations are taken into account (forward-backward) </li></ul></ul></ul><ul><ul><li>Problems </li></ul></ul><ul><ul><ul><li>Feature overlap in NER </li></ul></ul></ul><ul><ul><ul><ul><li>E.g. to extract previously unseen company names from a newswire article </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>the identity of a word alone is not very predictive </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>knowing that the word is capitalized, that is a noun, that it is used in an appositive, and that it appears near the top of the article would all be quite predictive </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><li>Would like the observations to be parameterized with these overlapping features </li></ul></ul></ul></ul><ul><ul><ul><li>Feature independence assumption </li></ul></ul></ul>Several features about same word can affect parameters
    108. 110. <ul><li>MEMMs [McCallum et. al, 2000] </li></ul><ul><ul><li>Discriminative </li></ul></ul><ul><ul><ul><li>Find parameters to maximize P(Y|X) </li></ul></ul></ul><ul><ul><li>No longer assume that features are independent </li></ul></ul><ul><ul><ul><li>f<Is-capitalized,Company>(“Apple”, Company) = 1. </li></ul></ul></ul><ul><ul><li>Do not take future observations into account (no forward-backward) </li></ul></ul><ul><ul><li>Problems </li></ul></ul><ul><ul><ul><li>Label bias problem </li></ul></ul></ul>
    109. 111. <ul><li>CRFs [Lafferty et. al, 2001] </li></ul><ul><ul><li>Discriminative </li></ul></ul><ul><ul><li>Doesn’t assume that features are independent </li></ul></ul><ul><ul><li>When labeling Y i future observations are taken into account </li></ul></ul><ul><ul><li>Global optimization – label bias prevented </li></ul></ul><ul><li>The best of both worlds! </li></ul>
    110. 112. <ul><li>Example </li></ul><ul><ul><li>[ORG U.S. ] general [PER David Petraeus ] heads for [LOC Baghdad ] . </li></ul></ul><ul><ul><li>Token POS Chunk Tag </li></ul></ul><ul><ul><li>--------------------------------------------------------- </li></ul></ul><ul><ul><li>U.S. NNP I-NP I-ORG </li></ul></ul><ul><ul><li>general NN I-NP O </li></ul></ul><ul><ul><li>David NNP I-NP B-PER </li></ul></ul><ul><ul><li>Petraeus NNP I-NP I-PER </li></ul></ul><ul><ul><li>heads VBZ I-VP O </li></ul></ul><ul><ul><li>for IN I-PP O </li></ul></ul><ul><ul><li>Baghdad NNP I-NP I-LOC </li></ul></ul><ul><ul><li>. . O O </li></ul></ul><ul><ul><li>CONLL format – Mallet </li></ul></ul><ul><ul><li>Major bottleneck is training data </li></ul></ul>
    111. 113. <ul><li>Context Induction approach [Talukdar2006] </li></ul><ul><ul><ul><li>Starting with a few seed entities, it is possible to induce high-precision context patterns by exploiting entity context redundancy. </li></ul></ul></ul><ul><ul><ul><li>New entity instances of the same category can be extracted from unlabeled data with the induced patterns to create high-precision extensions of the seed lists. </li></ul></ul></ul><ul><ul><ul><li>Features derived from token membership in the extended lists improve the accuracy of learned named-entity taggers. </li></ul></ul></ul>Pruned Extraction patterns Feature generation For CRF
    112. 114. <ul><li>Machine Learning </li></ul><ul><ul><li>Best performance </li></ul></ul><ul><li>Problem </li></ul><ul><ul><li>Training data bottleneck </li></ul></ul><ul><li>Pattern induction </li></ul><ul><ul><li>Reduce training data creation time </li></ul></ul>
    113. 115. <ul><li>Knowledge Engineering approach </li></ul><ul><ul><li>Manually crafted rules </li></ul></ul><ul><ul><ul><ul><li>Over lexical items <person> works for <organization> </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Over syntactic structures – parse trees </li></ul></ul></ul></ul><ul><ul><li>GATE </li></ul></ul><ul><li>Machine learning approaches </li></ul><ul><ul><li>Supervised </li></ul></ul><ul><ul><li>Semi-supervised </li></ul></ul><ul><ul><li>Unsupervised </li></ul></ul>
    114. 116. <ul><li>Supervised </li></ul><ul><ul><ul><li>BioText – extraction of relationships between diseases and their treatments [Rosario et. al 2004] </li></ul></ul></ul><ul><ul><ul><li>Rule-based supervised approach [Rinaldi et. al 2004] </li></ul></ul></ul><ul><ul><ul><ul><li>Semantics of specific relationship encoded as rules </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Identify a set of relations along with their morphological variants (bind, regulate, signal etc.) </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>subj(bind,X,_,_),pobj(bind,Y,to,_) prep(Y,to,_,_) => bind(X,Y). </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><li>Axiom formulation was however a manual process involving a domain expert. </li></ul></ul></ul></ul>
    115. 117. <ul><li>Hand-coded domain specific rules that encode patterns used to extract </li></ul><ul><ul><ul><li>Molecular pathways [Freidman et. al. 2001] </li></ul></ul></ul><ul><ul><ul><li>Protein interaction [Saric et. al. 2006] </li></ul></ul></ul><ul><li>All of the above in the biomedical domain </li></ul><ul><ul><li>Notice – specificity of relationship types </li></ul></ul><ul><ul><li>Amount of effort required </li></ul></ul><ul><ul><li>Also notice types of entities involved in the relationships </li></ul></ul>
    117. 119. <ul><li>Semantic Role Labeling </li></ul><ul><ul><li>Features </li></ul></ul><ul><li>Detailed tutorial on SRL is available </li></ul><ul><ul><li>By Yih & Toutanova here </li></ul></ul>
    118. 120. <ul><li>Other approaches </li></ul><ul><ul><li>Discovering concept-specific relations </li></ul></ul><ul><ul><ul><ul><li>Dmitry Davidov, et. al 2007, </li></ul></ul></ul></ul><ul><ul><li>preemptive IE approach </li></ul></ul><ul><ul><ul><ul><li>Rosenfeld & Feldman 2007 </li></ul></ul></ul></ul><ul><ul><li>Open Information Extraction </li></ul></ul><ul><ul><ul><li>Banko et. al 2007 </li></ul></ul></ul><ul><ul><ul><li>Self supervised approach </li></ul></ul></ul><ul><ul><ul><li>Uses dependency parses to train extractors </li></ul></ul></ul><ul><ul><li>On-demand information extraction </li></ul></ul><ul><ul><ul><li>Sekine 2006 </li></ul></ul></ul><ul><ul><ul><ul><li>IR driven </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Patterns discovery </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Paraphrase </li></ul></ul></ul></ul>
    119. 121. <ul><li>Rule and Heuristic based method </li></ul><ul><ul><li>YAGO Suchanek et. al, 2007 </li></ul></ul><ul><ul><li>Pattern-based approach </li></ul></ul><ul><ul><li>Uses WordNet </li></ul></ul><ul><li>Subtree mining over dependency parse trees </li></ul><ul><ul><li>Nguyen et. al, 2007 </li></ul></ul>
    120. 122. <ul><li>Entities (MeSH terms) in sentences occur in modified forms </li></ul><ul><ul><li>“ adenomatous ” modifies “ hyperplasia ” </li></ul></ul><ul><ul><li>“ An excessive endogenous or exogenous stimulation ” modifies “ estrogen ” </li></ul></ul><ul><li>Entities can also occur as composites of 2 or more other entities </li></ul><ul><ul><li>“ adenomatous hyperplasia ” and “ endometrium ” occur as “ adenomatous hyperplasia of the endometrium” </li></ul></ul>
    121. 123. Relationship head Subject head Object head Object head <ul><li>Small set of rules over dependency types dealing with </li></ul><ul><ul><li>modifiers ( amod, nn ) etc. subjects, objects ( nsubj, nsubjpass ) etc. </li></ul></ul><ul><li>Since dependency types are arranged in a hierarchy </li></ul><ul><ul><li>We use this hierarchy to generalize the more specific rules </li></ul></ul><ul><ul><li>There are only 4 rules in our current implementation </li></ul></ul><ul><li>Carroll, J., G. Minnen and E. Briscoe (1999) `Corpus annotation for parser evaluation'. In Proceedings of the EACL-99 Post-Conference Workshop on Linguistically Interpreted Corpora, Bergen, Norway. 35-41. Also in Proceedings of the ATALA Workshop on Corpus Annotés pour la Syntaxe - Treebanks, Paris, France. 13-20. </li></ul>
    122. 124. Modifiers Modified entities Composite Entities
    123. 125. <ul><li>Manual Evaluation </li></ul><ul><ul><li>Test if the RDF conveys same “meaning” as the sentence </li></ul></ul><ul><ul><li>Juxtapose the triple with the sentence </li></ul></ul><ul><ul><li>Allow user to assess correctness/incorrectness of the subject, object and triple </li></ul></ul>
    124. 128. <ul><li>Discovering informative subgraphs (Harry Potter) </li></ul><ul><ul><li>Given a pair of end-points (entities) </li></ul></ul><ul><ul><li>Produce a subgraph with relationships connecting them such that </li></ul></ul><ul><ul><ul><li>The subgraph is small enough to be visualized </li></ul></ul></ul><ul><ul><ul><li>And contains relevant “interesting” connections </li></ul></ul></ul><ul><li>We defined an interestingness measure based on the ontology schema </li></ul><ul><ul><li>In future biomedical domain the scientist will control this with the help of a browsable ontology </li></ul></ul><ul><ul><li>Our interestingness measure takes into account </li></ul></ul><ul><ul><ul><li>Specificity of the relationships and entity classes involved </li></ul></ul></ul><ul><ul><ul><li>Rarity of relationships etc. </li></ul></ul></ul><ul><ul><li>Cartic Ramakrishnan , William H. Milnor, Matthew Perry, Amit P. Sheth: Discovering informative connection subgraphs in multi-relational graphs. SIGKDD Explorations 7 (2): 56-63 (2005) </li></ul></ul>
    125. 129. <ul><li>Two factor influencing interestingness </li></ul>
    126. 130. <ul><li>Bidirectional lock-step growth from S and T </li></ul><ul><li>Choice of next node based on interestingness measure </li></ul><ul><li>Stop when there are enough connections between </li></ul><ul><li>the frontiers </li></ul><ul><li>This is treated as the candidate graph </li></ul>
    127. 131. <ul><li>Model the Candidate graph as an electrical circuit </li></ul><ul><ul><li>S is the source and T the sink </li></ul></ul><ul><ul><li>Edge weight derived from the ontology schema are treated as conductance values </li></ul></ul><ul><ul><li>Using Ohm’s and Kirchoff’s laws we find maximum current flow paths through the candidate graph from S to T </li></ul></ul><ul><ul><li>At each step adding this path to the output graph to be displayed we repeat this process till a certain number of predefined nodes is reached </li></ul></ul><ul><li>Results </li></ul><ul><ul><li>Arnold schwarzenegger, Edward Kennedy </li></ul></ul><ul><li>Other related work </li></ul><ul><ul><li>Semantic Associations </li></ul></ul>
    128. 134. <ul><li>Text Mining, Analysis  understanding  utilization in decision making  knowledge discovery </li></ul><ul><li>Entity Identification  focus change from simple to compound </li></ul><ul><li>Relationship extraction  implicit vs. explicit </li></ul><ul><li>Need more unsupervised approaches </li></ul><ul><li>Need to think of incentives to evaluate </li></ul>
    129. 135. <ul><li>Existing corpora </li></ul><ul><ul><li>GENIA , BioInfer many others </li></ul></ul><ul><ul><li>Narrow focus </li></ul></ul><ul><li>Precision and Recall </li></ul><ul><li>Utility </li></ul><ul><ul><li>How useful is the extracted information? How do we measure utility? </li></ul></ul><ul><ul><ul><li>Swanson’s discovery, Enrichment of Browsing experience </li></ul></ul></ul><ul><li>Text types and mining </li></ul><ul><ul><li>Systematically compensating for (in)formality </li></ul></ul>
    130. 136. <ul><li>http://www.cs.famaf.unc.edu.ar/~laura/text_mining/ </li></ul><ul><li>http://www.stanford.edu/class/cs276/cs276-2005-syllabus.html </li></ul><ul><li>http://www-nlp.stanford.edu/links/statnlp.html </li></ul><ul><li>http://www.cedar.buffalo.edu/~rohini/CSE718/References2.html </li></ul>
    131. 137. <ul><li>Automating Creation of Hierarchical Faceted Metadata Structures Emilia Stoica, Marti Hearst, and Megan Richardson, in the proceedings of NAACL-HLT , Rochester NY, April 2007 </li></ul><ul><li>Finding the Flow in Web Site Search , Marti Hearst, Jennifer English, Rashmi Sinha, Kirsten Swearingen, and Ping Yee, Communications of the ACM, 45 (9), September 2002, pp.42-49. </li></ul><ul><li>R. Kumar , U. Mahadevan , and D. Sivakumar , &quot;A graph-theoretic approach to extract storylines from search results&quot;,  in Proc. KDD, 2004, pp.216-225. </li></ul><ul><li>Michele Banko , Michael J. Cafarella , Stephen Soderland , Matthew Broadhead , Oren Etzioni: Open Information Extraction from the Web. IJCAI 2007 : 2670-2676 </li></ul><ul><li>Hearst, M. A. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of the 14th Conference on Computational Linguistics - Volume 2 (Nantes, France, August 23 - 28, 1992). </li></ul><ul><li>Dat P. T. Nguyen , Yutaka Matsuo, Mitsuru Ishizuka : Relation Extraction from Wikipedia Using Subtree Mining. AAAI 2007 : 1414-1420 </li></ul><ul><li>&quot;Unsupervised Discovery of Compound Entities for Relationship Extraction&quot; Cartic Ramakrishnan , Pablo N. Mendes , Shaojun Wang and Amit P. Sheth EKAW 2008 - 16th International Conference on Knowledge Engineering and Knowledge Management Knowledge Patterns </li></ul><ul><li>Mikheev, A., Moens, M., and Grover, C. 1999. Named Entity recognition without gazetteers. In Proceedings of the Ninth Conference on European Chapter of the Association For Computational Linguistics (Bergen, Norway, June 08 - 12, 1999). </li></ul><ul><li>McCallum, A., Freitag, D., and Pereira, F. C. 2000. Maximum Entropy Markov Models for Information Extraction and Segmentation. In Proceedings of the Seventeenth international Conference on Machine Learning </li></ul><ul><li>Lafferty, J. D., McCallum, A., and Pereira, F. C. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eighteenth international Conference on Machine Learning </li></ul>
    132. 138. <ul><li>Barbara, R. and A.H. Marti, Classifying semantic relations in bioscience texts, in Proceedings of the 42 nd ACL. 2004, Association for Computational Linguistics: Barcelona, Spain. </li></ul><ul><li>M.A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of COLING‘ 92, pages 539–545 </li></ul><ul><li>M. Hearst, &quot;Untangling text data mining,&quot; 1999. [Online]. Available: http://citeseer.ist.psu.edu/563035.html </li></ul><ul><li>Friedman, C., et al., GENIES: a natural-language processing system for the extraction of molecular pathways from journal articles. Bioinformatics, 2001. 17 Suppl 1: p. 1367-4803. </li></ul><ul><li>Saric, J., et al., Extraction of regulatory gene/protein networks from Medline. Bioinformatics, 2005. </li></ul><ul><li>Ciaramita, M., et al., Unsupervised Learning of Semantic Relations between Concepts of a Molecular Biology Ontology, in 19th IJCAI. 2005. </li></ul><ul><li>Dmitry Davidov, Ari Rappoport, Moshe Koppel. Fully Unsupervised Discovery of Concept-Specific Relationships by Web Mining . Proceedings, ACL 2007 , June 2007, Prague. </li></ul><ul><li>Rosenfeld, B. and Feldman, R. 2007. Clustering for unsupervised relation identification. In Proceedings of the Sixteenth ACM Conference on Conference on information and Knowledge Management (Lisbon, Portugal, November 06 - 10, 2007). </li></ul><ul><li>Michele Banko , Michael J. Cafarella , Stephen Soderland, Matthew Broadhead , Oren Etzioni : Open Information Extraction from the Web. IJCAI 2007 : 2670-2676 </li></ul><ul><li>Sekine, S. 2006. On-demand information extraction. In Proceedings of the COLING/ACL on Main Conference Poster Sessions (Sydney, Australia, July 17 - 18, 2006). Annual Meeting of the ACL. Association for Computational Linguistics, Morristown, NJ, 731-738. </li></ul><ul><li>Suchanek, F. M., Kasneci, G., and Weikum, G. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international Conference on World Wide Web (Banff, Alberta, Canada, May 08 - 12, 2007). WWW '07. </li></ul>