Getting Started with Unstructured Data
Upcoming SlideShare
Loading in...5

Getting Started with Unstructured Data



Slides used as part of a tutorial given at the Semantic Technology in Business conference in Washington D.C. Nov. 29 - Dec. 1, 2011.

Slides used as part of a tutorial given at the Semantic Technology in Business conference in Washington D.C. Nov. 29 - Dec. 1, 2011.



Total Views
Slideshare-icon Views on SlideShare
Embed Views



3 Embeds 214 118 95 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.


11 of 1

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Hi. On page 8 you make the claim that unstructured data comprises 85 % of the total available data. Could you tell me where you got this number from? Thanks for this presentation and the help.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Getting Started with Unstructured Data Getting Started with Unstructured Data Presentation Transcript

    • Getting Started with Unstructured Data Christine Connors & Kevin Lynch TriviumRLG LLC Semantic Tech & Business, Washington D.C. November 29, 2011Tuesday, November 29, 2011
    • Meta ✤ Presenter: Christine Connors ✤ @cjmconnors ✤ Presenter: Kevin Lynch ✤ @kevinjohnlynch ✤ Principals at www.triviumrlg.comTuesday, November 29, 2011
    • Agenda ✤ What is unstructured data? ✤ Where do we find it? ✤ How important is it? ✤ How do we visualize it? ✤ Machine processing for actionable data ✤ ToolsTuesday, November 29, 2011
    • What is unstructured data? ✤ Data which is ✤ Not in a database ✤ Does not adhere to a formal data model ✤ ContentTuesday, November 29, 2011
    • Isn’t that a misnomer? ✤ Problematic term ✤ The presence of object metadata or aesthetic markup does not alone give ‘structure’ in this sense of the word ✤ Object metadata = machine or applied properties ✤ Aesthetic markup = stylesheets; rendering information ✤ Semi-structured data is typically treated as unstructured for the purposes of machine processing and analysisTuesday, November 29, 2011
    • Types of ‘un’structured data ✤ Text-based documents ✤ Word processing, presentations, email, blogs, wikis, tweets, web pages, web components (read/write web) ✤ Audio/video filesTuesday, November 29, 2011
    • Where do we find it? ✤ Office productivity suites ✤ Content management systems ✤ Digital asset management systems ✤ Web content management systems ✤ Wikis, blogs, comment & discussion threads ✤ Social networking tools ✤ Twitter, Yammer, instant messengersTuesday, November 29, 2011
    • Is it really that important? Structured Unstructured 15% 85%Tuesday, November 29, 2011
    • What’s in that 80-85%? ✤ Progress reports - created in a word processorTuesday, November 29, 2011
    • What’s in that 80-85%? ✤ Dashboards - created in presentation softwareTuesday, November 29, 2011
    • What’s in that 80-85%? ✤ Progress reports - color coded text in a spreadsheetTuesday, November 29, 2011
    • What’s in that 80-85%? ✤ Brainstorming - in messaging systems ✤ Decision making - in emailTuesday, November 29, 2011
    • What’s in that 80-85%? ✤ Business intelligence - on the web and moreTuesday, November 29, 2011
    • How can we make the data more actionable? ✤ Identify it ✤ Convert to a format you can work with ✤ Add structure, meaning: ✤ information extraction ✤ annotation ✤ content analyticsTuesday, November 29, 2011
    • What about enterprise search? ✤ First line of defense ✤ Points you at the highest relevancy ranked data via pattern matching and statistical analysis ✤ Does not assist in other visualizations or transformations without further machine processingTuesday, November 29, 2011
    • Machine Processing Unstructured Natural Rules-based Statistical Semantic Data Language Classifica- Analysis Analysis Processing tion Machine Processing Platform Federated Search A P Index I Visualizations Data StoresTuesday, November 29, 2011
    • Let’s go a little deeper...Tuesday, November 29, 2011
    • Good News, Bad News ✤ Good: Basic text analysis tools are widely available; cheap or free ✤ Good: The range of information you can now consider has broadened; the intelligence you can bring to bear on that information has increased ✤ Bad: Skillsets not widely available (but they are available!) ✤ Good: You can get started right here, understanding, identifying the sources, and possible approachesTuesday, November 29, 2011
    • What Data Doesn’t Do ✤ From Coco Krumme in “Beautiful Data” ✤ Data doesn’t drive everything. ✤ Note: “narrative fallacy,” “confirmation bias,” “paradox of choice” ✤ Data doesn’t: scale (cognitively), alone explain, predict ✤ The real world doesn’t create random variables ✤ Data doesn’t stand aloneTuesday, November 29, 2011
    • Integrating Unstructured Data Images From Oracle 11g presentation at, November 29, 2011
    • The Goal: Usable Knowledge ✤ Information extraction is NOT the goal ✤ Information extraction is a means to an end ✤ Knowledge discovery is the goal ✤ To this end, we will perform lots of processing to move from bits to usable meaningTuesday, November 29, 2011
    • So many <near> synonyms ✤ Text analytics ✤ Content analytics ✤ Text mining ✤ Data mining ✤ Information extraction ✤ And then there’s Natural Language ProcessingTuesday, November 29, 2011
    • What’s the same? ✤ Moving from bits to meaning requires processing, and a lot of that processing is the same, no matter what you call it ✤ We will focus primarily on textual information todayTuesday, November 29, 2011
    • Natural Language ✤ From Peter Norvig’s “Natural Language Corpus Data: chapter in “Beautiful Data” ✤ Google’s 1 trillion-word corpus investigating probabilistic language models ✤ 13 million types (unique words, punctuation) ✤ 100k types cover 98% of the corpus ✤ For: word segmentation, spelling correction, language identification, spam detection, author identification ✤ %? = “chooses pain” ; “in sufficient numbers”Tuesday, November 29, 2011
    • Information Extraction ✤ Token identification - “tokenization” ✤ Word segmentation ✤ Sentence splitting ✤ Part-of-speech tagging - “POS” tagging (noun, verb, adverb, adjective, etc.) ✤ Phrase identification - noun phrase ✤ Entity extraction - people, places, events, dates, organizationsTuesday, November 29, 2011
    • Information Extraction ✤ Cluster analysis - group related information, where relationship may not be known ✤ Classification - mapping to specific categories ✤ Dependency identification / Rule generation ✤ Relationship detection - e.g. “Joe” “is CEO” at “IBM” ✤ Conference resolution (anaphoric reference resolution) ✤ e.g., “Joe is CEO at IBM. He is an IEEE member.” ✤ Summarization - key concepts or key sentencesTuesday, November 29, 2011
    • IR and IE ✤ IR (Information Retrieval) versus IE (Information Extraction) ✤ IR retrieves documents from collections; IE retrieves facts and structured information from collections ✤ In IR, the objects of analysis are documents; in IE, the objects of analysis are facts ✤ IE returns knowledge at a deeper level than traditional IR ✤ Results may be imperfect, and linking them back to documents adds value ✤ Sound familiar? (semantic web, linked data)Tuesday, November 29, 2011
    • Information Extraction Two primary system types Knowledge Engineering Learning Systems Rule based Use statistics or other machine learning Developed by experienced language engineers Developers do not need language engineering expertise Make use of human intuition Require only small amount of training data Require large amounts of annotated training data Development can be very time consuming Some changes may require re-annotation of the entire Some changes may be hard to accommodate training corpus From, November 29, 2011
    • Text Predicate Subject Object Two views of the semantic web Machine learning, natural language processing, artificial intelligence and linked data Images from WikipediaTuesday, November 29, 2011
    • Named Entities ✤ What is NER? ✤ Named Entity Recognition ✤ identifying proper names in texts, and classification into a set of predefined categories of interest ✤ Named entity recognition is the cornerstone of Information Extraction, providing a foundation from which to build complex information extraction systemsTuesday, November 29, 2011
    • Named Entities ✤ Person names ✤ Organizations (companies, government organizations, committees) ✤ Locations (cities, countries, rivers) ✤ Date and time expressions ✤ Measures (percent, money, weight) ✤ Email addresses, web addresses, street addresses ✤ Some domain-specific entities: names of drugs, medical conditions, names of ships, bibliographic references, etc.Tuesday, November 29, 2011
    • NOT Named Entities ✤ Artifacts - Wall Street Journal ✤ Common nouns, referring to named entities ✤ e.g. the company, the committee ✤ Name of groups of people and things named after people ✤ e.g. the Tories, the Nobel Prize ✤ Adjectives derived from names ✤ e.g. Bulgarian, Chinese ✤ Numbers which are not times, dates, percentages or money amounts, November 29, 2011
    • Break Time!Tuesday, November 29, 2011
    • Open Tools ✤ GATE – General Architecture for Text Engineering, from the University of Sheffield, with many users and excellent documentation. ✤ GATE has customizable document and corpus processing pipelines. GATE is an architecture, a framework, and a development environment, with a clean separation of algorithms, data, and visualization.Tuesday, November 29, 2011
    • GATE ✤ “The Volkswagen Beetle of language processing” ✤ “...more than a decade of collecting reusable code and building a community has lead [to] a mature ecosystem for solving language processing problems quickly.” ✤ Hamish Cunningham 2010Tuesday, November 29, 2011
    • GATE – Key Features ✤ Component-based development ✤ Automatic performance measurement ✤ Clean separation between data structures and algorithms ✤ Consistent use of standard mechanisms for components to communicate data ✤ Insulation from data formats ✤ Provision of a baseline set of language componentsTuesday, November 29, 2011
    • GATE – More... ✤ Free – open source, LPGL, Java ✤ Mature, at version 6, actively supported, 15 FTEs ✤ Comprehensive, standards-based, popular ✤ Used by thousands of companies, universities, and research laboratories ✤ Well-known, tested, researched, and very well-documentedTuesday, November 29, 2011
    • GATE Overview ✤ Architectural principles ✤ Non-prescriptive, theory neutral (strength and weakness) ✤ Re-use, interoperation, not reimplementation (diverse support, lots of plugins) ✤ (Almost) everything is a component, and component sets are user-extendable ✤ Component-based development ✤ CREOLE = modified Java Beans (Collection of REusable Objects for Language Engineering) ✤ The minimal component = 10 lines of Java, 10 lines of XML, 1 URLTuesday, November 29, 2011
    • GATE – Family ✤ GATE Developer – an integrated development environment for language processing components bundled with the most widely used Information Extraction system and a comprehensive set of plugins ✤ GATE Embedded – an object library optimized for inclusion in diverse apps ✤ GATE Teamware – web app, a collaborative annotative environment ✤ GATE Cloud – parallel distributed processingTuesday, November 29, 2011
    • GATE – Embedded From, November 29, 2011
    • GATE – Teamware ✤ GATE Teamware – web app, a collaborative annotative environment for high volume factory-style semantic annotation built with workflow ✤ Running in 5 minutes with Teamware virtual server from (itself open source): ✤ Reusable project templates ✤ Project-specific roles, users ✤ Applying GATE-based processing routines ✤ Project status, annotator activity, statisticsTuesday, November 29, 2011
    • GATE – First Cousins ✤ Ontotext KIM: UIs demonstrating the multi-paradigm approach to information management, navigation and search ✤ Ontotext Mimir: a massively scalable multi-paradigm index built on Ontotext’s semantic repository family, GATE’s annotation structures database, plus full-text indexing from MG4 ✤ Ontotext FactForge: ~4B Linked Data statements, query-ableTuesday, November 29, 2011
    • GATE – Ontotext KIM ✤ Ontotext KIM: UIs, tools, GATE Gazetteers, including a Linked Data gazetteer (experimental) ✤ Pre-loaded knowledge base for entities ✤ Tools to upload, query, tailor the knowledge base, algorithms, UI ✤ Can crawl web, including Linked Data, creating semantic index: your servers, theirs, or cloud ✤ Based on GATE and OWLIMTuesday, November 29, 2011
    • GATE – Ontotext KIM From:, November 29, 2011
    • GATE – Ontotext KIM StructureTuesday, November 29, 2011
    • GATE – Ontotext KIM PatternsTuesday, November 29, 2011
    • GATE – Ontotext KIM OntologyTuesday, November 29, 2011
    • GATE – Ontotext KIM FacetsTuesday, November 29, 2011
    • GATE – Ontotext MIMIR ✤ Ontotext Mimir: large scale indexing infrastructure supporting hybrid search (text, annotation, meaning); massively scalable multi-paradigm capability, combines MG4J full-text index and BigOWLIM semantic repository; query with text, structural info, and SPARQL ✤ Integrated with GATE, customizable, scalable ✤ Open source components ✤ Can federate multiple MIMIRs ✤ Low acquisition, management cost to scaleTuesday, November 29, 2011
    • GATE – Multi-paradigm ✤ Why “multi-paradigm?” Proliferation of retrieval technology options ✤ Full text, boolean, proximity, ranking; behavior mining, tag clouds; concept indexing: taxonomic, ontological; annotation-based ✤ Choice depends principally on content volume + value: ✤ High volume, low (average) value: web search ✤ Medium volume, higher (personal) value: social networks, photo sharing, tagging ✤ Low volume, high value: controlled vocabularies, taxonomies, ontologiesTuesday, November 29, 2011
    • GATE “Resources” ✤ Applications – groups of processes (that run on one or more documents) ✤ Language Resources – documents or document collections (corpus, corpora) ✤ Processing Resources – annotation tools that operate on text in documents ✤ Applications, made up of Processing Resources, operate on Language ResourcesTuesday, November 29, 2011
    • Plugins ✤ Applications – an application consists of any number of Processing Resources, run sequentially over documents ✤ Plugins – a plugin is a collection of one or more Processing Resources, bundled together. ✤ Plugins, then, are applications, that need to be loaded in order to access their Processing Resources.Tuesday, November 29, 2011
    • GATE – Plugins (I)Tuesday, November 29, 2011
    • GATE – Plugins (II)Tuesday, November 29, 2011
    • GATETuesday, November 29, 2011
    • GATE Annotations ✤ Annotations are central to understanding GATE ✤ Annotations are associated with each document ✤ Each annotation has: ✤ start and end offsets ✤ an optional set of features ✤ each feature has a name and a valueTuesday, November 29, 2011
    • GATE AnnotationsTuesday, November 29, 2011
    • GATE AnnotationsTuesday, November 29, 2011
    • Information Extraction ✤ TE: Template Elements ✤ NE: Named Entity recognition and typing ✤ TR: Template Relations ✤ CO: CO-reference resolution ✤ ST: Scenario Templates ✤ Example: The shiny red rocket was fired on Tuesday. It is the brainchild of Dr. Big Head. Dr. Head is a staff scientist at We Build Rockets Inc. ✤ NE: Entities are “rocket,” “Tuesday,” “Dr. Head” and “We Build Rockets” CO: “it” refers to the rocket; “Dr. Head” and “Dr. Big Head” are the same TE: the rocket is “shiny red” and Head’s “brainchild” TR: Dr. Head works for “We Build Rockets Inc.” ST: a rocket launching event occurred with the various participants From, November 29, 2011
    • ANNIE ✤ A Nearly-New Information Extraction System, packaged with GATE, used throughout examples, and a great place to start ✤ A collection of GATE Processing Resources to perform Information Extraction on unstructured text ✤ “Nearly new” – its name 10 years ago, that stuck ✤ Other information extraction systems include LingPipe and OpenNLP. GATE includes wrappers for LingPipe and OpenNLP, independently developed NLP pipelines. All three systems are provided as pre-built application through the GATE File menuTuesday, November 29, 2011
    • ANNIE ✤ “Processing Resources” inside ANNIE: ✤ Tokenizer, sentence splitter, part-of-speech tagger, gazetteers, named entity tagger, and an orthomatcher ✤ Also included are noun phrase and verb phrase chunkers ✤ Each “Processing Resource” inside ANNIE can be used as part of a pipeline you create to add annotations or modify existing ones ✤ ANNIE is a highly customizable, rule-based system, with very useful defaultsTuesday, November 29, 2011
    • ANNIE ✤ “Processing Resources” inside ANNIE: ✤ Gazetteer – lookup annotations (lists) ✤ JAPE transducer – date, person, location, organization, money, percent annotations ✤ Orthomatcher – adds match features to named entity annotations (coreference matching) ✤ Document Reset – removes annotationsTuesday, November 29, 2011
    • IE Steps in ANNIE ✤ “Tokenizer” performs Token identification and word segmentation ✤ “Sentence splitter” identifies sentences ✤ “POS” tagger performs Part-of-speech tagging – (noun, verb, adverb, adjective) ✤ Must run Tokenizer and Sentence Splitter before POS taggerTuesday, November 29, 2011
    • IE Steps in ANNIE ✤ “Gazetteers” – lists of names (people, cities, groups); you can modify or add lists ✤ Each list has features (majorType, minorType, language) ✤ Gazetteers generate “Lookup” annotations with features corresponding to the matched list. When the text matches a gazetteer entry, a Lookup annotation is created. ✤ Lookup annotation are used by ANNIE’s Named Entity transducer to for entity identification.Tuesday, November 29, 2011
    • ANNIE in GATETuesday, November 29, 2011
    • ANNIE in GATETuesday, November 29, 2011
    • ANNIE in GATETuesday, November 29, 2011
    • ANNIE Sequence Pipeline sequence matters: tokenizer, sentence splitter, POS tagger, gazetteerTuesday, November 29, 2011
    • IE Steps in ANNIE ✤ “NE Transducer” – Named Entity Transducer performs named entity recognition (NER) ✤ Once we have built up the processing resource pipeline with the previous steps (tokeniser, sentence splitter, POS tagger, gazetteer), we are ready to add the transducer for named entity recognition ✤ More specific information can be added to the features now, including the “kind” of entity, and the rules that were firedTuesday, November 29, 2011
    • IE Steps in ANNIE ✤ “OrthoMatcher” – orthographic co-reference matches proper names and their variants. ✤ Will match previously unclassified names, based on relations with classified entities ✤ Matches “Kevin Lynch” with “Dr. Lynch” ✤ Matches acronyms with expansionsTuesday, November 29, 2011
    • IE Steps in ANNIE ✤ Tokenizer, sentence splitter, and OrthoMatcher are language, domain, and application-independent ✤ Part-of-speech tagger is language dependent and application- independent ✤ Gazetteer lists are starting points (60K entries) ✤ ANNIE is a way to get started, with a framework for identifying the kinds of elements that matter to your work, and for quickly testing your ideas against existing dataTuesday, November 29, 2011
    • Annotations In ContextTuesday, November 29, 2011
    • Rules-based Classification ✤ Once a stand-alone project, now often part of annotation services ✤ Regex, Boolean and naive Bayesian algorithms executed on tokens ✤ And, Or, Not, Near (x), Multi, Stem, Exact, Phrase, et al (vendor or source dependent) ✤ Assigns documents to a taxonomic category ✤ Allow for greater control over depth and breadth of categories ✤ Human aided, machine processedTuesday, November 29, 2011
    • Rules-based ClassificationTuesday, November 29, 2011
    • Break Time!Tuesday, November 29, 2011
    • Visualization - PrefuseTuesday, November 29, 2011
    • Visualization - PrefuseTuesday, November 29, 2011
    • Visualization - PrefuseTuesday, November 29, 2011
    • Visualization - PrefuseTuesday, November 29, 2011
    • Visualization - PrefuseTuesday, November 29, 2011
    • Visualization - PrefuseTuesday, November 29, 2011
    • Visualization - GephiTuesday, November 29, 2011
    • Visualization - GephiTuesday, November 29, 2011
    • Visualization - CytoscapeTuesday, November 29, 2011
    • Quick! ✤ Take one large pile of text (documents, emails, tweets, patents, papers, transcripts, blogs, comments, acts of parliament, and so on and so forth) -- call this your corpus ✤ Pick a structured description of interesting things in the text (a telephone directory, or chemical taxonomy, or something from the Linked Data cloud) -- call this your ontology ✤ Use GATE Teamware to mark up a gold standard example set of annotations of the corpus (1.) relative to the ontology (2.) ✤ Use GATE Developer to build a semantic annotation pipeline to do the annotation job automatically and measure performance against the gold standard ✤ Take the pipeline from 4. and apply it to your text pile using GATE Cloud (or embed it in your own systems using GATE Embedded) ✤ Use GATE Mimir to store the annotations relative to the ontology in a multiparadigm index server. (For techies: this sits in the backroom as a RESTful web service.) ✤ Use Ontotext KIM to add semantic search, knowledge facet search, ontology browsing, entity popularity graphing, time series graphing, annotation structure search and (last but not least) boolean full text search. (More techy stuff: mash up these types of search with your existing UIs.)Tuesday, November 29, 2011
    • Data Warehousing / Business Intelligence ✤ Perspective ✤ Process ✤ Use cases ✤ Implications with unstructured dataTuesday, November 29, 2011
    • DW/BI Perspective ✤ Structured data is an incomplete version of the “truth” ✤ Until information is quantified, it is not very useful ✤ Discover facts, and give them structure ✤ Complement structured data with unstructured data; try to complete the picture (of the business, the customer, performance)Tuesday, November 29, 2011
    • DW/BI Process ✤ Extract, then formalize ✤ Give information structure, then associations ✤ Map to existing structures in the data warehouseTuesday, November 29, 2011
    • DW/BI Use Cases ✤ Report indexing (of metadata, of instances) ✤ Report sections become possible ✤ Self-service for consumers ✤ “BI Search” (of those reports) ✤ Include in portal ✤ As range of reports and users increases, unstructured data approaches have more valueTuesday, November 29, 2011
    • DW/BI Use Case Ideas ✤ For customers, products, complaints, locations: ✤ Voice recognition indexing ✤ RSS feeds ✤ Wikis, blogs (internal and external) ✤ Instant messagesTuesday, November 29, 2011
    • DW/BI Implications ✤ Have to store these results ✤ Have to model these results ✤ Have to map these results to something meaningful ✤ Have to include the results in a useful way (Where? Use taxonomies? Which ones?) ✤ Quality, cost, and complexity matter; extracted entities don’t relate directly to performance ✤ Not a replacement, an addition to the technologyTuesday, November 29, 2011
    • Some Technical Issues ✤ Quality ✤ Integration ✤ Concurrency ✤ Security ✤ SkillsTuesday, November 29, 2011
    • Additional Open Tools ✤ UIMA – Unstructured Information Management Architecture (IBM’s Watson uses this), originated at IBM, now an Apache project. ✤ Component software architecture with a document processing pipeline similar to GATE. Focus on performance and scalability, with distributed processing (web services).Tuesday, November 29, 2011
    • UIMA UIMA’s Basic Building Blocks are Annotators. They iterate over an artifact to discover new types based on existing ones and update the Common Analysis Structure (CAS) for upstream processing. UIMA CAS Representation now Common Analysis Structure (CAS) Aligned with XMI standard Relationship CeoOf Arg1:Person Arg2:Org Analysis Results (i.e., Artifact Metadata) Named Entity Person Organization Parser NP VP PP Fred Center is the CEO of Center Micros Artifact (e.g., Document) Chart by IBMTuesday, November 29, 2011
    • UIMA Image by IBMTuesday, November 29, 2011
    • Commercial Tools ✤ Oracle Data Mining (Text Mining) ✤ IBM SPSS ✤ SAS Text Miner ✤ Smartlogic ✤ Lots of acquisitions going on in the “big data” space ✤ HP acquired Autonomy ✤ Oracle acquired EndecaTuesday, November 29, 2011
    • A Note on Tools ✤ UIMA and GATE – comprehensive suite of capabilities, with learning curves. ✤ Commercial tools range from unstructured capabilities inside DBMSs like Oracle, to Business Objects business intelligence tools (who acquired Inxight from Xeroc Parc). ✤ Your mileage will vary. The biggest differentiator is your knowledge of your data.Tuesday, November 29, 2011
    • Questions?Tuesday, November 29, 2011
    • Thank you Christine Connors Kevin Lynch www.triviumrlg.comTuesday, November 29, 2011
    • What can unstructured data look like post-processing?Tuesday, November 29, 2011