The document discusses semantic content management for enterprises and national security, emphasizing the challenges of information overload and the need for effective content distribution and personalization. It delves into the technical aspects of semantic metadata extraction and the architecture of a semantic content organization and retrieval engine, which leverages distributed agents to improve information accessibility and relevance. The importance of ontology-driven systems for enhancing decision-making capabilities through actionable insights is highlighted, alongside the development of tools for automated content tagging and categorization.

1. SEMANTIC CONTENT MANAGEMENT FOR ENTERPRISES AND NATIONAL SECURITY Amit Sheth CTO, Voquette*, Inc. Large Scale Distributed Information Systems (LSDIS) Lab University Of Georgia; http://lsdis.cs.uga.edu *Now Semagix, http://www.semagix.com July 15, 2002 © Amit Sheth Keynote CONTENT- AND SEMANTIC-BASED INFORMATION RETRIEVAL @ SCI 2002

2. New Enterprise Content Management Challenges More variety and complexity More formats (MPEG, PDF, MS Office, WM, Real, AVI, etc) More types (Docs, Images -> Audio, Video, Variety of text-structured, unstructured) More sources (internal, extranet, internet, feeds) Saclability, Information Overload Too much data, precious little information (Relevance) Creating Value from Content How to Distribute the right content to the right people as needed? (Personalization -- book of business) Customized delivery for different consumption options (mobile/desktop, devices) Insight, Decision Making (Actionable)

3. New Enterprise Content Management Technical Challenges Aggregation Feed handlers/Agents that understand content representation and media semantics Push-pull, Web-DB-Files, Structured-Semi-structured-Unstructured data of different types Homogenization and Enhancement Enterprise-wide common view Domain model, taxonomy/classification, metadata standards Semantic Metadata– created automatically if possible Semantic Applications Search, personalization, directory, alerts, etc. using metadata and semantics (semantic association and correlation), for improved relevance, intelligent personalization, customization

4. The Semantic Web -- a vision with several views: · “The Web of data (and connections) with meaning in the sense that a computer program can learn enough about what data means to process it.” [B99] · “The semantic Web is an extension of the current Web in which information is given well-defined meaning, better enabling computers and people to work in cooperation.” [BHL01] · “The Semantic Web is a vision: the idea of having data on the Web defined and linked in a way that it can be used by machines not just for display purposes, but for automation, integration and reuse of data across various applications. [W3C01] Semantics: The Next Step in the Web’s Evolution

5. Semantics for the Web On the Semantic Web every resource (people, enterprises, information services, application services, and devices) are augmented with machine processable descriptions to support the finding, reasoning about (e.g., which service is best), and using (e.g., executing or manipulating) the resource. The idea is that self-descriptions of data and other techniques would allow context-understanding programs to selectively find what users want, or for programs to work on behalf of humans and organizations to make them more efficient and productive.

7. Digital Content and Semantics

8. Central Role of Metadata Applications Back End "A Web content repository without metadata is like a library without an index." - Jack Jia, IWOV “ Metadata increases content value in each step of content value chain.” Amit Sheth Where is the content? Whose is it? Produce Aggregate What is this content about? Catalog/ Index What other content is it related to? Integrate Syndicate What is the right content for this user? Personalize What is the best way to monetize this interaction? Interactive Marketing Broadcast, Wireline, Wireless, Interactive TV Semantic Metadata

9. A Metadata Classification Data (Heterogeneous Types/Media) Content Independent Metadata (creation-date, location, type-of-sensor...) Content Dependent Metadata (size, max colors, rows, columns...) Direct Content Based Metadata (inverted lists, document vectors, LSI) Domain Independent (structural) Metadata (C++ class-subclass relationships, HTML/SGML Document Type Definitions, C program structure...) Domain Specific Metadata area, population (Census), land-cover, relief (GIS),metadata concept descriptions from ontologies Ontologies Classifications Domain Models User More Semantics for Relevance to tackle Information Overload!!

10. Semantic Content Organization and Retrieval Engine (SCORE) technology Automatically aggregates and extracts information from disparate sources and multiple formats Automatically tags/annotates and categorizes content Automatically creates relevant associations Maps content topics and their relationships Semantic query engine relates information and knowledge both internal and external to the organization into a single view

11. SCORE Architecture

12. SCORE Architecture Distributed agents that automatically extract relevant semantic metadata from structured and unstructured content Fast main-memory based query engine with APIs and XML output CACS provides automatic classification (w.r.t. WorldModel) from unstructured text and extracts contextually relevant metadata Distributed agents that automatically extract/mine knowledge from trusted sources Toolkit to design and maintain the Knowledgebase Knowledgebase represents the real-world instantiation (entities and relationships) of the WorldModel WorldModel specifies enterprise’s normalized view of information (ontology)

13. Voquette Enterprise Semantic Platform Product Components World Model WM Toolkit Knowledgebase and Metabase Main Memory Index XML APIs Web Services Enterprise Applications EA EA EA Semantic Engine Search Alerts Portals Directory Personalize Enhancement Engine CA CA CA Content Agent Monitor Content Agents Databases XML/Feeds Websites Email Content Sources Entity Extraction, Enhanced Metadata, Domain Experts Automatic Classification Classification Committee Reports Documents Structured Semi- Structured Unstructured CA Toolkit Knowledge Agent Monitor KS KS KS KS KA KA KA Knowledge Sources Knowledge Agents KA Toolkit Knowledgebase KB Toolkit Knowledge Agent Monitor KS KS KS KS KA KA KA Knowledge Sources Knowledge Agents KA Toolkit Metabase Enhancement Engine CA CA CA Content Agent Monitor Content Agents Databases XML/Feeds Websites Email Content Sources Entity Extraction, Enhanced Metadata, Domain Experts Automatic Classification Classification Committee Reports Documents Structured Semi- Structured Unstructured CA Toolkit

14. PERSON (OFAC, FBI, DPL) -politician (OFAC, FBI, CIA, CA) politician associated with politicalOrganziation politician held politicalOffice politician associated with politicalOffice -terrorist (OFAC, FBI, DPL) terrorist memberOf organization terrorist appears on watchList -companyExecutive (MG) companyExecutive holdsOffice companyPosition person has permanent address address (OFAC, FBI) person has dob(date of birth) (OFAC, FBI) person has pob(place of birth) (OFAC, FBI) Knowledge Sources Used THING -event (ICT) terroristOrganization participated in terroristSponsoredEvent (ICT) -politicalOffice (CIA, CA) politicalOffice office(s) within govtOrganization politicalOffice associated with organization -watchList (OFAC, FBI, DPL) terroristOrganization appears on watchList (OFAC, FBI, DPL) -organization (OFAC, FBI, FAS, ICT, CA, CIA) organization appears on watchList organization memberOf suborganization -company company manufactures product (ZD) company identifiedBy tickeySymbol (H) companyposition position in company (MG) company memberOf industry (H) -tickerSymbol (H) tickerSymbol memberOf exchange (H) PLACE -organization located in place (H, OFAC) -religiousAffiliation practiced in place (CIA) -company headquarters in city (H) Entity Classes and Relationships populated by these knowledge sources: JIVA Market Guide (MG) ZDNet (ZD) Hoover’s (H) Data supplied from NASA (DPL) Federation of American Scientists (FAS) C entral Intelligence Agency (CIA) The Interdisciplinary Center (ICT) Federal Bureau of Investigation (FBI) Capital Advantage (CA) Office of Foreign Assets Control ( OFAC)

15. SCORE Capabilities Semantics (understanding of content and user needs) Extreme relevance Semantic associations Near real-time Multiple applications/usage patterns (not just search) Automation Scalability in all aspects

16. Technologies Involved Ontology driven architecture (definitional, assertional components Automatic Classification with classifier committee (multiple technologies, rather than one size fits all) Automatic Semantic Metadata Extraction/Annotation Semantic associations/ knowledge inferences Scalability throughout with distributed architecture and implementation (number of content and knowledge sources, indexing, etc.) Main memory implementation, incremental check pointing

17. Performance > 10,000 entities/relationships per hr. Population/update rate in a Knowledgebase with 1 million entities/relationships 1 minute (near real-time) Incremental Index Update Frequency 65ms Query Response Time (64 concurrent users) 1 - 10 ms Query Response Time (light load) > 1,980,000 Queries per server per hour

18. Information Extraction for Metadata Creation METADATA EXTRACTORS Key challenge: Create/extract as much (semantics) metadata automatically as possible WWW, Enterprise Repositories Digital Maps Nexis UPI AP Feeds/ Documents Digital Audios Data Stores Digital Videos Digital Images . . . . . . . . .

19. Video with Editorialized Text on the Web Automatic Categorization & Metadata Tagging (Web page) Auto Categorization Semantic Metadata

20. Extraction Agent Web Page Enhanced Metadata Asset Content Extraction and Knowledgebase Enhancement

21. Content Enhancement Workflow Semantic Metadata Syntax Metadata

22. Content Asset Index Evolution Extractor Agent for Bloomberg Scans text for analysis Metadata extracted automatically Asset Syntax Metadata Producer: BusinessWire Source: Bloomberg Date: Sept. 10 2001 Location: San Jose, CA URL: http://bloomberg.com/1.htm Media: Text Semantic Metadata Company: Cisco Systems, Inc. Creates asset (index) out of extracted metadata Asset Syntax Metadata Producer: BusinessWire Source: Bloomberg Date: Sept. 10 2001 Location: San Jose, CA URL: http://bloomberg.com/1.htm Media: Text Semantic Metadata Company: Cisco Systems, Inc. Topic: Company News Categorization & Auto-Cataloging System (CACS) Scans text for analysis Classifies document into pre-defined category/topic Appends topic metadata to asset Cisco Systems CSCO NASDAQ Company Ticker Exchange Industry Sector Executives John Chambers Telecomm. Computer Hardware Competition Nortel Networks Knowledge Base CEO of Competes with Syntax Metadata Asset Producer: BusinessWire Source: Bloomberg Date: Sept. 10 2001 Location: San Jose, CA URL: http://bloomberg.com/1.htm Media: Text Semantic Metadata Company: Cisco Systems, Inc. Topic: Company News Ticker: CSCO Exchange: NASDAQ Industry: Telecomm. Sector: Computer Hardware Executive: John Chambers Competition: Nortel Networks Headquarters: San Jose, CA Leverages knowledge to enhance metatagging Enhanced Content Asset Indexed Headquarters San Jose XML Feed Semantic Engine

23. Content which does contain the words the user asked for Extractor Agents Content which does not contain the words the user asked for, but is about what he asked for. Value-added Metadata Content the user did not think to ask for , but which he needs to know . Semantic Associations + + Intelligent Content End-User Intelligent Content Empowers the User

24. Example 1 – Snapshots (“Jamal Anderson”) Click on first result for Jamal Anderson View metadata. Note that Team name and League name are also included in the metadata Search for ‘Jamal Anderson’ in ‘Football’ View the original source HTML page. Verify that the source page contains no mention of Team name and League name . They are value-additions to the metadata to facilitate easier search.

25. Semantic Application Example – Research Dashboard Focused relevant content organized by topic ( semantic categorization ) Automatic Content Aggregation from multiple content providers and feeds Related relevant content not explicitly asked for (semantic associations) Competitive research inferred automatically Automatic 3 rd party content integration

26. Internal Source 1 Research Internal Source 2 External feeds/Web (e.g. Reuters) Voquette Metabase World Model Third-party Content Mgmt And Syndication Semantic Engine 1 2 3 4 Cisco story from Source 1 passed on to add semantic associations Consults Knowledge Base for Cisco ’s competition Returns result: Lucent is a competitor of Cisco Lucent story from external feeds picked for publishing as “semantically related” to Cisco story – passed on to Dashboard Story on Lucent Story on Cisco XCM-compliant metadata, XML or other format Semantic Application ASP/Enterprise hosted Extractor Agent 1 Extractor Agent 2 Extractor Agent 3 Metadata centric Content Management Architecture

27. Related Stock News Semantic Web – Intelligent Content Industry News Technology Products COMPANY EPA Regulations Competition COMPANIES in Same or Related INDUSTRY COMPANIES in INDUSTRY with Competing PRODUCTS Impacting INDUSTRY or Filed By COMPANY Important to INDUSTRY or COMPANY Intelligent Content = What You Asked for + What you need to know! SEC

28. led by Same entity Human-assisted inference Knowledge-based & Manual Associations Syntax Metadata Semantic Metadata

29. Blended Semantic Browsing and Querying (Intelligence Analyst Workbench)

30. Innovations that affect User Experience BSBQ: Blended Semantic Browsing and Querying Ability to query and browse relevant desired content in a highly contextual manner Seamless access/processing of Content, Metadata and Knowledge Ability to retrieve relevant content, view related metadata, access relevant knowledge and switch between all the above, allowing user to follow his train of thought dACE: dynamic Automatic Content Enhancement Ability to provide enhanced annotation features, allowing the user to retrieve relevant knowledge about significant pieces of content during content consumption Semantic Engine APIs with XML output Ability to create customized APIs for the Semantic Engine involving Semantic Associations with XML output to cater to any user application

31. Visionics AcSys Security Portal Check-in Interrogation Boarding Gate Airport Airspace Voquette Knowledgebase Metabase Threat Scoring Gov’t Watchlists News Media Web Info LexisNexis RiskWise Passenger Records Reservation Data Airline Data Airport Data Airline and Airport Data Future and Current Risks Airport LEO ARC AvSec Manager Data Management Data Mining IPG

32. Sources Used Content Sources : Africa News Service AFX News – Asia/UK/Europe AP Worldstream Asia Pulse BusinessWire ComputerWire (CTW) EFE News Services FWN Select Itar-TASS Knight Ridder News (Open) Knight-Ridder Open M2 - International M2 Airline Industry Information New World Publishing PR Newswire PRLine (PRL) Resource News International RosBusiness United Press International UPI Spotlights Knowledge Sources: FBI - Most Wanted Terrorists Denied Persons Lists Terrorism Files ICT Office of Foreign Asset Control (OFAC) Hamas terrorists CNN Locations FAA_Airport_Codes About.com Comtex_International Hindustan Times JerusalemPost CNN Newstrove_Hamas

33. Voquette’s Semantic Technology enables flight authorities to : - take a quick look at the passenger’s history - check quickly if the passenger is on any official watchlist - interpret and understand passenger’s links to other organizations (possibly terrorist) - verify if the passenger has boarded the flight from a “high risk” region - verify if the passenger originally belongs to a “high risk” region - check if the passenger’s name has been mentioned in any news article along with the name of a known bad guy Interrogation Kiosk – Unique Advantages of Voquette Smith John

34. Threat Score Components Smith John WATCHLIST ANALYSIS Action : Voquette’s rich knowledgebase is automatically searched for the possible appearance of this name on any of the watchlists Ability Proven : Ability to automatically aggregate relevant rich domain knowledge and automatically co-relate it and rank the threat factors to indicate threat level of the passenger on the watchlist front METABASE SEARCH Action : Voquette’s rich metabase is searched for this name and associated content stories mentioning the passenger’s name are retrieved Ability Proven : Ability to automatically aggregate and retrieve relevant content stories, field reports, etc. about the passenger that can be used by flight officials to determine if the passenger has any connections with known bad people or organizations appearsOn watchList : FBI KNOWLEDGEBASE SEARCH Action : Voquette’s rich knowledgebase is searched for this name and associated information like position, aliases, relationships (past or present) of this name to other organizations, watchlists, country, etc. are retrieved Ability Proven : Ability to automatically aggregate relevant rich domain knowledge about a passenger and automatically co-relate it with other data in the knowledgebase to present a visual association picture to the flight official LEXIS NEXIS ANNOTATION Action : Information about or related to the passenger returned by Lexis Nexis is enhanced by linking important entities to Voquette’s rich knowledgebase Ability Proven : Ability to automatically aggregate relevant rich domain knowledge, recognize entities in a piece of text and further automatically co-relate it with other data in the knowledgebase to present a clear picture about the passenger to the flight official Flight Coutry Check 45 0.15 Person Country Check 25 0.15 Nested Organizations Check 75 0.8 Aggregate Link Analysis Score: 17.7 LINK ANALYSIS Action : Semantic analysis of the various components (watchlist, Lexis Nexis, knowledgebase search, metabase search, etc.) to come up with an aggregate threat score for the passenger Ability Proven : Ability to automatically aggregate relevant rich domain knowledge, recognize entities in a piece of text, automatically co-relate it with other data in the knowledgebase, search for relevant content to present an overall idea of the threat level fo the passenger, allowing him to take quick action

35. Query Comparison: Voquette vs. RDBMS

36. JIVA Semantic Console Start-up Interface The mission of the JIVA project is to gather and analyze as much information of diverse kinds about suspected individuals, terrorist and other groups, organizations, events, etc. For this Terrorism domain, the JIVA Semantic Console provides an information retrieval interface (shown below) that displays some fundamental semantic attributes (based on a corresponding Terrorism domain model) to enable information retrieval in the right context. Most fundamental semantic attributes specific to the Terrorism domain (fully customizable) Syntactic or domain-independent attributes for general and media-specific search Analyst can enter search values in the appropriate attribute fields (to search in the right context) Analyst can choose the type of media of the desired content Once all other values are set, click the “ Search” button to search semantically Search interface with more search features (explained later) JIVA Functionality Interface

37. “ Complete Picture” View – Knowledgebase Results This section of the ‘Complete Picture’ shows factually known real-world information about the entity (person, organization, event, etc.) of interest along with its contextual classification(s) and relationships with other entities in the Knowledgebase, to provide a comprehensive overview of the entity. Such knowledge is kept up-to-date by means of automated knowledge extractor agents that aggregate such knowledge about millions of entities from various trusted knowledge sources. Entity’s canonical name Entity’s classifications in taxonomy Entity’s aliases and other names Entity’s real-world relationships to various other entities across multiple entity classes (as defined in the Terrorism domain model) Individual related entities are clickable to navigate to a new knowledge page for that entity e.g. Al Qaeda Knowledgebase Navigation While browsing through relevant knowledge, analyst can search for content on the focal entity or any of the related entities. The analyst can also search for specific relationships between two or more entities by checking corresponding entity boxes for search - Blended Semantic Browsing & Querying (BSBQ) Fraud investigation of focal entity placing it in one of five levels of threats, based on score JIVA

38. Facilitating Knowledge Discovery On clicking any bin Laden-related entity (e.g. Al Qaeda), a page is displayed to the analyst showing knowledge pertaining to that entity, which can be used in a BSBQ mode, as described on the previous screen. Continuing this integrated approach of Semantic Browsing and Querying, the analyst has the necessary ammunition to perform Knowledge Discovery . The analyst can follow his train of thought as he browses and queries to possibly discover unexpected relationships and links between entities at various levels in an indirect manner. Automatically uncovering such hidden related entities facilitates addition of new and meaningful entities and relationships to the analyst’s assessment tasks. JIVA

39. Wireless Application of Semantic Metadata and Automatic Content Enrichment  Clicking on the link for Cisco Analyst Calls displays a listing sorted by date. Semantic filtering uses just the right metadata to meet screen and other constrains. E.g., Analyst Call focuses on the source and analyst name or company. The icon denote additional metadata, such as “Strong Buy” by H&Q Analyst. MyStocks News Sports Music MyMedia    $  My Stocks CSCO NT IBM Market CSCO Analyst Call Conf Call Earnings    11/08 ON24 Payne 11/07 ON24 H&Q  11/06 CBS Langlesis CSCO Analysis

40. Scene Description Tree Retrieve Scene Description Track “ NSF Playoff” Node Enhanced XML Description MPEG-2/4/7 Enhanced Digital Cable Video MPEG Encoder MPEG Decoder Node = AVO Object Voqutte/Taalee Semantic Engine Produced by: Fox Sports Creation Date: 12/05/2000 League: NFL Teams: Seattle Seahawks, Atlanta Falcons Players: John Kitna Coaches: Mike Holmgren, Dan Reeves Location: Atlanta Object Content Information (OCI) Metadata-rich Value-added Node Create Scene Description Tree  GREAT USER EXPERIENCE Metadata’s role in emerging iTV infrastructure Channel sales through Video Server Vendors, Video App Servers, and Broadcasters License metadata decoder and semantic applications to device makers “ NSF Playoff”

41. Metadata for Automatic Content Enrichment Interactive Television This segment has embedded or referenced metadata that is used by personalization application to show only the stocks that user is interested in. This screen is customizable with interactivity feature using metadata such as whether there is a new Conference Call video on CSCO. Part of the screen can be automatically customized to show conference call specific information– including transcript, participation, etc. all of which are relevant metadata Conference Call itself can have embedded metadata to support personalization and interactivity.

42. Future Multimodal interfaces Multimodal semantics Multivalent Semantics

43. Metadata Usage: Keyword, Attribute and Content Based Access The VisualHarness system at LSDIS/UGA