2013 10-03-semantics-meetup-s buxton-mark_logic_pub

1. Slide 1 Copyright © 2013 MarkLogic® Corporation. All rights reserved. © COPYRIGHT 2013 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Stephen Buxton Lotico meetup, NYC October 3rd, 2013 Semantics is the future of Search is the future of Semantics

2. Slide 2 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Semantics is the future of Search is the future of Semantics  Introduction: what does the title mean?  MarkLogic: who's that?  Imagine a World with Search AND Semantics  Rich Search Applications … plus Semantics  Semantics Applications … plus Search  Bringing it all together: Use Cases  Where To Next?

4. Slide 4 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Two Hemispheres, One Brain Triples:  Highly structured  Atomic  Do one thing well XML:  Flexible structure  Rich documents  Rich applications

6. Slide 6 Copyright © 2013 MarkLogic® Corporation. All rights reserved. The MarkLogic Advantage Search & Query ACID Transactions High Availability / Disaster Recovery Government-grade Security Semantics Elasticity Cloud Deployment Hadoop for Storage & Compute The Only Enterprise NoSQL Database

7. Slide 7 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Real Value From Big Data Make The World More Secure Provide Access To Valuable Information Create New Revenue Streams Gain Insights to Increase Market Share Reduce Bottom Line Expense

9. Slide 9 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Dynamic Semantic Publishing BBC Sports  Size and Complexity:  # of athletes  # of teams  # of assets (match reports, statistics, etc.)  # of relations (facts)  Rich user experience  See information in context  Personalize content  Easy navigation  Intelligently serve ads (outside of UK)  Manageable  Static pages? Too many, too fast-changing  Limited number of journalists  Automate as much as possible The Challenge Goals

12. Slide 12 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Dynamic Semantic Publishing: A Solution  Store, manage documents  Stories  Blogs  Feeds  Profiles  Store, manage values  Statistics  Full-Text search  Performance, scalability  Robustness  Metadata about documents  Tagged by journalists  Added (semi-)automatically  Inferred  Facts reported by journalists  Real-world facts from the Open Data Web Document Database Triple Store

13. Slide 13 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Dynamic Semantic Publishing: A Solution  At query time, dynamically aggregate stories, blogs, feeds, images, profiles, results, statistics, videos for a particular concept such as "West Ham". (See Jem Rayfield, BBC, http://bbc.in/I1NdkB)  we are not publishing pages, but publishing content as assets which are then organized by the metadata dynamically into pages (John O'Donovan, BBC and PA) Document Database Triple Store

14. Slide 14 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Dynamic Semantic Publishing: A Solution  At query time, dynamically aggregate stories, blogs, feeds, images, profiles, results, statistics, videos for a particular concept such as "Chelsea". (See Jem Rayfield, BBC, http://bbc.in/I1NdkB)  we are not publishing pages, but publishing content as assets which are then organized by the metadata dynamically into pages (John O'Donovan, BBC and PA) Document Database Triple Storewith

16. Slide 16 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Why Semantic Technologies?  Triples are atomic – easy to create, manage, combine  Semantic Web shares data as triples  A natural choice for metadata and real-world facts  .. and facts embedded in a document  Standards encourage tools and sharing  Graph model – easy to follow links  Ontologies – infer new facts Because …

17. Slide 17 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Why Semantics and Search?  Many use cases need documents, triples, and data  One database means a simple, efficient, powerful architecture  Combination queries – query documents, triples, data in a single query – open up new possibilities Because …

19. Slide 20 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Rich Search Applications .. Made Richer Rich Search Application: • 60 million docs indexed • Links to email messages • Snippets about e-mail • Sender data • Facets • Search trends

22. Slide 23 Copyright © 2013 MarkLogic® Corporation. All rights reserved. The World of Triples Linked Open Data (Free semantic facts available to anyone) Proprietary Semantic Facts (Facts and Taxonomies in your organization) SemanticWorld DocumentWorld Facts from Free-Flowing Text (Derived from semantic enrichment) MarkLogic Facts in Documents (Part of metadata or added with authoring tools)

26. Slide 27 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Semantics + Search [2] How do you deal with …  Provenance – what's the source? And its source? And so on ..  Reification – what's my confidence? Author? Date?  Bi-Temporal – two date ranges Answer questions like:  Which countries did Nixon visit?  .. before 1974?  .. only show me answers where I have at least 80% confidence  .. and the source is AP Newswire OR BBC

27. Slide 28 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Triples and Documents Documents can contain triples <article> <meta> <title>Man bites dog</title> <sem:triple> <sem:subject>http://example.org/news/42</sem:subject> <sem:predicate>http://example.org/published</sem:predicate> <sem:object>2013-09-10</sem:object> </sem:triple> …

28. Slide 29 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Triples and Documents Triples can be annotated in documents <source>AP Newswire</source> <sem:triple date="1972-02-21" confidence="100"> <sem:subject>http://example.org/news/Nixon</sem:subject> <sem:predicate>http://example.org/wentTo</sem:predicate> <sem:object>China</sem:object> </sem:triple> …

29. Slide 30 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Triples and Documents Documents can contain triples <article> <meta> <title>Man bites dog</title> <sem:triple> <sem:subject>http://example.org/news/42</sem:subject> <sem:predicate>http://example.org/published</sem:predicate> <sem:object>2013-09-10</sem:object> </sem:triple> … Triples can be annotated in documents <source>AP Newswire</source> <sem:triple date="1972-02-21" confidence="100"> <sem:subject>http://example.org/news/Nixon</sem:subject> <sem:predicate>http://example.org/wentTo</sem:predicate> <sem:object>China</sem:object> </sem:triple> …

30. import module namespace sem = "http://marklogic.com/semantics" at "/MarkLogic/semantics.xqy"; sem:sparql(' SELECT ?country WHERE { <http://example.org/news/Nixon> <http://example.org/wentTo> ?country } ', (), (), cts:and-query( ( cts:path-range-query( "//sem:triple/@confidence", ">", 80) , cts:path-range-query( "//sem:triple/@date", "<", xs:date("1974-01-01")), cts:or-query( ( cts:element-value-query( xs:QName("source"), "AP Newswire" ), cts:element-value-query( xs:QName("source"), "BBC" ) ) ) ) ) ) Which countries did Nixon visit?  .. before 1974?  .. only show me answers where I have at least 80% confidence  .. and the source is AP Newswire OR BBC

31. Slide 32 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Semantics + Search [3]  Combination queries – simple, easy, efficient, powerful  Query across triples, documents, data  Leverage context Answer questions like:  I have a partial match on a license plate number – a witness to a crime says it starts with ABC  Are there any incident reports around the time of the crime, around the location of the crime, that mention a black van?  If yes, show me the names and addresses of the license holders

32. Slide 33 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Combination Query: Example <SAR> <title>Suspicious vehicle…Suspicious vehicle near stadium <date> <type> <threat> 2012-11-12Z observation/surveillance <type>suspicious activity <category>suspicious vehicle <location> <lat>37.497075 <long>-122.363319 <subject>IRIID <subject>IRIID <predicate> <predicate> isa value <triple> <triple> <object>license-plate <object>ABC 123 <description>A black van…A black van with license plate ABC 123 was observed parked behind the airport sign… </title> </date> </type> </type> </category> </threat> </lat> </long> </location> </subject> </subject> </predicate> </predicate> </object> </object> </description> </SAR> </triple> </triple> XML document – a SAR report

33. Slide 34 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Combination Query: Example <SAR> <title> Suspicious vehicle… <date> 2012-11-12Z <type> <threat> suspicious activity <category> suspicious vehicle <location> <lat> 37.497075 <long> -122.363319 <description> A black van… <subject> <subject> <predicate> <object> IRIID IRIID isa value license-plate ABC 123<predicate> <object> observation/surveillance <type> <triple> <triple>

34. Slide 35 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Combination Queries  SPARQL -centric:  SPARQL with XQuery built-in functions  SPARQL with a search argument  SPARQL with variable bindings  SPARQL with forest-ids  XQuery -centric: Inside an XQuery program:  sem:sparql( sparql query, search criteria )  cts:triples( subject, predicate, object , search criteria)  cts:triple-range-query( subject, predicate, object , [=,<,>] )

36. Slide 37 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Semantics and Search – Use Cases  Publishers and media organizations look to use the combined Enterprise NoSQL database and triple store as a way to speed implementation of Dynamic Semantic Publishing.  Financial institutions query triples in the context of a document to manage Reference Data and understand risk.  Intelligence, law enforcement, fraud investigators and analysts are able to discover connections and patterns in facts and documents to find bad guys faster.  Procurement agents are creating decision support tools to rationalize purchasing decisions, vendors and bid management.  Pharmaceutical companies and governing organizations are using facts and documents together to assess risks and to decide which drug trials to invest in.

38. Slide 39 Copyright © 2013 MarkLogic® Corporation. All rights reserved. Search: Understand, Discover, Make decisions Search Fetch documents Extract relevant facts Analyze Old Search Fetch facts, data, and documents in context Analyze and annotate Fetch supporting facts, data, and documents As needed New

41. http://www.marklogic.com/summit-series/