Semantic Technologies for Big Data

  • 8,102 views
Uploaded on

 

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,102
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
282
Comments
0
Likes
7

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Semantic Technologies for Big Data Marin Dimitrov (Ontotext) XML Amsterdam 2012
  • 2. XML Amsterdam 2012 Semantic Technologies for Big Data Sep 2012 #2
  • 3. About Ontotext• Provides products and services for creating, managing and exploiting semantic data – Founded in 2000 – Offices in Bulgaria, USA and UK• Major clients and industries – Media & Publishing (BBC, Press Association) – HCLS (AstraZeneca, UCB) – Cultural Heritage (The British Museum, The National Archives, Polish National Museum, Dutch Public Library) – Defense and Homeland Security Semantic Technologies for Big Data Sep 2012 #3
  • 4. Outline• Semantic Technologies for the Enterprise• Semantic Technologies for Big Data• Success stories Semantic Technologies for Big Data Sep 2012 #4
  • 5. SEMANTIC TECHNOLOGIES FOR THEENTERPRISE Semantic Technologies for Big Data Sep 2012 #5
  • 6. The need for a smarter Web• "The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation.“ (Tim Berners-Lee, 2001)• “PricewaterhouseCoopers believes a Web of data will develop that fully augments the document Web of today. You’ll be able to find pieces of data sets from different places, aggregate them without warehousing, and analyze them in a more straightforward, powerful way than you can now.” (PWC, May 2009) Semantic Technologies for Big Data Sep 2012 #6
  • 7. Linked Data• Linked Data is a set of principles that allows publishing, querying and consumption of RDF data, distributed across different servers• Design principles – Use unambiguous identifiers for resources (URIs) – Use HTTP URIs (dereference-able) – Provide useful information for URI lookups – Interlink resources Semantic Technologies for Big Data Sep 2012 #7
  • 8. The Semantic Web timeline RDF RDF 2 DAML+OIL OWL OWL 2 SPARQL SPARQL 1.1 RIF RDFa SAWSDL LOD SKOS HCLS SSN RDB2RDF PIL GLD LDP1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 Semantic Technologies for Big Data Sep 2012 #8
  • 9. Enterprise Information Management Challenges• Many disparate data sources and data silos• Many point-to-point interfaces• Data sources with similar/inconsistent information• Complex data integration processes inadequate for changing business requirements• Most of the knowledge is hidden in texts• Difficult to integrate & analyse structured data and text Semantic Technologies for Big Data Sep 2012 #9
  • 10. Semantic Web and Linked Data Opportunities for the Enterprise• Simplify the information integration processes – Flexible, easy to evolve data model – Bottom-up / incremental integration – Efficiently integrate structured and unstructured data• Provide an enterprise metadata layer – Unified metadata vocabulary for the enterprise – Align the legacy data silos – Improve the information sharing and reuse Semantic Technologies for Big Data Sep 2012 #10
  • 11. Semantic Web and Linked Data Opportunities for the Enterprise (2)• Discovery and enrichment of information – Interlink people, organisations, events, etc. – Enrich enterprise content with structured annotations – Discover implicit links and relationships• Unified access to information within the enterprise – Simplified infrastructure based on open web standards• Information interchange across a value chain – Easy publishing and consumption of Linked Data• Augments existing IT assets and technologies – No need for disruptive replacement Semantic Technologies for Big Data Sep 2012 #11
  • 12. XML and RDF: friends or foes• Complement each other – XML best for content, structure and interchange format – RDF for metadata layer and semantics• Typical use case – Many XML content data sources • Content stored in an XML store (XQuery and XSLT) – Structured data sources & external Linked Data • RDF-ized and stored in an RDF store (SPARQL) – Metadata extracted from content • stored in an RDF store (SPARQL) • semantic search and metadata driven content delivery Semantic Technologies for Big Data Sep 2012 #12
  • 13. BBC Sports (c) BBCSemantic Technologies for Big Data Sep 2012 #13
  • 14. Added value of RDF• Explicit semantics – Intended meaning of entities and relations• Global identifiers (URIs)• Simple and flexible graph-based data model• Easier data mapping & integration – Bottom-up / incremental data integration with owl:sameAs• Inference of implicit information• Working with distributed information – Linked Data, federated SPARQL Semantic Technologies for Big Data Sep 2012 #14
  • 15. Added value of RDF• Descriptive / agile schema – Open World Assumption, don’t restrict predicates – Generated dynamically from data• Queries based on meaning – Not depending on structure / order of statements• Data and queries may use different vocabularies• Exploratory queries• Choice of OWL2 profiles – Tradeoff features vs performance – New profiles may emerge in the future Semantic Technologies for Big Data Sep 2012 #15
  • 16. SEMANTIC TECHNOLOGIES FOR BIGDATA Semantic Technologies for Big Data Sep 2012 #16
  • 17. The three V’s of Big Data• Velocity – Streaming, sensor, real-time data – Solution: distributed processing & storage – Semantic challenge: stream reasoning• Volume – Petabytes of data – Solution: distributed processing & storage – Semantic challenge: distributed reasoning & querying• Variety – Structured, semi-structured and unstructured data – Semantic Technologies (RDF) are a good fit Semantic Technologies for Big Data Sep 2012 #17
  • 18. Types of Big Data (NIST)• Type 1 – Velocity (-), Volume (-), Variety (+) – Perfect fit for Semantic Technologies• Type 2 – Velocity and/or Volume, Variety (-) – Only horizontal scalability required, traditional approaches are a good enough fit• Type 3 – All V’s – Semantic Technologies not a good fit yet, but moving in that direction Semantic Technologies for Big Data Sep 2012 #18
  • 19. Semantic Technologies for Volume and Velocity• Promising ongoing research• Distributed inference with Hadoop/Storm• Stream reasoning – Continuous queries – Continuous (dynamic) semantics• SPARQL to Pig translation• Distributed RDF stores on top of NoSQL• C-SPARQL, EP-SPARQL, CQELS Semantic Technologies for Big Data Sep 2012 #19
  • 20. Linked Open Data Cloud (Sep 2011) (c) Cyganiak & Jentzsch Semantic Technologies for Big Data Sep 2012 #20
  • 21. From Big Linked Data to Linked Big Data• Big Linked Data – Big Data approach adopted by the Linked Data community • In particular handling Volume and Velocity – Exponential growth of Linked Data in the last 5 years• Linked Big Data – Linked Data approach adopted by the Big Data community – RDF data model for Variety – Enrich Big Data with metadata and semantics – more powerful analytics on top of it – Interlink Big Data sets – Simplify data access and data integration Semantic Technologies for Big Data Sep 2012 #21
  • 22. SUCCESS STORIES Semantic Technologies for Big Data Sep 2012 #22
  • 23. Typical Use Cases for Linked Data and Semantic Technologies• Publish / consume Linked Data across enterprises – Linked Data is not necessarily free data – Facilitate data interchange within the value chain• Information integration within the enterprise – Integrated asset management / align data silos – Master Data Management• Knowledge discovery and semantic search – Integrate structured and unstructured data – Enrich and interlink information – Semantic search and exploration of information Semantic Technologies for Big Data Sep 2012 #23
  • 24. Semantic Information Integration (Ontotext) Semantic Technologies for Big Data Sep 2012 #24
  • 25. The National Archives (Ontotext)• Challenge – Large archive of various UK Government websites since 1997 – Lots of duplicated information & documents – Inefficient search & navigation• Semantic Knowledge Base project goals – Integrate multiple data sources – Extract information & metadata from archived documents – Interlink the web archive with data.gov.uk and LOD data – Advanced search & navigation of the archive Semantic Technologies for Big Data Sep 2012 #25
  • 26. The National Archives (Ontotext) Front Ends: Semantic Search O1 SPARQL A 3rd party C O2 Ontology graph B D Editors exploration O3 Data Trans- formation and Semantic Repository Integration Semantic Annotation SKB Ontologies Factual Knowledge (TNA data, LOD, data.gov.uk) Identity Semantic annotations ResolutionAnnotation Process(GATE Teamware) Semantic Index Semantic Technologies for Big Data Sep 2012 #26
  • 27. The National Archives (Ontotext)• The numbers – 2.5 billion input files – 40TB compressed archive data – 10 billion RDF triples stored in OWLIM – 33,000 EC2 hours used on AWS – Dynamic EC2 cluster (180 instances average, 500 max)• Major challenges – Complex pre-processing of documents – De-duplication of information & documents – EC2/RRS performance & reliability Semantic Technologies for Big Data Sep 2012 #27
  • 28. Dutch Public Library (Ontotext + Dayon)• Challenge – Many disparate data sources, inefficient search• Goals – Data integration – Automated metadata generation – Open search platform• Numbers – 500 heterogeneous data sources – 40 million cultural heritage artifacts to be describes – 6-8 billion triples to be stored into the knowledge base Semantic Technologies for Big Data Sep 2012 #28
  • 29. Linked Life Data (Ontotext)• Challenge – Disparate, heterogeneous and unaligned data silos lock valuable biomedical information• Goals – Semantic warehouse integrating and interlinking public biomedical data sources – Interactive discovery and exploration• Numbers – 25+ heterogeneous biomedical data sources integrated – 1 billion entities described – 5.5 billion RDF triples Semantic Technologies for Big Data Sep 2012 #29
  • 30. Linked Life Data (Ontotext) Semantic Technologies for Big Data Sep 2012 #30
  • 31. Linked Life Data-as-a-Service (Ontotext)• More data sources• Large scale text mining over the LOD cloud• Adapted for specific use cases• UCB use case – 2 billion entities described – 11 billion RDF triples Semantic Technologies for Big Data Sep 2012 #31
  • 32. Dynamic Semantic Publishing (Ontotext)• Challenge – Difficult & slow to aggregate content from various sources• Goals – Metadata generation for news (semantic annotation) – Interlink & categorize content – Metadata driven web pages• Numbers – Nearly real-time processing & annotation required – Tens of millions (SPARQL) queries to the knowledge base per day Semantic Technologies for Big Data Sep 2012 #32
  • 33. Trillion RDF triples (Franz Inc.)• Use case – Use RDF for the customer management database of a telecom• Challenge – 4,000 triples per customer, more than a trillion for the whole customer base• Numbers – 1 trillion triples stored in AllegroGraph by Franz Inc • Hardware requirements undisclosed • The 310 billion triple result used 8-CPU system with 2TB RAM Semantic Technologies for Big Data Sep 2012 #33
  • 34. uRiKA (Cray/YarcData)• Big Data appliance for graph analytics – Based on the Threadstormtm architecture – Up to 8K processors, 512TB RAM, 350TB/hr IO throughput• In-memory RDF database• SPARQL 1.0 engine Semantic Technologies for Big Data Sep 2012 #34 (c) YarcData
  • 35. TAKEAWAYS Semantic Technologies for Big Data Sep 2012 #35
  • 36. Semantic Technologies for Big Data• Rich ecosystem of Semantic Technologies since 1999• Strong Enterprise focus in the last 5 years• Semantic Technologies provide opportunity for reducing the cost and complexity of data integration• Common metadata layer for the enterprise• More powerful ways to find and explore information• RDF complements XML within the enterprise• Semantic Technologies are a good fit for Big Data’s Variety Semantic Technologies for Big Data Sep 2012 #36
  • 37. Semantic Technologies for Big Data• Velocity and Volume still challenging for Semantic Technologies, but lots of progress in that direction• Linked Data will grow into Big Linked Data, but Big Data will also benefit from evolving into Linked Big Data• Interesting success stories for Semantic Technologies in Big Data scenarios Semantic Technologies for Big Data Sep 2012 #37
  • 38. THANK YOU! Semantic Technologies for Big Data Sep 2012 #38