Your SlideShare is downloading. ×
0
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Big Data and the Semantic Web: Challenges and Opportunities
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Big Data and the Semantic Web: Challenges and Opportunities

3,047

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,047
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
118
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaBig Data and the Semantic Web:Challenges and OpportunitiesSrinath SrinivasaOpen Systems LaboratoryIIIT Bangalorehttp://osl.iiitb.ac.in/sri@iiitb.ac.in
  • 2. Big Data Tech Conclave, 26—27 April 2013Bangalore, Indiahttp://www.bda2013.net/
  • 3. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaOSL ReleasesTopical Anchors: Given a list of noun phrases, identify a semantic topic for these terms.Powered by Wikipedia co­occurrence graph hosted by AgamaWeb APIs enable use of Topical Anchors in third party applications 
  • 4. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaOSL ReleasesTopic Expansion: Given aterm, expands it intosemantically relevant topicalclusters with differentsenses.Uses co-occurrencedatasets from Wikipedia2006 or 2011.Web APIs enable use bythird party applications
  • 5. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaOSL ReleasesAgama: A graph database for storing large undirected graphs for efficient traversal (not structure­based retrieval)Currently Agama powers a co­occurrence graph of all noun­phrases from Wikipedia articles hosted in OSL, managing 10s of millions of nodes and 100s of millions of edges 
  • 6. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaMore data beats better algorithms..meetsNo data is an island..
  • 7. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaOutline● Big Data Characteristics● Big Data Analytics● Pattern­driven and Model­driven Analytics● Big Data and the Semantic Web● Semantic Challenges● The myth of a global ontology● Convergent and divergent semantics● Semantic interoperability ● Technology Challenges● Storage, traversal and retrieval of large­scale semantic networks● Inference on Big Data● On the road ahead
  • 8. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaBig DataData that is ● Too large to be processed by conventional databases and data management techniques (Volume)● Too diverse in structure that no single data model captures all elements of the data (Variety)● Transient and/or impermanent, especially when pertaining to dynamic phenomena (Velocity)
  • 9. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaBig Data● Transaction records● Network streams● Experimental output● Social media data ● Demographic records● Citation data ● Clickstreams● Log data● Weather data ● …
  • 10. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSome Big Data Stats● YouTube users upload 48 hours of video every minute http://gigaom.com/2011/05/25/youtube­48­hours­of­video­per­minute/● Facebook data grows by 500TB daily http://www.slashgear.com/facebook­data­grows­by­over­500­tb­daily­23243691/● WalMart handles more than 1 million customer transactions every hour http://www.economist.com/node/15557443● Akamai analyzes 75 million events per day for targeted advertising http://wikibon.org/blog/taming­big­data/● 90% of data in the world today was created in the last 2 years http://wikibon.org/blog/big­data­infographics/ 
  • 11. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaBig Data AnalyticsExamine Big Data for useful (often actionable) knowledgeThe long spectrum of Big Data AnalyticsPattern identificationAssociation rule miningClassification/ClusteringRecord LinkageSecurity analyticsComplex EventProcessingOpinion miningPredictive modelingPattern drivenModel driven
  • 12. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaPattern Driven Analytics● Discovery and visualization of recurring patterns in datasets● Mostly quantitative●  Paradigms in pattern discovery:● Sampling and aggregation● Thresholding and filteringImage Source: Wikipedia
  • 13. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaPattern Driven AnalyticsSampling and Aggregation● Query based pattern aggregation● Based on an initial idea of what we are looking forHypothesisDataQuery Patterns Aggregation Presentation
  • 14. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaPattern Driven AnalyticsTresholding and Filtering● Based on sifting through the entire dataset (or a view) to look for “interesting” patterns without the context of a queryDataInterestingnesscriteriaPatterns FilteringandSegregationPresentation
  • 15. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaModel Driven AnalyticsAnalytics as a model­discovery problemWeddingImages source: WikipediaObservableDataLatentConcept
  • 16. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaModel Driven Analytics● Pattern discovery coupled with semantic modeling● Non­trivial qualitative modeling challenges● Model discovery:● Descriptive model discoveryFit a model to explain the observed data● Predictive model discoveryDiscover a model that can predict values of data elements into the future
  • 17. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaLinked DataImage source: WikipediaThe Linked DataCloud as ofSeptember 2011
  • 18. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaLinked Data● Using Semantic Web technologies to connect data elements from disparate data sources● From Web of Documents to Web of Data● Elements of Linked Data● URIs ● HTTP● Resource Description Framework (RDF)● Serialization formats (RDFa, RDF/XML, N3, Turtle, and others)
  • 19. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaBig Data and the Semantic WebBig DataSemantic WebModel DiscoveryCatalyzation andPredictive Modeling
  • 20. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaBig Data        Semantic Web● One of the main elements of the Linked Data Cloud: DBpedia is built from a Big Data resource: Wikipedia● Open Biomedical Ontology (OBO) (http://www.oboedit.org/) created from mining PubMed publications● Enterprise scale Big Data Analytics helping build organizational models, operational intelligence solutions, etc. Example: Anzo software suite by Cambridge Semantics (www.cambridgesemantics.com), Loom data management suite by Revelytix (www.revelytix.com)
  • 21. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSemantic Web       Big DataSchema.org● Collection of schemata on various topics that are recognized by major search providers and used to semantically interpret web contentSourceMap● Linked data augmented with web content and crowdsourced data used to provide details about companies like their carbon footprint, energy use, water use, etc. www.sourcemap.com OpenSteetMap● Linked data augmenting crowdsourced data on www.openstreetmap.org helped in detailed mapping of disaster scenario during the Jan 2010 Haiti earthquake (http://www.scientificamerican.com/article.cfm?id=berners­lee­linked­data)
  • 22. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaBig Data and the Semantic Web: ChallengesSemantic challenges● The myth of a global ontology● Convergent and divergent semanticsTechnology and system challenges● Characteristics of a semantic graph● Managing graph structured data
  • 23. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaThe Myth of a Global OntologySeveral “core” semantic ontologies exist:● WordNet● YAGO● OpenCyc● SUMOHowever, none of them (even automated ones) can capture all possible semantic associations and all possible perspectives on a given topic
  • 24. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaThe Myth of a Global OntologyThe open world problem● We dont know what we dont know.. ● Representation bias in big data sourcesThe neutral­but­useless perspective● Localized, utilitarian descriptions often more useful than neutral, global descriptions. Ex: Use of “zones” as a geographical element in Indian Railways● Difficult for disparate perspectives to co­exist in a single Ontology, violating design principles like Occams razor
  • 25. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaConvergent and Divergent SemanticsWikipedia article onWest BankconflictPalestine POVIsraeli POVHistorians POVUNs POVEncyclopedic Semantics
  • 26. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaConvergent and Divergent SemanticsIPLevent scheduleTraffic planningAdvertisement planningaround IPLLegal structuringaround IPLTV programmeschedulingSecurityplanning
  • 27. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSemantic Interoperability● Binary predicates like RDF may not capture complete semantics of the associationBut it is too difficult to work with higher­order predicates● Semantic queries are characterized by contextual relevance and default assumptions● Linked Data can be useful primarily within the context of a modelModel­building from predicates as complex a problem as identifying predicates from data
  • 28. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSemantic Challenges: Summary● Hard to distinguish data from noise without a modelEspecially hard when we are using data to help build a model!● There may not be a single global model explaining the data● Model construction as challenging, if not more challenging, as predicate mining● No clarity on the underlying processes that aid in knowledge aggregationKnowledge aggregation happens differently depending on the kind of knowledge being aggregated (encyclopedic versus operational knowledge) 
  • 29. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaTech ChallengesStoring Big Semantic Data● Semantic data not amenable to physical access coherence to be efficiently stored in relational tables● Logical proximity of triples, more important than physical proximity● Read/Write storage models change logical proximity● RDF graphs tend to be extremely dense and/or clustered● Need efficient methods of graph storage and retrieval 
  • 30. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSemantic store for Big Data● Databases optimized to store and retrieve interrelated sets of triples of the form (subject, predicate, object) ● Query models based on answering graph queries (usually in SPARQL) rather than SQL queries●  Main design criteria: storage and read­ahead policies of triples based on their logical proximity rather than physical proximity in order to enable Bulk Synchronous Parallel (BSP) processing
  • 31. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSemantic store for Big DataAllegroGraph  (http://www.franz.com/agraph/allegrograph/)● NoSQL Graph based native storage for RDF triples● ACID compliant● Interfaces with Solr for free text indexing ● Triple and text level indexing● MongoDB integration● RDFS++ Reasoning with dynamic materialization ● SPARQL queries on named graphs and Prolog based inferencing engine
  • 32. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSemantic store for Big DataSesame http://www.openrdf.org/●  Open source Java framework for parsing, storing, querying and inferencing over RDF data ● Collections of RDF triples can be manipulated in memory using a graph data model● Compliant with SPARQL 1.1 protocol recommendation ● Provides two levels of APIs: SAIL (Storage and Inference Layer) for low level RDF processing and Repository layer for programmatic interfacing with Sesame
  • 33. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSemantic store for Big DataMulgara http://www.mulgara.org/ ● Native storage model for RDF● Supports multiple models (databases) per server● ACID transactions and concurrency support ● Copy­on­write­ cache semantics● Full­text search and support for data types● Primarily useful as a repository – no evidence of support for logical inferences over RDF 
  • 34. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaSemantic store for Big DataOther examples:● InfiniteGraph from Objectivity http://www.objectivity.com/● Big­Data http://www.bigdata.com/bigdata/blog/ – A high scale­out storage and computing engine● Agama https://github.com/arrac/agama/wiki/Agama – Storage, search and traversal support (Ruby library) for very large graphs ● Neo4j http://www.neo4j.org/ – Embedded, disk­based transactional graph database written in Java 
  • 35. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaLogical inference over Big Data● Problem: Find factual answers to specific questions by reasoning over large­scale data.  ● Performing extremely large­scale deductions over large semantic datasets in interactive response time ● Need to contend with potentially inconsistent predicates, incomplete or missing values and default assumptions● Varieties of inference over datasets● Deduction● Induction● Abduction● Statistical inference
  • 36. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaLogical inference over Big DataCommon approaches for scalable inferencing:● Horn clause inferencing● Variants of random walks on knowledge graphs● Distributed MCMC (Markov Chain Monte Carlo) methods
  • 37. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaHorn ClausesHorn clauses are predicates of the form:atomic sentence with no negation and a single consequentHorn clause knowledge bases can be resolved using “backward chaining” starting from the consequent and building a tree of antecedents until they are grounded in factsHorn clause resolution can be scaled over large datasets by parallelizing resolutions using MapReduce  p1∧p2∧...∧pn →u
  • 38. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaRandom Walks on Big DataRandom walks on RDF graphs as a means of:● Belief materialization● Soft inferencea c ed fbR RRRAssuming transitivity of R
  • 39. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaRandom Walks on Big DataLarge scale graph processing solutions for scaling random walks over Big Data: ● Apache Giraph http://giraph.apache.org/ ● Pregel [Malewicz et al., 2010]● Grappa http://www.cs.washington.edu/node/4217/ 
  • 40. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaMCMCA “generic” problem solving method based on local sampling, useful for soft inferences on semantic dataTime homogeneous Markov Chain:
  • 41. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaMCMCA homogeneous Markov chain can be represented as a set of “states” and “transition probabilities” across statesGiven an initial “prior” probability distribution across states           the “stationary distribution” or “equilibrium condition” is defined as: 
  • 42. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaMCMCMarkov Chain Monte CarloGiven a state space S and an “equilibrium” distribution       choose a sample s of the state space S so that a Markov chain on s results in      as the stationary distributionMCMC for logical inferenceFor a logical inference problem, the equilibrium condition would be of the form [0,1]m defined over a set of m predicatesExample Sampling algorithms for MCMCGibbs Sampling http://en.wikipedia.org/wiki/Gibbs_sampling Metropolis­Hastings algorithm http://en.wikipedia.org/wiki/Metropolis%E2%80%93Hastings_algorithm 
  • 43. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaScaling MCMC for Big DataDistributed MCMCSeveral models are explored for distributing MCMC computations over large datasets making them amenable to diffusing computations. Some examples include: [Murray 2010; Singh et al 2011]Distributional models for MCMC beyond the scope of this talk.. 
  • 44. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaOn the road ahead..Some promising directions for Big Data and Semantics● Diffusion models for large scale inference● Cognitive models for semantics over large scale data● Model­based reasoning and reasoning across models● Soft (probabilistic) inferences, confidence measures, relevance feedback● Continuous learning over Big Data 
  • 45. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaThank You!
  • 46. Big Data Tech Conclave, 26—27 April 2013Bangalore, IndiaReferences● Neal Madras. Introduction to Markov Chain Monte Carlo. http://www.cs.cornell.edu/selman/cs475/lectures/intro­mcmc­lukas.pdf ● Grzegorz Malewicz, Matthew H. Austern, Aart J.C Bik, James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski. 2010. Pregel: a system for large­scale graph processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data (SIGMOD 10). ACM, New York, NY, USA, 135­146. DOI=10.1145/1807167.1807184 http://doi.acm.org/10.1145/1807167.1807184● Ni Lao, Tom Mitchell, and William W. Cohen. 2011. Random walk inference and learning in a large scale knowledge base. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 11). Association for Computational Linguistics, Stroudsburg, PA, USA, 529­539. ● Lawrence Murray, Distributed Markov Chain Monte Carlo. Proceedings of NIPS 2010 Workshop on Learning on Cores, Clusters and Clouds. http://lccc.eecs.berkeley.edu/ ● Stefan Schoenmackers, Oren Etzioni, and Daniel S. Weld. 2008. Scaling textual inference to the web. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 08). Association for Computational Linguistics, Stroudsburg, PA, USA, 79­88.● Stefan Schoenmackers, Oren Etzioni, Daniel S. Weld, and Jesse Davis. 2010. Learning first­order Horn clauses from web text. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP 10). Association for Computational Linguistics, Stroudsburg, PA, USA, 1088­1098.● Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2011. Large­scale cross­document coreference using distributed inference and hierarchical models. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies ­ Volume 1 (HLT 11), Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, USA, 793­803.   

×