The Social Data Web

1. transparency, collaboration and information sharing solution architecture tools and techniques using the social data web george thomas, 1105 ea2009

2. agenda <ul><li>An overview of Web Oriented Architecture (WOA) design principles that have made the Web the most successful distributed computing platform ever created will be given. </li></ul><ul><li>Technologies for exposing raw data and publishing semantically enriched structured data for persistence and syndication on the Web as public records will be described. </li></ul><ul><li>Technologies that enable interoperability across these published assets and currently disparate data sources to achieve low cost, large scale data federation will be described. </li></ul><ul><li>Widgets and services that consume and transform this data for interactive and integration purposes will be discussed in the context of different stakeholder views. </li></ul><ul><li>A Web-scale approach to Business Intelligence leveraging Cloud Computing approaches to data archive analysis will be described. </li></ul><ul><li>Finally, the applicability of the proposed solution architecture to the Federal Segment Architecture Methodology and tools like Visualization to Understand Expenditures in IT will be discussed. </li></ul>

4. Web Oriented Architecture (WOA) <ul><li>REpresentational State Transfer (REST) </li></ul><ul><ul><li>The architectural style of the World Wide Web </li></ul></ul><ul><ul><li>aka Resource Oriented Architecture (ROA) </li></ul></ul><ul><li>hyperlinks dereference (information) resource representations </li></ul><ul><ul><li>HTTP URI's and content negotiation </li></ul></ul><ul><ul><ul><li>user agent prefers .htm, .xml, .rdf, .etc </li></ul></ul></ul><ul><li>statefulness </li></ul><ul><ul><li>servers maintain resource state, clients maintain application state </li></ul></ul><ul><li>RESTful Web services </li></ul><ul><ul><li>HTTP uniform interface </li></ul></ul><ul><ul><ul><li>CRUD analog to HTTP PUT/GET/POST/DELETE </li></ul></ul></ul><ul><ul><li>contrast to Remote Procedure Call (RPC) style Web services </li></ul></ul><ul><ul><ul><li>SOAP/WSDL, you design the methods to invoke </li></ul></ul></ul><ul><li>global visibility (the Web) and persistence (permalinks) </li></ul><ul><ul><li>caching, crawling, indexing </li></ul></ul>

6. XForms - human data capture <ul><li>Orbeon server side XForms engine, Ajax browser GUI's </li></ul><ul><ul><li>catalog and builder apps </li></ul></ul><ul><ul><li>create new XSD bound forms </li></ul></ul><ul><ul><li>populate, persist, search </li></ul></ul><ul><ul><li>Tomcat and eXist </li></ul></ul><ul><ul><li>off-line capability </li></ul></ul><ul><ul><li>transformation pipeline </li></ul></ul>

7. Atom Publishing Protocol (APP) <ul><li>automated invocation of the RESTful Web service </li></ul><ul><ul><li>HTTP PUT/POST the spreadsheet or XML instance doc </li></ul></ul><ul><ul><ul><li>to atomserver.codehaus.org </li></ul></ul></ul><ul><li>where else is APP used? </li></ul><ul><ul><li>Google Data API's, Microsoft Live Framework </li></ul></ul>

8. Atom Syndication Format <ul><ul><li>transform XForm or APP captured info into XHTML+RDFa </li></ul></ul><ul><ul><li>(permalinked) public recordset in feed entry <content> </li></ul></ul>

9. the london-gazette.co.uk

10. london-gazette.co.uk/listing small, discreet, component ontology/data-domain-metamodels

11. web page = web service

12. RDFa enabled 'deep link' discovery <ul><ul><li>Rich Snippets from Google </li></ul></ul><ul><ul><li>SearchMonkey from Yahoo </li></ul></ul>

14. goal: federated dataset correlation <ul><li>graph based dynamic schema evolution across silos </li></ul><ul><ul><li>centralization/normalization not required (or realistic/practical!) </li></ul></ul>

15. Web as DB - Web API <ul><ul><li>Linking Open (Government) Data (LOD) </li></ul></ul><ul><ul><li>SPARQL endpoints </li></ul></ul>linkeddata.org

16. browse: from web of docs to web of data

17. http://data.linkedmdb.org/ page /actor/10 <ul><li>content negotiation, user agent prefers; </li></ul><ul><ul><li>human (html) or machine (rdf/xml) readable </li></ul></ul>RDF/N3

18. http://data.linkedmdb.org/ page /actor/10 <ul><li>now at the bottom of the same page/actor/10 </li></ul><ul><ul><li>triple is Subject (S) Predicate (P) Object (O) </li></ul></ul><ul><ul><ul><li>10 (S) vocabulary: property (P) <object> (O) </li></ul></ul></ul><ul><ul><li>properties link to other dataset instances </li></ul></ul><ul><ul><ul><li>that use different datatype definitions </li></ul></ul></ul><ul><ul><li>note D2R app, expose RDB as RDF, SPARQL to SQL </li></ul></ul>

19. http://data.linkedmdb.org/ data /actor/10 <ul><li><subject> has predicate {space} object1 , objectN ; repeat until . </li></ul><ul><li><http://data.linkedmdb.org/resource/actor/10> </li></ul><ul><li>foaf:page <http://www.freebase.com/view/guid/9202a8c04000641f800000000007821e> , <http://www.imdb.com/name/nm0000564/> ; </li></ul><ul><li>owl:sameAs <http://mpii.de/yago/resource/Peter_O%27Toole> , <http://dbpedia.org/resource/Peter_O%27Toole> ; </li></ul><ul><li>rdf:type movie:actor , </li></ul><ul><li>foaf:Person . </li></ul><ul><li>this is an 'N3' RDF serialization, instead of RDF/XML (or others) </li></ul><ul><li>some properties have RESTful SPARQL queries as <objects> </li></ul><ul><li>foaf:person rdfs:seeAlso <http://data.linkedmdb.org/sparql?query=DESCRIBE+<http://xmlns.com/foaf/0.1/Person> </li></ul>

20. Web based SPARQL query builder http://dbpedia.org/ is powered by http://www.openlinksw.com 'Virtuoso' that provides a 'SPARQL endpoint' (DRM 'query point')

21. creates dbpedia.org query <ul><li>use response data in next query </li></ul>

22. authoritative metadata - provided tags!! <ul><ul><li>using standardized datatype and property specifications </li></ul></ul><ul><ul><li>ontologies emerges from social folksonomy </li></ul></ul>http://commontag.org

24. indexing/searching the Data Web

25. aggregation and live data reporting http://sig.ma

26. many to many set visualization http://mqlx.com/~david/parallax interface used to aggregate data across multiple (data) 'bases' on http://freebase.com

27. ad-hoc analyst/end-user 'meshups'

28. schema/bizmo/federal_enterprise <ul><li>bizmo.freebase.com = OMG BMM + CPIC (+SOA...) </li></ul><ul><ul><li>Obama is an instance of the Federal Enterprise type </li></ul></ul><ul><li>Federal Enterprise (S) Fed Ent Goal (P) Goal (O) </li></ul>

29. /rdf/bizmo.federal_enterprise (excerpt) <ul><li>(W3C/FBase) <subject/topic> <predicate/property> <object/topic> </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo. federal_enterprise > <http://rdf.freebase.com/ns/type.object. name > "Federal Enterprise"@en. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo. federal_enterprise > <http://rdf.freebase.com/ns/freebase.type_profile. instance_count > "1" ^^<http://www.w3.org/2001/ XMLSchema#long >. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo. federal_enterprise> <http://rdf.freebase.com/ns/type.type. instance> <http://rdf.freebase.com/ns/guid.9202a8c04000641f800000000c61962c>. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo.federal_enterprise> <http://rdf.freebase.com/ns/type.type. properties > <http://rdf.freebase.com/ns/base.bizmo.federal_enterprise. federal_enterprise_strategy >. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo.federal_enterprise> <http://rdf.freebase.com/ns/type.type. properties > <http://rdf.freebase.com/ns/base.bizmo.federal_enterprise. federal_enterprise_tactic >. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo.federal_enterprise> <http://rdf.freebase.com/ns/type.type. properties > <http://rdf.freebase.com/ns/base.bizmo.federal_enterprise. federal_enterprise_directive >. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo.federal_enterprise> <http://rdf.freebase.com/ns/type.type. properties > <http://rdf.freebase.com/ns/base.bizmo.federal_enterprise. federal_enterprise_objective >. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo.federal_enterprise> <http://rdf.freebase.com/ns/type.type. properties> <http://rdf.freebase.com/ns/base.bizmo.federal_enterprise. federal_enterprise_information_technology_budget >. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo.federal_enterprise> <http://rdf.freebase.com/ns/type.type. properties > <http://rdf.freebase.com/ns/base.bizmo.federal_enterprise. federal_enterprise_goal >. </li></ul><ul><li><http://rdf.freebase.com/ns/base.bizmo.federal_enterprise> <http://www.w3.org/1999/ xhtml/vocab#license > < http://creativecommons.org/licenses/by/3.0/ >. </li></ul>

30. connecting the data dots: <ul><li>create the following subject/predicate/object or topic/property/topic schema : </li></ul><ul><li>Goal / amplifies / Vision </li></ul><ul><li>Objective / quantifies / Goal </li></ul><ul><li>Federal Enterprise / (has) Fed Ent Goal / (of type) Goal </li></ul><ul><li>Federal Agency / maintains / Exhibit 53 </li></ul><ul><li>Exhibit 53 / contains (multiple) / Exhibit 53 Recordset(s) </li></ul><ul><li>Exhibit 53 Recordset / Supports Federal Goal / (of type) Goal </li></ul><ul><li>then create instances with data from http://it.usaspending.gov: </li></ul><ul><li>Obama / is of type / Federal Enterprise </li></ul><ul><li>Obama / has a Fed Ent Goal / Health Care Reform </li></ul><ul><li>HHS / is of type / Federal Agency </li></ul><ul><li>HHS / maintains / HHS Exhibit 53 </li></ul><ul><li>HHS Exhibit 53 / contains / Nat Health Info Network Connect </li></ul><ul><li>Nat Health Info Network Connect / supports Obama Goal / Health Care Reform </li></ul>

31. search all 'bases' for 'Exhibit 53' http://mqlx.com/~david/parallax interface to http://bizmo.freebase.com

32. base/bizmo/e53 returns <ul><li>a collection (2 instances) of an Exhibit 53 topic </li></ul><ul><ul><li>one from HHS and GSA (data from it.usaspending.gov) </li></ul></ul><ul><li>triple in Exhibit 53 topic schema </li></ul><ul><ul><li>Exhibit 53 (S) contains (P) Exhibit 53 Recordset (O) </li></ul></ul>

33. discovering unknown data structures <ul><li>the power of 'faceted' search and browsing </li></ul><ul><li>interactive query – which of these? </li></ul><ul><ul><li>Ex53 Recordset (S) Supports Federal Goal (P) ? (O) </li></ul></ul>

34. traversing the data graph <ul><li>from info about an IT investment </li></ul><ul><li>to info about Administration priorities </li></ul><ul><li>2 Ex53's to 3 Recordsets to 1 that has Obama Goal </li></ul><ul><ul><li><uri> (S) <uri> (P) <uri> (O) </li></ul></ul>

35. http://freemix.it - more faceted filtering

36. scatter chart driven by tag clouds

37. more multi-dataset faceted meshups

38. drag & drop metadata/data 'curation'

39. publish new freemix merged dataset choose a stylesheet, view lenses and facets to include for your end users to interact with

41. crowdsourced analytics shown using 'Top Braid Composer Maestro' from http://topquandrant.com 'SPARQLMotion' script – also see Yahoo | Deri http://pipes.yahoo.com | http://pipes.deri.org

42. cloud scale analytics (petabyte batch) <ul><li>proprietary Google </li></ul><ul><ul><li>GFS, BigTable and MapReduce </li></ul></ul><ul><ul><li>page rank impl </li></ul></ul><ul><li>open source Apache Hadoop </li></ul><ul><ul><li>HDFS, HBase and MapReduce </li></ul></ul><ul><ul><li>entity, RDFa extraction </li></ul></ul><ul><li>Amazon EMR, Cloudera </li></ul><ul><ul><li>COSS prof service providers </li></ul></ul>facebook.com

43. talis.com/platform - cloud graph store <ul><li>Software as a Service, enabling rapid development with zero deployment costs </li></ul><ul><li>a simple, consistent web API for storing, managing and retrieving both structured and unstructured data </li></ul><ul><li>flexible, schema-free metadata that allows applications to be easily evolved </li></ul><ul><li>a range of data access and query options enabling easy integration into both new and existing applications </li></ul><ul><li>access control options to support hosting of both public and private data </li></ul><ul><li>a data hosting solution that is founded on open internet standards and web architectural best practices </li></ul><ul><li>... </li></ul><ul><li>every resource in your (data)store has a unique URL from which its metadata can be retrieved with a single web request </li></ul><ul><li>SPARQL queries can be used to perform more complex queries, retrieving results as a tabular result set or as RDF </li></ul><ul><li>content negotiation can be used to retrieve data as RDF, XML, or JSON allowing you to chose the right format for your application </li></ul>

45. application to EA discipline <ul><li>getting there from here </li></ul><ul><ul><li>stop: </li></ul></ul><ul><ul><ul><li>publishing / analyzing / visualizing unstructured data </li></ul></ul></ul><ul><ul><ul><li>using structure data only in file or message exchanges </li></ul></ul></ul><ul><ul><li>start: </li></ul></ul><ul><ul><ul><li>align Gov and Web architecture (including EA KB's!) </li></ul></ul></ul><ul><ul><ul><li>publish component ontologies on the Web </li></ul></ul></ul><ul><ul><ul><li>and begin linking their metadata and data </li></ul></ul></ul><ul><ul><ul><li>using the Social Data Web </li></ul></ul></ul><ul><ul><li>continue: </li></ul></ul><ul><ul><ul><li>embrace emergent structure and continuous improvement </li></ul></ul></ul><ul><ul><ul><li>using open source and enabling long-tail crowd-sourcing </li></ul></ul></ul>

46. q&a - discussion <ul><li>thanks for your time and attention! </li></ul><ul><li>contact me </li></ul><ul><ul><li>http://xri.net/=george.thomas </li></ul></ul><ul><ul><li>GSA OCIO Chief Enterprise Architect </li></ul></ul><ul><ul><li>FCIOC-AIC Services Subcommittee Chair </li></ul></ul><ul><ul><li>W3C eGov IG invited expert </li></ul></ul><ul><ul><li>OMG GovDTF Steering Committee </li></ul></ul><ul><ul><li>Graduate School Faculty SOA Instructor </li></ul></ul>

The Social Data Web

George Thomas

The Social Data Web