Dublinked Technology Workshop Overview


             15th Dec 2011


            Tim McCarthy, NUIM


                                         IDC/EMC, 2011
Dublinked Technology Workshop Overview

14.40 – 15.20 Presentations
14.40    Transportation data – Data Access Issues   Brendan O’Brien, DCC
14.50    Spatial Web services                       Eamonn Doyle, ESRI
15.00    Linked Data & Linked Data Catalogues       Deirdre Lee & Fadi Maali, DERI
15.10    Semantic approach to Data Description      Chis Matheus, Alcatel

15.20 – 15.50 Breakout Sessions
Group-1 Data Publishing                             Dominic Byrne
Group-2 Data Discovery                              Tim McCarthy
Group-3 Web Services                                Eamonn Doyle, ESRI
Group-4 Advanced Functions                          Pól MacAonghusa, IBM

15.50 – 16.25 Summary Presentation
4 X Groups        Main points                       Spokesperson
All               Conclusion, Actions

16.25 – 16.30 Wrap-up
Dublinked - The value of digital data
    •EU Open Data Strategy (Released 12th Dec 2011)
    http://ec.europa.eu/information_society/policy/psi/index_en.htm



    Relevant Reports include:
    •Euros140B                                                                     IDC/EMC, 2011

    (Vickery Study 2011) in direct/indirect economic gains from PSI


    •Pricing of PSI
    http://ec.europa.eu/information_society/policy/psi/docs/pdfs/report/11_2012/summary.pdf



    •Models for Supply & Charging for PSI
    http://ec.europa.eu/information_society/policy/psi/docs/pdfs/report/11_2012/models.pdf


    •Apps Market Snapshot
    http://ec.europa.eu/information_society/policy/psi/docs/pdfs/report/11_2012/apps_market.pdf
Background
At the heart of Dublinked is the idea that Dublin Local Authorities (as well as organizations in
the public, private & academic communities) generate large volumes of data pertaining to
Dublin City & region and would like to share this digital resource with wider society

However, in reality, there are technical, legal, commercial, political, organisational challenges
in sharing data:

Are organizations willing to share data and information? Does prevailing politics promote data sharing?
Do all stakeholders support the same philosophy? How to create incentives across different communities,
such as central government, commercial & academic to share data? Who pays for releasing data? Who is
responsible for maintaining data quality? How do metadata, citations, credits and data tagging promote
increased data sharing? How can you have context travel with data? Can we trace usage as approaches
to determine high value data? While search engines are often used to locate data and information, how
do portals provide curated, authoritative sources and build communities of collaboration? How are data
kept current? Who provides APIs, and higher level visualisation modules? How do we balance the need
for authoritative data and volunteered data? How do we safe-guard privacy? How do we support the
need for rapid access self-service data in the situations such as natural disasters and emergencies? How
do we maintain adequate controls to protect sensitive environmental information? Is licensing suitable
for product development & commercialisation? How can new web information service idea be tested?
People already have meaningful questions - how do we support their ability to generate information
about what they want to know? How do we scale up a regional collaborative data innovation platform
to the national level?...........................................
Responding to the needs for a new kind of story-telling, collaboration
& innovation
  •Generation of vast quantities of digital data

  •Digital advancements, disruptive technologies

  •Open Data initiatives (Transparency, Economic, Engagement)

  •Citizen participation

  •Data Processing, Analysis & Visualisation were historically the preserve of the
  computer-specialist, scientist & statistician

  •Change in societal expectations – requirement of near real-time or predicted
  information on mobile and web based

  •Once information is published & discovered, what are the best approaches to turning
  data & information into knowledge?
Technology Issues – Barriers, Enablers, Challenges & Opportunities



 Data Publishing                             Web Services
     •Data Sourcing                             •Static & Dynamic data streams
     •Pre-processing data (Google Refine)       •API
     •Data Validation                           •Client-Server Architectures
     •Cloud                                     •SmartPhone & Mobile platforms
     •Structures
     •Formats


 Data Discovery
     •Metadata                               Advanced Functions
     •Federated Data Catalogues                  •Analytics
     •Local, Regional & National Platforms       • Visualisation
     •Search Tools                               •Collaboration
     •Linked Data & Linked Data Catalogues
Open Cities – Technical Requirments & Open Government Data Platform




  This document contains a list of requirements for the Open
  Cities open data platform and contains a list of tool                          https://github.com/opengovplatform/opengovplatform
  specifications. In Section 1, we present a general
  overview of typical data management processes. Sections 3
  and 4 contain a listing of functional and non-functional
  requirements for the open data platform, respectively. In
  Section 5 we list the specifications for varying tools that we
  are currently considering for use in the open data platform.


  http://opencities.net/sites/opencities.net/files/content-
  files/repository/D4.4.2%20Requirements%20for%20tools%20for%20Open%20Data.pdf
Web Services Styles & APIs
Remote Procedure Call (RPC)
Distributed Function, not loosely coupled, related approaches include CORBA, Microsoft RPC .Net Remoting

Service Orientated Architecture (SOA)
Web services used to implement a SOA, message orientated, loose coupling, use enterprise service buses that combine
message orientated processing & Web services to create event driven SOA

Representational State Transfer (REST)
Attempt to describe architecture that use HTTP (or similar) by constraining the interface to standard, well know
operations (Get, Post, Put, Delete). The emphasis is interacting with a stateful resources rather than messages or
operations. Clean URLs are associated with REST concept. Architectures based on REST can use WSDL to describe SOAP
messaging over HTTP & can be implemented as an abstraction on top of SOAP or can be created without using SOAP at
all. Four basic design principles:
              •Use HTTP methods explicitly.
              •Be stateless.
              •Expose directory structure-like URIs.
              •Transfer XML, JavaScript Object Notation (JSON), or both.

API
 API is typically a defined set of Hypertext Transfer Protocol (HTTP) request messages, along with a definition of the
structure of response messages, which is usually in an Extensible Markup Language (XML) or JavaScript Object Notation
(JSON) format.

While Web API is virtually a synonym for web service, the recent trend (so-called Web 2.0) has been moving away from
Simple Object Access Protocol (SOAP) based services towards more direct Representational State Transfer (REST) style
communications. Web APIs allow the combination of multiple services into new applications known as mashups.
Interoperability - Open Geospatial Consortium http://www.opengeospatial.org/
. OGC® Standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services
and mainstream IT. The standards empower technology developers to make complex spatial information and services
accessible and useful with all kinds of applications. The OGC standards baseline comprises more than 30 standards,
including:


 CSW - Catalog Service for the Web: access to catalog information
 GML - Geography Markup Language:
 GeoXACML - Geospatial eXtensible Access Control Markup Language
 KML - Keyhole Markup Language:
 Observations and Measurements
 OGC Reference Model - a complete set of reference models
 OWS - OGC Web Service Common
 Sensor Observation Service[4] (SOS)
 Sensor Planning Service[5] (SPS)
 SensorML - Sensor Model Language
 SFS - Simple Features - SQL
 Styled Layer Descriptor (SLD)
 WCS - Web Coverage Service:
 WFS - Web Feature Service
 WMS - Web Map Service: provides map images
 WMTS - Web Map Tile Service: provides map image tiles
 WPS - Web Processing Service: remote processing service
ProgrammableWeb - Mashups, APIs, and the Web as Platform




                                             http://www.programmableweb.com/
Data SF – Real-time Parking & Pricing API
Guardian Newspaper (UK) Content API
TfL Cycle Hire




  data is being scraped every
  minute from the Barclays
  Cycle Hire Map and is then
  cached for further use
TfL Transportation Data Feeds
Tube API
Advanced Functions
THREDDS (Thematic Realtime Environmental Distributed Data Services)
ERDDAP – “middleman between you and various remote data servers”




                                         http://coastwatch.pfeg.noaa.gov/erddap/index.ht
                                         ml
GeoNode is an open source platform that facilitates the creation, sharing, and collaborative
use of geospatial data. The project aims to surpass existing spatial data infrastructure
solutions by integrating robust social and cartographic tools.
Weave – Analysis & Visualisation




                                   http://ivpr.github.com/Weave/
Many Eyes - IBM




  http://www-958.ibm.com/software/data/cognos/manyeyes/
Dashboards
             Various Types
             •Strategic
             •Analytic
             •Operational
SmartApps
            App market (estimated to be US$35B in
            2015).....PSI providing a significant percentage
            of the underlying data to App development
            eg Weather & Transport Apps. For
            PSI, increased commercial potential for Smart
            App development can be realised if data-
            streams are multi-thematic, real-time and
            readily integrated….need for clear
            licensing/guidelines on use/re-
            use....indications are we are still at the start
            here in terms of market development.

            Source :
            http://ec.europa.eu/information_society/policy/psi/docs/pdfs/report/11_2012/app
            s_market.pdf

Dublinked tech workshop_15_dec2011

  • 1.
    Dublinked Technology WorkshopOverview 15th Dec 2011 Tim McCarthy, NUIM IDC/EMC, 2011
  • 2.
    Dublinked Technology WorkshopOverview 14.40 – 15.20 Presentations 14.40 Transportation data – Data Access Issues Brendan O’Brien, DCC 14.50 Spatial Web services Eamonn Doyle, ESRI 15.00 Linked Data & Linked Data Catalogues Deirdre Lee & Fadi Maali, DERI 15.10 Semantic approach to Data Description Chis Matheus, Alcatel 15.20 – 15.50 Breakout Sessions Group-1 Data Publishing Dominic Byrne Group-2 Data Discovery Tim McCarthy Group-3 Web Services Eamonn Doyle, ESRI Group-4 Advanced Functions Pól MacAonghusa, IBM 15.50 – 16.25 Summary Presentation 4 X Groups Main points Spokesperson All Conclusion, Actions 16.25 – 16.30 Wrap-up
  • 3.
    Dublinked - Thevalue of digital data •EU Open Data Strategy (Released 12th Dec 2011) http://ec.europa.eu/information_society/policy/psi/index_en.htm Relevant Reports include: •Euros140B IDC/EMC, 2011 (Vickery Study 2011) in direct/indirect economic gains from PSI •Pricing of PSI http://ec.europa.eu/information_society/policy/psi/docs/pdfs/report/11_2012/summary.pdf •Models for Supply & Charging for PSI http://ec.europa.eu/information_society/policy/psi/docs/pdfs/report/11_2012/models.pdf •Apps Market Snapshot http://ec.europa.eu/information_society/policy/psi/docs/pdfs/report/11_2012/apps_market.pdf
  • 4.
    Background At the heartof Dublinked is the idea that Dublin Local Authorities (as well as organizations in the public, private & academic communities) generate large volumes of data pertaining to Dublin City & region and would like to share this digital resource with wider society However, in reality, there are technical, legal, commercial, political, organisational challenges in sharing data: Are organizations willing to share data and information? Does prevailing politics promote data sharing? Do all stakeholders support the same philosophy? How to create incentives across different communities, such as central government, commercial & academic to share data? Who pays for releasing data? Who is responsible for maintaining data quality? How do metadata, citations, credits and data tagging promote increased data sharing? How can you have context travel with data? Can we trace usage as approaches to determine high value data? While search engines are often used to locate data and information, how do portals provide curated, authoritative sources and build communities of collaboration? How are data kept current? Who provides APIs, and higher level visualisation modules? How do we balance the need for authoritative data and volunteered data? How do we safe-guard privacy? How do we support the need for rapid access self-service data in the situations such as natural disasters and emergencies? How do we maintain adequate controls to protect sensitive environmental information? Is licensing suitable for product development & commercialisation? How can new web information service idea be tested? People already have meaningful questions - how do we support their ability to generate information about what they want to know? How do we scale up a regional collaborative data innovation platform to the national level?...........................................
  • 5.
    Responding to theneeds for a new kind of story-telling, collaboration & innovation •Generation of vast quantities of digital data •Digital advancements, disruptive technologies •Open Data initiatives (Transparency, Economic, Engagement) •Citizen participation •Data Processing, Analysis & Visualisation were historically the preserve of the computer-specialist, scientist & statistician •Change in societal expectations – requirement of near real-time or predicted information on mobile and web based •Once information is published & discovered, what are the best approaches to turning data & information into knowledge?
  • 6.
    Technology Issues –Barriers, Enablers, Challenges & Opportunities Data Publishing Web Services •Data Sourcing •Static & Dynamic data streams •Pre-processing data (Google Refine) •API •Data Validation •Client-Server Architectures •Cloud •SmartPhone & Mobile platforms •Structures •Formats Data Discovery •Metadata Advanced Functions •Federated Data Catalogues •Analytics •Local, Regional & National Platforms • Visualisation •Search Tools •Collaboration •Linked Data & Linked Data Catalogues
  • 7.
    Open Cities –Technical Requirments & Open Government Data Platform This document contains a list of requirements for the Open Cities open data platform and contains a list of tool https://github.com/opengovplatform/opengovplatform specifications. In Section 1, we present a general overview of typical data management processes. Sections 3 and 4 contain a listing of functional and non-functional requirements for the open data platform, respectively. In Section 5 we list the specifications for varying tools that we are currently considering for use in the open data platform. http://opencities.net/sites/opencities.net/files/content- files/repository/D4.4.2%20Requirements%20for%20tools%20for%20Open%20Data.pdf
  • 8.
    Web Services Styles& APIs Remote Procedure Call (RPC) Distributed Function, not loosely coupled, related approaches include CORBA, Microsoft RPC .Net Remoting Service Orientated Architecture (SOA) Web services used to implement a SOA, message orientated, loose coupling, use enterprise service buses that combine message orientated processing & Web services to create event driven SOA Representational State Transfer (REST) Attempt to describe architecture that use HTTP (or similar) by constraining the interface to standard, well know operations (Get, Post, Put, Delete). The emphasis is interacting with a stateful resources rather than messages or operations. Clean URLs are associated with REST concept. Architectures based on REST can use WSDL to describe SOAP messaging over HTTP & can be implemented as an abstraction on top of SOAP or can be created without using SOAP at all. Four basic design principles: •Use HTTP methods explicitly. •Be stateless. •Expose directory structure-like URIs. •Transfer XML, JavaScript Object Notation (JSON), or both. API API is typically a defined set of Hypertext Transfer Protocol (HTTP) request messages, along with a definition of the structure of response messages, which is usually in an Extensible Markup Language (XML) or JavaScript Object Notation (JSON) format. While Web API is virtually a synonym for web service, the recent trend (so-called Web 2.0) has been moving away from Simple Object Access Protocol (SOAP) based services towards more direct Representational State Transfer (REST) style communications. Web APIs allow the combination of multiple services into new applications known as mashups.
  • 9.
    Interoperability - OpenGeospatial Consortium http://www.opengeospatial.org/ . OGC® Standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services and mainstream IT. The standards empower technology developers to make complex spatial information and services accessible and useful with all kinds of applications. The OGC standards baseline comprises more than 30 standards, including: CSW - Catalog Service for the Web: access to catalog information GML - Geography Markup Language: GeoXACML - Geospatial eXtensible Access Control Markup Language KML - Keyhole Markup Language: Observations and Measurements OGC Reference Model - a complete set of reference models OWS - OGC Web Service Common Sensor Observation Service[4] (SOS) Sensor Planning Service[5] (SPS) SensorML - Sensor Model Language SFS - Simple Features - SQL Styled Layer Descriptor (SLD) WCS - Web Coverage Service: WFS - Web Feature Service WMS - Web Map Service: provides map images WMTS - Web Map Tile Service: provides map image tiles WPS - Web Processing Service: remote processing service
  • 10.
    ProgrammableWeb - Mashups,APIs, and the Web as Platform http://www.programmableweb.com/
  • 11.
    Data SF –Real-time Parking & Pricing API
  • 12.
  • 13.
    TfL Cycle Hire data is being scraped every minute from the Barclays Cycle Hire Map and is then cached for further use
  • 14.
  • 15.
  • 16.
  • 17.
    THREDDS (Thematic RealtimeEnvironmental Distributed Data Services)
  • 18.
    ERDDAP – “middlemanbetween you and various remote data servers” http://coastwatch.pfeg.noaa.gov/erddap/index.ht ml
  • 19.
    GeoNode is anopen source platform that facilitates the creation, sharing, and collaborative use of geospatial data. The project aims to surpass existing spatial data infrastructure solutions by integrating robust social and cartographic tools.
  • 20.
    Weave – Analysis& Visualisation http://ivpr.github.com/Weave/
  • 21.
    Many Eyes -IBM http://www-958.ibm.com/software/data/cognos/manyeyes/
  • 22.
    Dashboards Various Types •Strategic •Analytic •Operational
  • 23.
    SmartApps App market (estimated to be US$35B in 2015).....PSI providing a significant percentage of the underlying data to App development eg Weather & Transport Apps. For PSI, increased commercial potential for Smart App development can be realised if data- streams are multi-thematic, real-time and readily integrated….need for clear licensing/guidelines on use/re- use....indications are we are still at the start here in terms of market development. Source : http://ec.europa.eu/information_society/policy/psi/docs/pdfs/report/11_2012/app s_market.pdf