![The Open Data Movement
Our proposal is simple: […] the federal
government […] is to provide data that is easy
for others to reuse, rather than to help citizens
use the data in one particular way or another
Open infrastructures that enable citizens to
make their own uses of the data
Reverse the current policy, which is to regard
government websites themselves as the primary
vehicle for the distribution of public data, and
open infrastructures for sharing the data as a
laudable but secondary objective [Robinson,Yu,
Zeller and Felten 2008]
2](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Dealing with Open Data in Istat
Similar to Dealing with Open Data in Istat (20)
More from Giovanni Barbieri
More from Giovanni Barbieri (20)
Recently uploaded
Recently uploaded (20)
Dealing with Open Data in Istat
- 1. Dealing with Open Data at ISTAT Giovanni A. Barbieri Statistics Italy (Istat)
- 2. The Open Data Movement Our proposal is simple: […] the federal government […] is to provide data that is easy for others to reuse, rather than to help citizens use the data in one particular way or another Open infrastructures that enable citizens to make their own uses of the data Reverse the current policy, which is to regard government websites themselves as the primary vehicle for the distribution of public data, and open infrastructures for sharing the data as a laudable but secondary objective [Robinson,Yu, Zeller and Felten 2008] 2
- 3. Crowdsourcing Government Transparency Government information that is nominally publicly available is in fact difficult to access either because it is not online or, if it is online, because it is not available in useful and flexible formats [Brito 2008] “Structured data” Associated structured XML file would allow a user to sort the data by ascending or descending date, alphabetically by headline or author, by number of words, and in many other ways 3
- 4. Open Data Ecosystem “Open data are adding a new dimension to big data analytics and giving rise to novel, data-driven innovations.” (McKinsey Global Institute Report, Oct. 2013) From Citizens IBM BLOG Wide Range of Open Data Consumers Citizens that would like to learn characteristics of the places they are, e.g. with mobile apps showing location- specific features Journalists that need to access data for updated and aware information communication Educators that are helped in their teaching task by access to data on different application domains 4
- 5. Official Statistics meets Open Data Official Statistics can more easily reach such wide range of users if conveyed through open data Recent technologies advances in the open data community enable new advanced dissemination channels for Official Statistics 5 Reinforcing trust Getting closer to users Reaching new users Giving information back Improving metadata
- 6. Linked Open Data Semantic Web Technological Standards OWL Knowledge Representation Linked Open Data - LOD 6
- 7. Why is Linked Data an Opportunity? Linked Data as a semantically rich paradigm for data representation Rich enough for the strict requirements of Official Statistics Formal and well-defined data structures, i.e. ontologies Linked Data as an international standard (W3C) Tools availability and independence Beyond statistical users RDF: Resource Description Framework (W3C) (subject-predicate-object) 7
- 8. Istat’s Linked Open Data Portal - 1 Istat LOD Portal: http://datiopen.istat.it English Version: http://datiopen.istat.it/index.php?language=eng 8
- 9. Platform for • Selecting • Navigating • Searching • Querying • Visualizing Open Data The platform allows • Direct access to data via Web Services • M2M solutions (e.g. GIS-LOD) • Data conversion • Export to productivity tools • Visualization by means of external tools Istat’s Linked Open Data Portal - 2 9
- 10. STEP 1 STEP 3STEP 2 Istat’s Linked Open Data Portal - 3 Steps to a «perfect» data portal 10
- 11. Istat’s Linked Open Data Portal - 4 Guided Access Freedom of access Typeofinteraction Free Access Human basic MachineTo Machine Navigation Guided queries Query REST on SPARQL EndPoint Query via SPARQL EndPoint Web Service Download Human Advanced 11
- 12. Predefined Queries (Set of simple and customizable queries) Free Queries (SPARQL Queries) Navigation Guided Queries Download Typeofinteraction Human Guided Access Free Access Interaction Modes Freedom of access Basic Advanced (Human) User technical skills Intermediate Free Query via SPARQL EndPoint 12
- 13. Use Case 1: Spatial Querying App that displays on a map some population indicators of the nearest census sections to specific GPS coordinates LOD when accompanied by spatial information allow to access data using spatial queries 13
- 14. Use Case 2: Federated Querying - 1 Federated query on Istat and ISPRA, i.e. the query accesses Istat and ISPRA portals With LOD, it is very easy to compare data coming from different sources (linked for example at territorial level) Query on one Portal Results dynamically retrieved from both portals 14 ISPRA - The Italian National Institute for Environmental Protection and Research Istat
- 15. Use Case 2: Federated Querying - 2 ISTAT data: Census Buildings ISPRA data: Data on land use / soil consumption Example Query: Municipality-level analysis of land use / soil consumption and number of buildings by period of construction Dynamically generated! 15
- 16. Use Case 3: Istat as Open Data Provider in SPOD Social discussion (on the left) about a graphic representation of Census Data (on the right) Dynamically generated! 16 SPOD: Social Platform for Open Data
- 17. Conclusions A dissemination strategy based on open data does put the Official Statistics users at the centre: Reaching them through different channels e.g. apps and social media Making easier for them to retrieve data e.g. federated queries that make transparent the distribution of data on different portals Providing richer services to them e.g. spatial querying and dynamical visualizations 17 Open data and in particular Linked Open Data have a leading role in data innovation for Official Statistics
- 18. • Macroscale vs microscale modeling – Pseudo-Einstein (as simple as possible but not simpler) – Von Neumann (agent-based modeling) • Technological constraint enabling technology • It widens the space of what is feasible: – In production: our experience with SBS.Frame – In analysis and research… • A paradigm shift? – Statistical mechanics vs agent-based modeling – Just because you can doesn’t mean you should • Back to open data – From dissemination to release (“data liberation” at StatCan) to the development of information – The regulators need to introduce new rules in line with the new scenarios (Don't think of an elephant!: know your values and frame the debate) One More Thing 18
- 19. Thanks to all my colleagues in Istat contributing to the LOD Portal Special thanks to Monica Scannapieco and Stefano De Francisci Questions and clarifications: contact me at barbieri@istat.it Acknowledgments and Thanks 18
Editor's Notes
- Nomi
- Tim Berners-Lee inventore del WWW (ipertesto) nel 1989 Our proposal is simple: The new administration should specify that the federal government’s primary objective as an online publisher is to provide data that is easy for others to reuse, rather than to help citizens use the data in one particular way or another [Robinson,Yu, Zeller and Felten 2008] We argue that when providing data on the Internet, the federal government’s core objective should be to build open infrastructures that enable citizens to make their own uses of the data. If, having achieved that objective, government takes the further step of developing finished sites that rely on the data, so much the better. Our proposal would reverse the current policy, which is to regard government websites themselves as the primary vehicle for the distribution of public data, and open infrastructures for sharing the data as a laudable but secondary objective
- Tim Berners-Lee inventore del WWW (ipertesto) nel 1989 Our proposal is simple: The new administration should specify that the federal government’s primary objective as an online publisher is to provide data that is easy for others to reuse, rather than to help citizens use the data in one particular way or another [Robinson,Yu, Zeller and Felten 2008] We argue that when providing data on the Internet, the federal government’s core objective should be to build open infrastructures that enable citizens to make their own uses of the data. If, having achieved that objective, government takes the further step of developing finished sites that rely on the data, so much the better. Our proposal would reverse the current policy, which is to regard government websites themselves as the primary vehicle for the distribution of public data, and open infrastructures for sharing the data as a laudable but secondary objective
- Open data are a key enabler of data-driven innovation. The McKinsey report "Open data: Unlocking innovation and performance with liquid information“ remarks the significant economic value can be generate from open data benefits, “including increased efficiency, new products and services, and a consumer surplus (cost savings, convenience, better products)”. The report also remarks how open data enhances big data’s impact “by creating transparency, exposing variability, and enabling experimentation; helping companies to segment populations and thus to customize actions directed at them; replacing or supporting human decision making; and spurring innovative business models, products, and services”. There is a wide range of open data consumers. Open data customers range from Citizens, who would like to access data in any time and any location through mobile apps, Journalists, who need timely and trustable data, Educators who can rely on data from different application domains for their teaching materials. From the providers perspective, being them public or private providers, there is the need to offer advanced services flexible enough to meet the requirements of such a wide range of users.
- NSOs are producers of data, hence it is a major priority for them to reach data consumers and provide them information of excellent quality. Open data are an instrument to reach data consumers more easily. There are new advanced technologies and standards available in the open data community that permit to build new advanced dissemination channels for official statistics, as we will see in the rest of the presentation. When Official Statistics meets Open Data, there are several benefits: (i) Reinforce trust in NSO role, by a data provision that is aligned with state-of-the art standards and technologies and which is «certified» by the quality guarantees of Official Statistics. (ii) Getting closer to users, by an enhanced accessibility that, for instance, permits users to access data by mobile devices. (iii) Reaching new users, by developing services flexible enough to address the requirements of different classes of users (basic and advanced). (iv) Give back information to users, in a «virtuous» loop where respondents of OS can get back information elaborated on the basis of data they provided (Virtuous loop with Official statistics respondents). (v) Information enrichment, by providing information products with a metadata equipment that permits proper interpretation and usage of such products.
- Linked Open Data is a new paradigm for integrating and publishing data. It relies on Semantic Web technologies and standards on one side and on principles of Knowledge Representation on the other side. Technological standards, mostly proposed by the World Wide Web Consortium (W3C), have the important role to make IT vendors developing tools on the «same» concepts, so ensuring organizations adopting such standards to be vendor-independent. From the scientific and academic world, knowledge representation pushes the usage of rich metadata models (e.g. ontologies) that can be expressed by semantic Web languages and formats. The combination of these two worlds (technology standards and knowledge representation) is the basis of the Linked Data paradigm, proposed by Tim-Berners Lee (the inventor of the World Wide Web) in 2006. Linked Open Data (LOD) are Linked Data whose content is open and are the most advanced type of open data.
- Tim Berners-Lee inventore del WWW (ipertesto) nel 1989
- Istat published its own LOD portal in May 2015. The portal is the first dissemination system launched by Istat that fully complies with the Italian guidelines to enhance public information as well as the European ones on open data, mainly Revision of the Directive 2003/98/EC on the re-use of public sector information published on 26 june 2013 (Full compliance with Italian and European open data guidelines). In its first year of publication, the portal had over 23,000 unique visitors, about 400,000 page views and nearly one million individual accesses. The portal´s main goals are: Providing a single access point to Istat’s open data as part of the Italian national data cloud, in connection with dati.gov.it, the Italian national open-data portal; Providing machine-to-machine data access at the finest granularity level (i.e. each single value); Flexible querying via LOD paradigm; Advanced navigation mechanisms (faceted browsing, graph browsing, etc.). This means that the user “click-by-click” can discover data, instead of posing a known query: this is the case when the user wants to see what it is offered and does not have in mind a specific dataset to look for. He\She can navigate on graphs, clicking the nodes or by facets (i.e. by data features selected to organize the navigation task).
- The Linked Open Data (LOD) web site allows to access and browse data in open format based on technology and Semantic Web standards. The LOD can be inquired directly from any application and responds to the need, expressed by the community of users, to have standardized and interoperable data. Need to disseminate data from the Italian 15th Census of Population and Housing Territorial level: Census Section (submunicipality) 402.903 Census Sections Hard to manage the huge number of Census Sections through our dissemination warehouse I.stat
- By means of progressive steps, the LOD portal allows to implement effective solutions for sharing data among Statistical Organizations and making them available to the users communities
- This plan shows the degrees of accessibility (Access Freedom axis) vs the user interactions types (Interaction axis). As represented, there are two main degrees of accessibility, namely: guided access and free access. The guided access means that the user can rely on some predefined choices in accessing data, while the free one means that the user can pose free queries to the system or can freely navigate it. The user interaction types distinguishes between machine to machine users (i.e. software applications that access the portal) and human users that can be basic or advanced. In particular, looking at each quadrant: Human Basic/Guided: they can perform data download of predefined datasets. Human Basic/Free: they are able to discover information by using available navigation systems, i.e. systems that click-by-click permit to find out the needed information. Human Advanced/Guided: they can perform queries but in a guided way, i.e. they do not write the queries from scratch, but such queries are automatically generated through visual systems. Human Advanced/Free: they can write queries in a free way, without any specific limitation. MachineTo Machine/Guided: Data available on the portal can be accessed via pre-defined Web services. There are currently two available web services: one for accessing Census Indicators and another one for Labour Market areas. MachineToMachine/Free: Data available on the portal can be accessed by a software application in a free way. This means that another software application can pose a free query and retrieve data from the portal. Our GIS system, for instance, is able to retrieve data from our LOD portal exactly in this way (internal systems interoperability).
- Starting from this slide, we will see three possible use cases concerning the usage of the LOD portal. The three use cases are: spatial querying, federated querying and LOD portal as a data provider for a social platform. This first use case shows how it is possible to perform spatial queries of LOD data, and, in addition, how to interact with it through a mobile device. In particular, given some GPS coordinates, the user can retrieve population indicators related to the nearest census sections (nearest to the closest point of the polygon corresponding to the census section) and visualize them on a mobile phone. Thinking of a user query interested to discover how many children are in a certain area, in the figure, it is shown the number of residents with age lower than 5 years for the 6 census sections closer to the centre of the Italian municipality Anguillara Sabazia.
- One of the major advantage of the LOD paradigm is the possibility of interconnecting different dissemination systems (data interoperability across different organizations). Indeed, it is possible to integrate data available at different sites so that data can be accessed as if they were available at a single site (location transparency). The mechanism according to which this is possible is known as «federated queries», that is querying one portal and access to all the other federated portals. We linked the data of Istat LOD Portal with those of the ISPRA (The Italian National Institute for Environmental Protection and Research) LOD portal. The linkage was performed at territorial level, and in particular at municipality level (through Istat’s municipality codes). Given such a linkage, a query can be posed to Istat portal asking simultaneously for Istat’s data and ISPRA’s data. The query is executed by accessing to both portals in a dynamic way and the result is returned to the user. The fact that also the ISPRA portal is queried is completely transparent for the final users. Istat is part of National and European statistical systems that would greatly benefit from this concrete data interoperability, namely the National Statistical System (SISTAN) and the European Statistical System (ESS). By adopting this paradigm, it would be much easier to produce «integrated dissemination products» as outputs directly attributable to such systems. In the next slide a specific example of federated querying is shown.
- In this slide, we show by example how to exploit the link beetween Istat LOD portal and ISPRA LOD portal previously mentioned. Let us suppose that a user needs to retrieve and graphical display a specific combination of Istat’s data and ISPRA’s data, namely municipality-level analysis of soil consumption and buildings (percentage) by period of construction. He/she can perform three steps: 1. A query can be posed to Istat’s portal for retrieveing data. 2. By exploiting the link beetween the Istat LOD portal and the ISPRA one, data results are retrieved at Istat’s site. 3. Retrieved data are visually showed by an Istat’s application that dynamically takes as input such results and diplays a bar graph based on both Istat’s and ISPRA’s data. In this approach that federates portals by different organizations, it is very much important to keep information about the provenance of data. This is well supported by the LOD paradigm, and indeed data published on Istat’s LOD portal do have «attached» this kind of metadata that certify the provenance in terms of the agent that published data (Who), the published resource (What) and the time information related to it (When).
- The SPOD platform makes a query and retrieve Istat’s data that are shown as a graphic on the right. The graphic reports Italian Resident popoulation by age for the Rome municipality. On the left there is a discussion carried out by different users on such a dynamically retrieved graphic. The SPOD platform is indeed an example of a third party subject that accesses Istat’s data and provides advanced services to the final users. This is a very much promising business model for provision of services based on open data, especially by public institutions like Istat: the open data provider does not have necessarily to spend resources on building and maintaining services based on open data, but, instead, it could be up to third parties to have this specific role.
- User-centered dissemination is strongly enabled by open data. The use cases have shown: Users can be reached through different channels that are close to where they are (mobile apps) and what they are used to do (social media). Users can retrieve data in an easier way, due to federated queries that make transparent the distribution of data across federated portals. Users can benefit from richer services like spatial querying and dynamical visualizations. Though NSOs are adopting open data strategies since several years, Linked Open Data can give new impetus to such strategies. That’s why we assign to open data and in particular to Linked Open Data a strategic role in the innovation landscape for Official Statistics.