Open Data Trentino presented at the European Commission (JRC)
1. Open Government Data*
http://dati.trentino.it/
Lorenzino Vaccari
Autonomous Province of Trento, Trento, Italy
lorenzino.vaccari@provincia.tn.it
*Part of this presentation is taken from the “Open Government Data Tutorial” gave at CLEI2013 Conference
by Lorenzino Vaccari and Juan Pane (Universidad Nacional de Asuncion, Paraguay)
1
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
2. In this presentation…
• Introduce Open Government Data
•
•
Intro (Part 1)
Issues (Part 2)
• If you need it, how can you organize it?
•
Real experience (Part 3)
• Reusing open data
•
•
2
Applications (Part 4)
Semantic layer (Part 5)
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
3. 3
Lorenzino Vaccari - Autonomous Province
Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
Juan Pane, Lorenzino Vaccari of Trento, Trento,http://www.point-fort.com/index.php?2012/01/25/805-why-how-what
15/10/2013
http://www.point-fort.com/index.php?2012/01/25/805-why-how-what
4. What?
“is data that can be freely used, reused and redistributed by anyone –
subject only, at most, to the requirement to attribute and sharealike.” *
*(Source:
4
)
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://www.opendefinition.org
5. use
reuse
“open” =
redistribution
commercial reuse
derivative works
BUT, may require:
- attribution
- share alike
http://myfbcovers.com/uploads/covers/2012/06/09/16628a1094aa012f7c6e0025902480d2/watermarked_cover.jpg
5
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how
6. The value is in its use
6
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
7. 7
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q
10. Why
The Open data are the knowledge base to:
Improve the economic grow and the
entrepreneurship based on the development of digital
services reusing Public Sector Information
Answer to social needs through the publication of
innovative services and applications
Aims at reducing the cost of the public
administrative activities within Public – Private
Partnerships (PPP)
Improve the transparency of the activities of the public
institutions and the participation of the citizens to
these activities
10
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how
11. How - Principles
Tim Berners-Lee (5-Stars of Linked Open Data)
http://5stardata.info/
vs.
Tim Davis (5-Stars of Open Data Engagement)
http://www.timdavies.org.uk/2012/01/21/5-stars-of-open-data-engagement/
vs.
OGD: Ten principles for opening up government
information http://sunlightfoundation.com/policy/documents/ten-open-data-principles/
…
11
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
12. 5 Stars Linked Open Data
12
Tim Berners-Lee
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://5stardata.info
13. 5-Stars of Open Data Engagement
Tim Davis
Create Community
13
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://msnbcmedia.msn.com/j/MSNBC/Components/Photo/_new/pb-121007-spain-tarragona-pyramid-nj-02.photoblog900.jpg
14. Open Government Data: Ten principles
for opening up government information
1.
Completeness
2.
Primacy (primary source)
3.
Timeliness
4.
Ease of Physical and Electronic Access
5.
Machine readability
6.
Non-discrimination
7.
Use of Commonly Owned Standards
8.
Licensing
9.
Permanence
10. Usage Costs
14
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
15. State of the Art
What is happening around us?
• Globally
• Europe
• Italy
15
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
16. Open Data Charter - G8
The principles are:
Open Data by Default
Quality and Quantity
Useable by All
Releasing Data for Improved Governance
Releasing Data for Innovation
https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex
16
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://opensource.com/government
17. Open Data Census (OKF)
http://census.okfn.org/
17
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://opensource.com/government
19. Open Data Barometer (ODI)
http://census.okfn.org/
19
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://opensource.com/government/13/7/open-data-charter-
20. OGD in Europe
screenshots
20
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://epsiplatform.eu/content/european-psi-scoreboard
21. OGD in Europe
Insert table
http://epsiplatform.eu/content/european-psi-scoreboard
http://epsiplatform.eu/content/psi-scoreboard-indicator-list
21
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
22. 22
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://open-data.europa.eu/
23. OGD in Italy
23
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://www.dati.gov.it/content/infografica
24. OGD: Part 2 - Issues
24
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
25. 25
Lorenzino Vaccari - Autonomous Province
Juan Pane, Lorenzino Vaccari of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
08/10/2013
http://evian-thesource.com/kids-having-fun/
http://evian-thesource.com/kids-having-fun/
26. Open Data. Oh ohh
Organizational
26
Legal
Adoption
Barriers
Technical
Contextual
Lorenzino Vaccari - Autonomous Province
Juan Pane, Lorenzino Vaccari of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
08/10/2013
http://www.wallpapermania.eu/wallpaper/trick-or-treat-cute-pumpkins-lanterns-halloween-wallpaper
27. 27
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://de.straba.us/wp-content/uploads/2012/08/barrieres_for_implementation_of_ogd.png
28. Organizational Barriers
• Not ready
• Lack of resources
• IT
• Human
• Don’t want to be ready
28
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://montcomediation.org/images/MCMC_MyWayYourWay.jpg
29. Legal barriers
Open the Data
All the data that was produced using public money
has to be made publicly available (with exceptions)
vs Privacy
You cannot open data that could allow correlation
of private personal data
Or the complete lack of
legislation!
29
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
30. Adoption barriers
Data is not contextualized
People are not informed
Opening data is a complex task, opening cleaned data is
even more complex.
Unclear licenses
30
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
31. Technical Barriers
Access to data:
Organizational
Technical, Downtimes, logins,
Payment fees
Fragmentation, incomplete data,
scattered
Format
Cataloging, indexing, search
Lack of explicit semantics, metadata
Data is not reliable
Conflicting standards, models,
ontologies
31
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
32. Barriers
Zuiderwijk et al 2010
http://www.ejeg.com/issue/download.html?idArticle=255
Listed 118 socio-technical impediments for
opening data in the literature.
Findability
Usability
Understandablity
Quality
Linking
Comparability and compatibility
Metadata
….
32
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
33. Context Barriers
Privileged access to data
Other companies what to avoid legislation of privacy.
Transparency is bad for fraudulent business
33
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://img.gawkerassets.com/img/182n8vzdlg1iojpg/original.jpg
34. 34
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://netdna.webdesignerdepot.com/uploads/photo_manipulation/manipulation-9.jpg
35. Part 3 - Real Experience
35
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
36. Our story started with GeoData…
36
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://www.territorio.provincia.tn.it
37. 5 Stars Linked Geo Data Catalog
DBpedia
37
TrentinoGeoData
Freebase
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
38. The “Open Data in Trentino” project
• The “Open Data in Trentino” project is a 3 years initiative finalized to
develop an open data infrastructure to enhance Service
Innovation for Trentino following the PAT strategy for services
innovation enabled by ICT. The project will be developed within a
partnership between Trento RISE and the Autonomous Province of
Trento (PAT) according to the innovation PAT model
• Goals
• Improved quality of life for citizens
• Open Data and local businesses
• Transparency
• Improved efficiency and productivity
38
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
39. Workplan – Best practices
Not only a Project, but also a “Change
management process”
Best Practices:
-
39
Guidelines (metadata, formats, licences)
Point of contact (domain, operator)
ONE dataset each provider
Community Building
Distributed catalog
Clear Licences
Enterprises
Courses
Contest
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
40. Guidelines
Nome (Acronimo)
Descrizione
Tipo di Dato
Estensione
del file
Comma Separated Value (CSV)
Dato tabellare
Formato testuale per l'interscambio testuale di tabelle, le cui righe corrispondono a
linee e i cui valori delle singole colonne sono separati da una virgola (o punto e
virgola)
Geographic Markup Language (GML)
Dato geografico
Formato XML utile allo scambio di dati territoriali di tipo vettoriale
vettoriale
Keyhole Markup Language (KML)
Formato basato su XML creato per gestire dati territoriali in tre dimensioni nei
programmi Google Earth, Google Maps
Open Document Format (ODF)
Formato per l'archiviazione e lo scambio di documenti di testo, fogli di calcolo,
diagrammi e presentazioni
Resource Description Framework (RDF)
Basato su XML, e' lo strumento base proposto da World Wide Web
Consortium (W3C) per la codifica, lo scambio e il riutilizzo di metadati
strutturati e consente l'interoperabilità tra applicazioni che si scambiano
informazioni sul Web
ESRI Shapefile (SHP)
Lo Shapefile ESRI è un popolare formato vettoriale per sistemi informativi
geografici. Il dato geografico viene distribuito normalmente attraverso tre o quattro
files (se indicato il sistema di riferimento delle coordinate). Il formato è stato
rilasciato da ESRI come formato (quasi) aperto
Extensible Markup Language (XML)
E' un formato di markup, ovvero basato su un meccanismo che consente di
definire e controllare il significato degli elementi contenuti in un documento o in un
testo attraverso delle etichette (markup)
40
.csv
Dato geografico
vettoriale
.kml
Dato tabellare
.odc
.gml
Dato strutturato .rdf
Dato geografico
vettoriale
.shp, .shx, .dbf,
.prj
Dato strutturato .xml
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
41. Tecnological platform
…
Comune
Meteo GeoDati Statistica Comune Trasporti
Meteo GeoDati Statistica Trento
Trasporti
Trento
41
…
Etc…
Etc…
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
43. Data Sources Plan
Dati SA T #28
Catasto #
18
S GC CSW
#9
Bilanci o
PAT #37,
38
A ff F inanziari
Consuienze
#36
Gestione
S trade
#16
Immi grazi one
#52
P ersonal
e
P AT #41
Imp Funivie
#50
Idrom etri
ci
#26 Comune
T rento
update
20
Novembre
43
30
Turism o S TU
#53
A genzie F orestali
#64
Incendi #65
A ttività Culturalii
Dati Progettone#59
#63
S ovr. Beni A rch
#60
Dati
Motorizzazione Dati Scuola #61
#72
Dati Energia
#30
Trentino Cultura
#32
S ervzio Europa #56
Dati Traffico Stra
#70
20
Dicembre
Osservatorio
Lavori Pubb #17
T rasporti 2.0 #6
P ub. Eserci zi
#49
Gesti oni Patri monialii
#71
M in. Linguisti che
#48
Ufficio Ri fiuti
#34
10
Dati Cons. P rov #3
Attività
Form #58
Imp Depurazione
#68
Opere Civili #69
El ettorali #35
10
S ervizio
Istruzione #57
Ci nform i Stranieri
#66
30
10
20
Gennaio
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
30
44. Catalog
http://okfn.org
(2004)
The Open Knowledge Foundation (OKF) is a
non-profit organisation founded in 2004 and
dedicated to promoting open data and open
content in all their forms – including government
data, publicly funded research and public domain
cultural content.
44
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
45. http://dati.trentino.it*
* Available for all the data providers of Trentino
45
Analysis: http://dati.trentino.it/stats
Admin: http://dati.trentino.it/admin
Harvesting: http://dati.trentino.it/harvest
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
47. Legal Issues
Permissions: share, create, adapt
Actual interoperability!
Constraints: nothing!
Permissions: share, create, adapt
Actual interoperability!
Constraints: nothing!
http://www.hoax-slayer.com/images/privacy.jpg
http://www.destateparks.com/images/general_info/privacy_policy.jpg
47
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
48. Organizational Issues - Macro
48
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
49. Organizational Issues - Micro
49
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
50. Community building
Municipalities
Municipalities
“Consorzio dei
“Consorzio dei
Comuni”
Comuni”
Educational Institutes
Educational Institutes
“Comunità di Valle”
“Comunità di Valle”
of Trentino
of Trentino
Research Institutes
Research Institutes
Private Companies
Private Companies
Citizens
Citizens
50
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
52. Also Trentino is going to launch a challenge to build software applications and creative
products (multimedia, audiovisual products, posters, illustrations) based on the datasets
published on the http://dati.trentino.it open data catalog.
#ODTChallenge will be the official hashtag for our first open data challenge in Trentino!
52
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
53. 53
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
54. 580 datasets
provided by 10 departments of PAT…
8 months until now
68.555 visits
7.988 unique visits
2.516 downloads
62,64% new
visitors
37,36% returning
visitors
20
15
10
6
54
reporting errors
asking for new data
new suggestions
OD Applications
Agriculture
Culture
Geographical Data
Welfare
Weather Forecast
Social policies
Statistics
Transports
…MUNICIPALITY OF TRENTO, and
INFORMATICA TRENTINA
NOW
- ALL the departmnets demand to be involved
- Plus other local actors
100% ENTHUSIASTIC
REACTIONS
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
55. Want to Know more? A couple of links
55
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
56. 56
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://www.theodi.org/
57. 57
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://schoolofdata.org/
58. 58
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://schoolofdata.org/online-resources/
59. OGD: Part 4 - Applications
59
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
63. Open Source, Open Data, Open Hardware
http://limaio.innovacion.pe/
http://www.limaio.com/demo
63
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
64. 64
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://www.mysociety.org/2007/more-travel-maps/morehousing
65. Johann MITTHEISZ (CIO der Stadt Wien)
Total hours to develop 38 applications:
around 2.600
City of Wien saved around
208.000 Euro
65
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://www.slideshare.net/BrigitteLutz/keynote-mittheisz-cio-stadt-wien/16
66. Beyond Data
(The OpenStreetMap Case)
66
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
67. OpenStreetMap
OpenStreetMap project creates and provides
geographical data, such as road maps, freely
available to anyone. Behind the establishment and
growth of the project have been restrictions on
use or availability of map information across
much of the world and the advent of inexpensive
portable satellite navigation devices.
OpenStreetMap is a free map of the
world, created by someone like you
67
~
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
68. 68
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=11.12042&lat=46.07224&zoom=18
69. Watercolor maps
69
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://content.stamen.com/files/cartography/index_watercolor.html#18.00/46.07204/11.12097
70. From maps to blankets…
70
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://softcities.net
71. Sharing Data Globally
(the eHabitat example)
71
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
73. GEOSS Data Sharing
Principles
• Full and Open
Exchange of Data,
recognizing Relevant
International
Instruments and
National Policies
• Data and Products at
Minimum Time delay
and Minimum Cost
http://www.geoportal.org/web/guest/geo_home
73
• Free of Charge or
minimal Cost for
Research and
Education
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
74. GEOSS for biodiversity
74
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://www.eurogeoss-broker.eu/
75. The eHabitat Model
75
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://ehabitat-wps.jrc.ec.europa.eu/ehabitat/
76. OGD: Part 5 – Semantic Layer
76
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
78. Lack of explicit semantics
The real meaning of the data was kept in the
developers mind when creating the data
78
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://goo.gl/npEHKr
79. Lack of explicit semantics
Can lead to things like:
79
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
80. Semantic heterogeneity
Difference in the meaning of local data
80
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
81. Available
Data Catalog
Data Catalog
Structured
Open formats
Redefenceable
Linked
81
Entity centric
Entity centric
Importing tool
Importing tool
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
82. Entity centric: Added value
Aggregated data
Accurate data, manually curated
Unique identifiers, distributed perspectives
Re-think identifiers
Semantified values
E1
name
name
Ignacio P. F.
nationality italian
born in
Paraguay
lives in
Trento
date of birth
1980
affiliation
82
Juan Pane
E2
Univ. Trento
affiliation
PF-UNA
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
83. Entities
Real world: is something that has a distinct,
separate existence, although it need not be a
material (physical) existence. Has a set of
properties, which evolve over time. Example:
Mental: personal (local) model created and
maintained by a person that references and
describes a real world entity.
Digital: capture the semantics of real world
entities, provided by people.
83
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
84. Entity based Semantic Layer:
• Address the integration problems due to
semantic heterogeneity:
• Different formats
• Different identifiers
• Implicit semantics
• Homonyms, synonyms, aliases
• Partial knowledge
• Knowledge evolution
http://www.webfoundation.org/2011/11/5-staropen-data-initiatives/
84
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
85. The semantic Layer: why?
Applications use
entities instead of raw
data
REST/HTTP
REST/HTTP
Importing
Importing
Tool
Tool
Importing
Importing
Tool
Tool
v0
85
Importing
Importing
Tool
Tool
i
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
i+1
86. Importing
steps
1.
Selection
Schema Matching
3.
Data Validation
4.
Semantic Enrichment
5.
Reconciliation
6.
Exporting
7.
86
2.
Publishing
8.
Take raw data from dati.trentino.it
Map to an EntityType
Cleanse data
Link data to entities/concepts
Update/insert entities
Export to Entitypedia
Publish to dati.trentino.it
Get insights about entities
Visualization
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
87. 1. Source Selection
Import one data file at a time
87
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
88. 2. Schema Matching
Select a target type of entity -> correspondences between the input
columns and the output attributes
nome
provincia
descrizione
Andalo (1047)
Provincia di
Trento
Canazei (1450)
Trento Prov.
88
funivie
lat
long
Sorge su un'ampia sella 3
prativa al centro...
654463
712857
Situato all'estremità
settentrionale della...
511504
147444
2
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
89. 3. Data Validation
Applies format and structure validation and possible automatic
transformations needed to have the input data in the expected
format.
89
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
90. 4. Semantic Enrichment (1/2)
Entity disambiguation : Transform text references into links to
existing entities.
90
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
91. 4. Semantic Enrichment (2/2)
Natural Language Processing : Extract concepts and entity
references from free-text.
91
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
92. 5. Reconciliation
Run Identity Management Algorithms to identify each row as a new
or existing entity.
Result
•No Match
•Match
•Multiple
Matches
Action:
•Use ID
•New ID
•Ignore Row
92
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
93. 6. Exporting
At this point:
We know what to export.
All values for target attributes conform to the expected format.
All text has been semantified (NLP).
All textual references to entities are converted to links
Each row has an identifier
v0
93
i
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
i+1
94. 7. Publishing
Put back the semantified entities into CKAN so that the
entities can be Open Data and can be found in the same
catalog as the original data.
Developers and find the data files of the cleaned,
aggregated entities
But can also interact with the entities via the Entitypedia
APIs
8. Visualization
Search and Navigation
94
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
96. BEYOND
96
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
http://www.youtube.com/watch?v=Bq_ZWl1ZXA0
97. Thanks to all the Open Data in Trentino Team and in particular to:
Juan Pane, Maurizio Napolitano, Marco Combetto, Moaz Reyad and Luca Paolazzi
97
Lorenzino Vaccari - Autonomous Province of Trento, Trento, Italy - lorenzino.vaccari@provincia.tn.it 20/12/13
Editor's Notes
http://www.infogineering.net/data-information-knowledge.htm
Knowledge
Firstly, let’s look at Knowledge. Knowledge is what we know. Think of this as the map of the World we build inside our brains. Like a physical map, it helps us knowwhere things are – but it contains more than that. It also contains our beliefs and expectations. “If I do this, I will probably get that.” Crucially, the brain links all these things together into a giant network of ideas, memories, predictions, beliefs, etc.
It is from this “map” that we base our decisions, not the real world itself. Our brains constantly update this map from the signals coming through our eyes, ears, nose, mouth and skin.
You can’t currently store knowledge in anything other than a brain, because a brain connects it all together. Everything is inter-connected in the brain. Computers are not artificial brains. They don’t understand what they are processing, and can’t make independent decisions based upon what you tell them.
There are two sources that the brain uses to build this knowledge - information and data.
Data
Data is/are the facts of the World. For example, take yourself. You may be 5ft tall, have brown hair and blue eyes. All of this is “data”. You have brown hair whether this is written down somewhere or not.
In many ways, data can be thought of as a description of the World. We can perceive this data with our senses, and then the brain can process this.
Human beings have used data as long as we’ve existed to form knowledge of the world.
Until we started using information, all we could use was data directly. If you wanted to know how tall I was, you would have to come and look at me. Our knowledge was limited by our direct experiences.
Information
Information allows us to expand our knowledge beyond the range of our senses. We can capture data in information, then move it about so that other people can access it at different times.
Here is a simple analogy for you.
If I take a picture of you, the photograph is information. But what you look like is data.
I can move the photo of you around, send it to other people via e-mail etc. However, I’m not actually moving you around – or what you look like. I’m simply allowing other people who can’t directly see you from where they are to know what you look like. If I lose or destroy the photo, this doesn’t change how you look.
So, in the case of the lost tax records, the CDs were information. The information was lost, but the data wasn’t. Mrs Jones still lives at 14 Whitewater road, and she was still born on 15th August 1971.
The Infogineering Model (below) explains how these interact…
https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex
Principle 1: Open Data by Default
11) We recognise that free access to, and subsequent re-use of, open data are of significant value to society and the economy.
12) We agree to orient our governments towards open data by default.
13) We recognise that the term government data is meant in the widest sense possible. This could apply to data owned by national, federal, local, or international government bodies, or by the wider public sector.
14) We recognise that there is national and international legislation, in particular pertaining to intellectual property, personally-identifiable and sensitive information, which must be observed.
15) We will:
establish an expectation that all government data be published openly by default, as outlined in this Charter, while recognising that there are legitimate reasons why some data cannot be released.
2.Principle 2: Quality and Quantity
16) We recognise that governments and the public sector hold vast amounts of information that may be of interest to citizens.
17) We also recognise that it may take time to prepare high-quality data, and the importance of consulting with each other and with national, and wider, open data users to identify which data to prioritise for release or improvement.
18) We will:
release high-quality open data that are timely, comprehensive, and accurate. To the extent possible, data will be in their original, unmodified form and at the finest level of granularity available;
ensure that information in the data is written in plain, clear language, so that it can be understood by all, though this Charter does not require translation into other languages;
make sure that data are fully described, so that consumers have sufficient information to understand their strengths, weaknesses, analytical limitations, and security requirements, as well as how to process the data; and
release data as early as possible, allow users to provide feedback, and then continue to make revisions to ensure the highest standards of open data quality are met.
3.Principle 3: Usable by All
19) We agree to release data in a way that helps all people to obtain and re-use it.
20) We recognise that open data should be available free of charge in order to encourage their most widespread use.
21) We agree that when open data are released, it should be done without bureaucratic or administrative barriers, such as registration requirements, which can deter people from accessing the data.
22) We will:
release data in open formats wherever possible, ensuring that the data are available to the widest range of users for the widest range of purposes; and
release as much data as possible, and where it is not possible to offer free access at present, promote the benefits and encourage the allowance of free access to data. In many cases this will include providing data in multiple formats, so that they can be processed by computers and understood by people.
4.Principle 4: Releasing Data for Improved Governance
23) We recognise that the release of open data strengthens our democratic institutions and encourages better policy-making to meets the needs of our citizens. This is true not only in our own countries but across the world.
24) We also recognise that interest in open data is growing in other multilateral organisations and initiatives.
25) We will:
share technical expertise and experience with each other and with other countries across the world so that everyone can reap the benefits of open data; and
be transparent about our own data collection, standards, and publishing processes, by documenting all of these related processes online.
5.Principle 5: Releasing Data for Innovation
26) Recognising the importance of diversity in stimulating creativity and innovation, we agree that the more people and organisations that use our data, the greater the social and economic benefits that will be generated. This is true for both commercial and non-commercial uses.
27) We will:
work to increase open data literacy and encourage people, such as developers of applications and civil society organisations that work in the field of open data promotion, to unlock the value of open data;
empower a future generation of data innovators by providing data in machine-readable formats.
You can contribute to report the status of the PSI of your country
You can contribute to report the status of the PSI of your country
Based oin the results of a crowdsourcing tools
If you want to get more details about the score board the list of indicators used in this score is public and the link is avaiable on the slide
Based oin the results of a crowdsourcing tools
If you want to get more details about the score board the list of indicators used in this score is public and the link is avaiable on the slide
Get more info abuot Tel Aviv
Till now everything is happiness and joy, like these kids in tel aviv, even with a hard weather, they see the potential to enjoy and use the most basic instruments fhat they have at their disposal to have fun, provided that their goal is to have fun..
BUT, working in an open data initialtive can be scary too when time comes, and there are several issues in several categories that we need to deal with if we are to have a great project, let us consider that each issue is a kid, that witll trick or treat us in the project, and that if we are not ready, they will play tricks on us.
Working in a open data project is like halloween, you must know that the kids will come and tick or traet you, and you must be prepared with candies (solutions), and that possibly, each kid will ask for a different kind of candie, actually, for sure they will
Open Nuts!
Open Government Data activities in Austria
Gregor Eibl*, Brigitte Lutz**
How does opening government data compare to opening nuts? Opening government data can
be compared well with the act of opening a nut.
The kernel of the walnut for example, which is protected by the hard shell and is a valuable
food for animals, is a calorie-rich winter food supply for birds, squirrels and other rodents.
Let’s assume that government data is a rich supply for third parties (businesses, NGOs, citizens,
universities, other government agencies…), this information resource is often protected well from
third parties. Making this data available and easy to use is one of the core claims in the debate of
open government. You will see later in this reflection that the principles of open government data
all have the aim to make these valuable information resources easily available.
Just like cracking a nut to access the rich fruit.
In this reflection we will shortly talk about the hard shell, which the open government data
movement will have to crack. First the activities, which shows what Austria has done so far to
crack the shell and remove the barriers and finally the first fruits, which became available through
the efforts of opening the nut itself.
1. The Hard Shell of Open Government Data
Figure 1 depicts some barriers encountered during the implementation of opening government
data in Austrian public administrations. Only through the opening process of government data,
data islands were identified, which had to be integrated and harmonized. Other data sources are consciously kept secret, which explains the resistance of some data
owners. Arguments were raised, that opening can have unpredictable results (overwhelming
feedback, new requirements,…) about the quality of data or new technical features.
On the other hand administrations have to deal with missing resources, like lower budgets in the
time of financial crisis, additional distribution cost with the open data portals and missing human
resources capable of handling the new tasks.
Raw data eliminates the possibility to publish only “censored” and non-critical data and
information. Some administrations have concerns to publish their data in a quality that was
sufficient for internal purposes, but not enough for the broad public audience.
Preparation Phase
Open Data for Geographic datasets
Guidelines deliberation which are an italian best practice
The catalog (experimental)
Start of the project
First group of Data Sources
The catalog on line: http://dati.trentino.it
Activities on organizational, legal, communities issues
Open Data Challenge
Analysis of the semantic entities for the PAT
Join the OKF CKAN community
Platform evolution
New processes development
Data quality and reuse analysis/Impact assessment
Cooperation with / involvement of the local ICT companies
Modeling and implementing the semantic entities,
Semantic Tools development
The Open Knowledge Foundation (OKF) is a non-profit organisation founded in 2004 and dedicated to promoting open data and open content in all their forms – including government data, publicly funded research and public domain cultural content. The OKF CKAN project is the world’s leading open source data portal platform.
CKAN (Comprehensive Knowledge Archive Network) is a powerful data management system that makes data accessible – by providing tools to streamline publishing, sharing, finding and using data. CKAN is aimed at data publishers (national and regional governments, companies and organizations) wanting to make their data open and available.
Member of staff dedicated to to make communities building. Mainly involved with different data providers. The main activities are: event organizations, training courses, meetup, dissemination, social networking (twitter, facebook account). International community involment: openstreetmap, wikipedia, OKF. Trentino partecipate actively to these communities. Last weekend we had the OSMIT event with a mapping party which will involve some of our local public. Administration. Promote the use of openstreetmap by the institution to integrate public geographical information and openstreetmap crowdsourced datasets.
We are going to involve research intitutions to integrate these datasets (added value datasets) and to gather research intitution datasets. For the PA departments, we are going to identify some needs of the Pat reguarding which datasets are interesting for the PA (data is power…). We have many local actors which are going to be involved into our publication process (comunità di valle, municipalities), etc. We have made some courses for the public administration (PA community building) by disseminnating the culture of data among the PA employers to facilitate the realese and understanding of the open data process. We are following the 5 stars model proposed by Tim Davies (Open data research network researcher).
Besides the offline activities we have a social media strategy (ttwitter, Facebook) etc…
Member of staff dedicated to to make communities building. Mainly involved with different data providers. The main activities are: event organizations, training courses, meetup, dissemination, social networking (twitter, facebook account). International community involment: openstreetmap, wikipedia, OKF. Trentino partecipate actively to these communities. Last weekend we had the OSMIT event with a mapping party which will involve some of our local public. Administration. Promote the use of openstreetmap by the institution to integrate public geographical information and openstreetmap crowdsourced datasets.
We are going to involve research intitutions to integrate these datasets (added value datasets) and to gather research intitution datasets. For the PA departments, we are going to identify some needs of the Pat reguarding which datasets are interesting for the PA (data is power…). We have many local actors which are going to be involved into our publication process (comunità di valle, municipalities), etc. We have made some courses for the public administration (PA community building) by disseminnating the culture of data among the PA employers to facilitate the realese and understanding of the open data process. We are following the 5 stars model proposed by Tim Davies (Open data research network researcher).
Besides the offline activities we have a social media strategy (ttwitter, Facebook) etc…
Besides the offline activities we have a social media strategy (ttwitter, Facebook) etc…
The Open Data Institute is catalysing the evolution of open data culture to create economic, environmental, and social value. It helps unlock supply, generates demand, creates and disseminates knowledge to address local and global issues.
We convene world-class experts to collaborate, incubate, nurture and mentor new ideas, and promote innovation. We enable anyone to learn and engage with open data, and empower our teams to help others through professional coaching and mentoring.