Techniques and principles to build creative applications based on Opendata. Overview of already developed solutions, modules and succesful case histories in the Public/Government sector

1. When Drupal meets Opendata
PHPDay 2011 - Social tag #phpday

2. Chi siamo?

3. Chi siamo?
Stefano Mainardi,
sviluppatore PHP, CEO di
Twinbit e fondatore del
network italiano ILDN
Piacere di conoscervi!
stefano@twinbit.it
twitter : @stefanomainardi

4. Chi siamo?
Paolo Mainardi, sviluppatore
PHP, CTO di Twinbit e Stefano Mainardi,
fondatore del network italiano sviluppatore PHP, CEO di
ILDN Twinbit e fondatore del
network italiano ILDN
Piacere di conoscervi!
Piacere di conoscervi!
paolo@twinbit.it
stefano@twinbit.it
twitter : @paolomainardi
twitter : @stefanomainardi

5. Chi siamo?
Paolo Mainardi, sviluppatore
PHP, CTO di Twinbit e Stefano Mainardi,
fondatore del network italiano sviluppatore PHP, CEO di
ILDN Twinbit e fondatore del
network italiano ILDN
Piacere di conoscervi!
Piacere di conoscervi!
paolo@twinbit.it
stefano@twinbit.it
twitter : paolomainardi
twitter : stefanomainardi
Piacere, io sono Drupal! :-)

6. Quanti di voi utilizzano Drupal?

7. Cosa sono gli “Open Data”?

8. definizione “accademica”
Con Open Data si fa riferimento ad una filosofia, che è al tempo
stesso una pratica, che implica che alcune tipologie di dati siano
liberamente accessibili a tutti sul Web, senza restrizioni di copyright,
brevetti o altre forme di controllo che ne limitino la riproduzione.
da Wikipedia

9. e quindi?

10. http://www.flickr.com/photos/f7oor/405046410/

11. Data is a precious thing and will last longer than the systems
themselves.
Tim Berners-Lee
http://www.flickr.com/photos/f7oor/405046410/ 1/2

12. I dati sono noiose scatole marroni, ed è così che ce li
immaginiamo, no? Perché i dati di per sé non sono di
immediata applicazione Ma in realtà, i dati determinano
tantissime cose nelle nostre vite e ciò accade perché c’è
qualcuno che prende quei dati e ne fa qualcosa.
credits: Ernesto Bellisario “Come si fa OpenData”
Tim Berners-Lee
http://www.flickr.com/photos/f7oor/405046410/ 2/2

13. “Economia della conoscenza”

19. ma in italia cosa succede?

22. il movimento OpenData italiano
http://linkedopendata.it/
http://spaghettiopendata.org/
http://datagov.it/
http://www.agoradigitale.org/
http://www.openpolis.it/

24. quali formati dobbiamo utilizzare?

25. .KML .XML .CSV .TXT .GTFS .JSON

26. .KML .XML .CSV .TXT .GTFS .JSON
insomma, formati standard che siano machine readable
l’importante è non far nascere un flame sui formati :-)

27. .KML .XML .CSV .TXT .GTFS .JSON

28. .KML .XML .CSV .TXT .GTFS .JSON
Mobile Apps

29. .KML .XML .CSV .TXT .GTFS .JSON
Mobile Apps Web Apps

30. .KML .XML .CSV .TXT .GTFS .JSON
Mobile Apps Web Apps Data API

31. I dati sono il carburante, le applicazioni sono il mezzo

33. 47 web, iPhone & Facebook Apps
$2,300,000 risparmiati
$50,000 il costo delle applicazioni
+5000% in ROI

39. Perchè Drupal ?

40. Perchè Drupal ?
• Una piattaforma pensata per fare Open Data (D7 RDFa, Nodes, Taxonomy)

41. Perchè Drupal ?
• Una piattaforma pensata per fare Open Data (D7 RDFa, Nodes, Taxonomy)
• Semplici interfacce di amministrazione per utenti non esperti (D7 UX)

42. Perchè Drupal ?
• Una piattaforma pensata per fare Open Data (D7 RDFa, Nodes, Taxonomy)
• Semplici interfacce di amministrazione per utenti non esperti (D7 UX)
• Semplice integrazione con i servizi di terze parti (CKAN, OpenCalais, OSM, Mapbox ecc..)

43. Perchè Drupal ?
• Una piattaforma pensata per fare Open Data (D7 RDFa, Nodes, Taxonomy)
• Semplici interfacce di amministrazione per utenti non esperti (D7 UX)
• Semplice integrazione con i servizi di terze parti (CKAN, OpenCalais, OSM, Mapbox ecc..)
• Importanti case history governative di successo (List of Drupal sites in
Government: http://groups.drupal.org/node/19885)

44. Perchè Drupal ?

45. Perchè Drupal ?
•CCK - Fields in core

46. Perchè Drupal ?
•CCK - Fields in core
•Feeds

47. Perchè Drupal ?
•CCK - Fields in core
•Feeds
•Views - Views datasource

48. Perchè Drupal ?
•CCK - Fields in core
•Feeds
•Views - Views datasource
•Geo Modules (Openlayers, Geofield)

49. Perchè Drupal ?
•CCK - Fields in core
•Feeds
•Views - Views datasource
•Geo Modules (Openlayers, Geofield)
•Supporto nativo per i Linked Data (RDFa)

50. Perchè Drupal ?
•CCK - Fields in core
•Feeds
•Views - Views datasource
•Geo Modules (Openlayers, Geofield)
•Supporto nativo per i Linked Data (RDFa)
•SPARQL

51. Case history di successo
http://data.worldbank.org

52. Case history di successo
http://www.aquaknow.net

53. Case history di successo
http://data.gov.uk
CKAN Dataset
Faceted Search by Apache Solr

54. Case history di successo
http://www.openbelgium.be

55. Phase 1: Creare Open Data
Qualsiasi contenuto è potenzialmente un OpenData.....

56. Phase 1: Creare Open Data
Qualsiasi contenuto è potenzialmente un OpenData.....
“A piece of content or data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the
requirement to attribute and share alike.” (Open Knowledge Definition)

57. Phase 1: Creare Open Data
Qualsiasi contenuto è potenzialmente un OpenData.....
“A piece of content or data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the
requirement to attribute and share alike.” (Open Knowledge Definition)
Seguiamo queste semplici regole:

58. Phase 1: Creare Open Data
Qualsiasi contenuto è potenzialmente un OpenData.....
“A piece of content or data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the
requirement to attribute and share alike.” (Open Knowledge Definition)
Seguiamo queste semplici regole:
•Specificare la fonte

59. Phase 1: Creare Open Data
Qualsiasi contenuto è potenzialmente un OpenData.....
“A piece of content or data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the
requirement to attribute and share alike.” (Open Knowledge Definition)
Seguiamo queste semplici regole:
•Specificare la fonte
•Usare una licenza aperta

60. Phase 1: Creare Open Data
Qualsiasi contenuto è potenzialmente un OpenData.....
“A piece of content or data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the
requirement to attribute and share alike.” (Open Knowledge Definition)
Seguiamo queste semplici regole:
•Specificare la fonte
•Usare una licenza aperta
•Usare formati aperti e semplici da processare automaticamente (CSV, XLS,
XML, JSON)

61. Phase 1: Creare Open Data
Qualsiasi contenuto è potenzialmente un OpenData.....
“A piece of content or data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the
requirement to attribute and share alike.” (Open Knowledge Definition)
Seguiamo queste semplici regole:
•Specificare la fonte
•Usare una licenza aperta
•Usare formati aperti e semplici da processare automaticamente (CSV, XLS,
XML, JSON)
OpenDefinition: http://www.opendefinition.org/okd/
Open Data Commons: http://www.opendatacommons.org
Cretive Commons: http://www.opendatacommons.org

62. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario

63. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario

64. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario
DataTypes Vocabulary
License Vocabulary

65. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario
DataTypes Vocabulary
License Vocabulary
Fields

66. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario

67. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario
Open data creation form

68. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario
Open data creation form

69. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario
Open data creation form
Taxonomy

70. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario
Open data creation form
Taxonomy
Data properties

71. Phase 1: Creare Open Data
Creare un nuovo content type ed un vocabolario
Open data creation form
Taxonomy
Data properties
Comments

72. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?

73. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?
core
x/html

74. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?
core
x/html
RDFa

75. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?
core
x/html RSS
RDFa

76. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?
core contrib modules
x/html RSS json
xml
rdf
RDFa csv

77. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?
core contrib modules
x/html RSS json
xml
rdf
RDFa csv
Views + Views Datasource

78. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?
core contrib modules
charts
x/html RSS json kml {$other formats}
xml
rdf
RDFa csv
Views + Views Datasource

79. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?
core contrib modules
charts
x/html RSS json kml {$other formats}
xml
rdf
RDFa csv http://drupal.org/project/modules
Views + Views Datasource

80. Phase 2: Distribuire Open Data
Come posso distribuire i miei dati ?
Posso essere anche solo un servizio per la tua applicazione:
http://drupal.org/project/services
XMLRPC, JSON, JSON-RPC, REST, SOAP, AMF, etc.
core contrib modules
charts
x/html RSS json kml {$other formats}
xml
rdf
RDFa csv http://drupal.org/project/modules
Views + Views Datasource

81. Phase 2: Distribuire Open Data
Views Example: Create a new display

82. Phase 2: Distribuire Open Data
Views Example: Create a new display
Filter by “data” type contents

83. Phase 2: Distribuire Open Data
Views Example: Create a new display
Select fields to print
Filter by “data” type contents

84. Phase 2: Distribuire Open Data
Views Example: Create a new display
Print results on a table
Select fields to print
Filter by “data” type contents

85. Phase 2: Distribuire Open Data
Views Example: Create a new display
Print results on a table
Select fields to print
Filter by “data” type contents
Expose search fields

86. Phase 2: Distribuire Open Data
Views Example: Create a new display
Output type
Print results on a table
Select fields to print
Filter by “data” type contents
Expose search fields

87. Phase 2: Distribuire Open Data
Views Example: Results

88. Phase 2: Distribuire Open Data
Views Example: Results
Results

89. Phase 2: Distribuire Open Data
Views Example: Results
Exposed form
Results

90. Phase 2: Distribuire Open Data
Views Example: Results
Exposed form
Results
RSS Export

91. Phase 2: Distribuire Open Data
Views Example: Results
Exposed form
Results
RSS Export Click to download in other formats!

92. Phase 2: Distribuire Open Data
Example: Geo tools

93. Phase 2: Distribuire Open Data
Openlayers module
Example: Geo tools

94. Phase 2: Distribuire Open Data
Openlayers module
Example: Geo tools
Drupal nodes

95. Phase 2: Distribuire Open Data
Example: Geo tools
Layer manager

96. Phase 2: Distribuire Open Data
Example: Geo tools
Layer manager

97. Phase 2: Distribuire Open Data
Example: Geo tools Views Datasource (json)
Layer manager

98. Phase 2: Distribuire Open Data
Example: Apache Solr -- http://drupal.org/project/apachesolr

99. Phase 2: Distribuire Open Data
Used on Drupal.org Example: Apache Solr -- http://drupal.org/project/apachesolr

100. Phase 2: Distribuire Open Data
Used on Drupal.org Example: Apache Solr -- http://drupal.org/project/apachesolr
Faceted Search

101. Phase 2: Distribuire Open Data
Used on Drupal.org Example: Apache Solr -- http://drupal.org/project/apachesolr
• Powerful text extraction Faceted Search
• Multisite search
• Attachment search

102. Phase 3: Data processing
Feeds: http://drupal.org/project/feeds

103. Phase 3: Data processing
Feeds: http://drupal.org/project/feeds
Pluggable fetcher

104. Phase 3: Data processing
Feeds: http://drupal.org/project/feeds
Pluggable fetcher
Pluggable parser

105. Phase 3: Data processing
Feeds: http://drupal.org/project/feeds
Pluggable fetcher
Pluggable parser
Pluggable processors

106. Phase 3: Data processing
Feeds: http://drupal.org/project/feeds
Pluggable fetcher
Pluggable parser
Pluggable processors

107. Phase 3: Data processing
Feeds Parser
Feeds Xpath Parser: http://drupal.org/project/feeds_xpathparser
Feeds QueryPath Parser: http://drupal.org/project/feeds_querypath_parser
Data importing projects
http://drupal.org/project/migrate
“The migrate module provides a flexible framework for migrating content into Drupal from other sources (e.g., when converting a web site from another CMS to Drupal)”
http://drupal.org/project/data
“Data module helps you model, manage and query related sets of tables. It offers an administration interface and a low level API for manipulating tables and accessing their contents. D
module provides Views integration for displaying table data and Drupal search integration for searching table content.”

108. Linked Data
http://groups.drupal.org/semantic-web
http://www.w3.org/2001/sw/

109. RDF
http://openspring.net/blog/2011/05/01/background-research-work-leading-to-rdf-in-drupal-7-released-as-part-of-my-masters

110. RDF
http://drupal.org/project/rdfx
http://drupal.org/project/rdf (d6)
API http://api.drupal.org/api/drupal/modules--rdf--rdf.module/group/rdf/7

111. RDF
http://drupal.org/project/rdfx
http://drupal.org/project/rdf (d6)
API http://api.drupal.org/api/drupal/modules--rdf--rdf.module/group/rdf/7
• RDFx: extends core RDF support by providing extra APIs and additional serialization formats such as RDF/XML,
NTriples, Turtle...

112. RDF
http://drupal.org/project/rdfx
http://drupal.org/project/rdf (d6)
API http://api.drupal.org/api/drupal/modules--rdf--rdf.module/group/rdf/7
• RDFx: extends core RDF support by providing extra APIs and additional serialization formats such as RDF/XML,
NTriples, Turtle...
• RDF UI: allows site administrators to specify the RDF mappings via a user interface

113. RDF
http://drupal.org/project/rdfx
http://drupal.org/project/rdf (d6)
API http://api.drupal.org/api/drupal/modules--rdf--rdf.module/group/rdf/7
• RDFx: extends core RDF support by providing extra APIs and additional serialization formats such as RDF/XML,
NTriples, Turtle...
• RDF UI: allows site administrators to specify the RDF mappings via a user interface
• Evoc: user interface to import vocabularies which can be used in RDF UI

114. RDF
http://drupal.org/project/rdfx
http://drupal.org/project/rdf (d6)
API http://api.drupal.org/api/drupal/modules--rdf--rdf.module/group/rdf/7
• RDFx: extends core RDF support by providing extra APIs and additional serialization formats such as RDF/XML,
NTriples, Turtle...
• RDF UI: allows site administrators to specify the RDF mappings via a user interface
• Evoc: user interface to import vocabularies which can be used in RDF UI
This RDF data can be made available in a SPARQL endpoint with the SPARQL module.

115. SPARQL
http://drupal.org/project/sparql
SPARQL is a query language for pattern matching against RDF graphs. The syntax resembles SQL, making for a friendly learning
curve, but SPARQL is far more powerful, enabling queries spanning multiple disparate (local or remote) data sources containing
heterogeneous semistructured data.

116. SPARQL
http://drupal.org/project/sparql
SPARQL is a query language for pattern matching against RDF graphs. The syntax resembles SQL, making for a friendly learning
curve, but SPARQL is far more powerful, enabling queries spanning multiple disparate (local or remote) data sources containing
heterogeneous semistructured data.
•SPARQL API module

117. SPARQL
http://drupal.org/project/sparql
SPARQL is a query language for pattern matching against RDF graphs. The syntax resembles SQL, making for a friendly learning
curve, but SPARQL is far more powerful, enabling queries spanning multiple disparate (local or remote) data sources containing
heterogeneous semistructured data.
•SPARQL API module
•SPARQL Registry module

118. SPARQL
http://drupal.org/project/sparql
SPARQL is a query language for pattern matching against RDF graphs. The syntax resembles SQL, making for a friendly learning
curve, but SPARQL is far more powerful, enabling queries spanning multiple disparate (local or remote) data sources containing
heterogeneous semistructured data.
•SPARQL API module
•SPARQL Registry module
•SPARQL Endpoint module

119. LOD Cloud
“Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/”

120. References
The Semantic Web, Linked Data and Drupal, Part 1: Expose your data using RDF
http://www.ibm.com/developerworks/web/library/wa-rdf/index.html?ca=drs-
The Semantic Web, Linked Data and Drupal, Part 2: Combine linked datasets with Drupal 7 and SPARQL Views
http://www.ibm.com/developerworks/web/library/wa-rdf/index.html?ca=drs-
Background research work leading to RDF in Drupal 7 released as part of my Master's thesis
http://openspring.net/blog/2011/05/01/background-research-work-leading-to-rdf-in-drupal-7-released-as-part-of-my-masters
Background research work leading to RDF in Drupal 7 released as part of my Master's thesis
http://openspring.net/blog/2011/05/01/background-research-work-leading-to-rdf-in-drupal-7-released-as-part-of-my-masters
“Hi, I'm Lin Clark. I support open source and linked data.”
http://lin-clark.com/

121. one more thing
è in arrivo il primo contest italiano sugli OpenData
http://www.appsforitaly.org/

122. Questions?

123. Rimaniamo in contatto!
Stefano Mainardi Paolo Mainardi
stefano@twinbit.it paolo@twinbit.it
twitter : @stefanomainardi twitter : @paolomainardi

124. Grazie!