Open Data and CKAN Data Catalogues

Open Data & coding data.gov.uk
David Read
Open Knowledge Foundation [email_address]

Contents
The context: Linked Open Data
Our data catalogue: CKAN
data.gov.uk using CKAN
Discussion

Open Data
”Data is expensive to create” ”But think of the mutual benefits of it being open”
Accessible Allowed to use and republish Without restiction

Science UEA criticised for a "culture of withholding information." CC-BY-SA http://commons.wikimedia.org/wiki/User:ChrisO

Geographic data

Public data

Linking data Dr. Hans Rosling, Professor of Global Health, Karolinska Institute, Sweden (TED talk)

Linking data 2

Linked data

Opening government data
Transparency --> effectiveness
Labour and Conservatives agree (!) with Cambridge economists:
Making government datasets public will bring a £6bn boost to UK economy
(We have paid for it...)

Open Data and Open Software
Zero cost
Good performance
Principles: Many hands make light work / natural selection / wisdom of crowd / on shoulders of giants
Not a proprietry format
No supplier lock-in

Infrastructure Software Data Licence GPL PDDL, ODbL, ODC-By (OKF 2007-) isitopendata.org (OKF 2009-) Modules/Linking Lib, egg Spreadsheet, database, RDF/OWL Human Discovery CKAN (OKF 2008-) Automatic Distribution Apt-get, CPAN, easy_install CKAN datapkg (OKF 2008-) Hosting Sourceforge, PyPI, bitbucket archive.org / knowledgeforge.net Community freshmeat data.gov.uk email list closest?

Open Knowledge Foundation
Aim: promote Open Knowledge
Founded 2004 as a 'not for profit' organisation
Strong connections with Cambridge University
A key director: Rufus Pollock
Volunteer driven
Create software tools (CKAN, KnowledgeForge), organise conferences, licenses, create visuals & mash-ups (Where Does My Money Go, Open Shakespeare), campaigns (Panton Principals)

Introducing... CKAN
”Comprehensive Knowledge Archive Network” ...well... a fancy Data Catalog
”CKAN is a registry or catalogue system for datasets or other "knowledge" resources. CKAN aims to make it easy to find, share and reuse open content and data, especially in ways that are machine automatable.”

Dataset name title version url author licence notes extras Tag name Resource url format description hash CKAN data model * * * Group name title description * *

API
REST
$ curl http://ckan.net/api/rest/package ["2000-us-census-rdf", "32000-naples-florida-businesses-kml", "aaoe-87", "acawiki", "adb-sdbs", "addgene", "adopt-a-roadside", …
Search
$ curl "http://ckan.net/api/search/resource?url=.fr&all_fields=1" {"count": 6, "results": [{"id": "819c811c-7afc-4d4f-a7f8-aca0b2a84df5", "package_id": "0ad0dbb9-e1b7-43d6-9fae-ca92a889e871", "url": "http://www.frst.govt.nz/funding/futurefunding", "format": "Spreadsheet", "description": "Future funding (FRST): Spreadsheet", "hash": "", "position": 0}, {"id": ... $ curl http://ckan.net/api/rest/package/coins-data {"id": "78eccf9d-d5b3-4dbd-8ada-6801cfd7e4c8", "name": "coins-data", "title": "COINS data", "version": null, "url": "http://data.gov.uk/dataset/coins", "author": "HM Treasury (UK Government)", "author_email": null, "maintainer": null, "maintainer_email": null, "notes": "### AboutrnrnThe UK Government's HM Trea...

Datapkg (dpm)
Getting a data package
$ datapkg index-add file:///.... $ datapkg update $ datapkg search "military spending" military: Military Spending 1890-1914 military-norm: Military Spending 1890-1914 (normalized) $ datapkg install military-norm Downloading military-norm and dependencies. $ datapkg plot military $ datapkg create military-uk-usa table.csv ”Military spending UK vs USA” $ datapkg register military-uk-usa
Upload derivative data

CKAN communities
Europe: Austria, Hungary, Germany, Italy, Finland, Netherlands, France, Norway
North America: Colorado, Canada
Australasia: New Zealand

Sharing metadata ckan.net canada.ckan.net it.ckan.net no.ckan.net data.gov.no data.gov.it

Architecture view controller model Drupal front-end Pylons front-end (genshi, routes, repoze.who) Vdm - Versioned Domain Model Postgres REST & Search APIs sqlalchemy Logic layer Data import scripts SOLR Search Export scripts repoze.who Atom feeds sqlalchemy-migrate

data.gov.uk
Gordon Brown invited Tim Berners-Lee for exciting digital plans
David Cameron supportive
Run by Antonio Acuna at the Cabinet Office, aided by The National Archives
”Raw Data Now”, then improve and link
COI team produce Drupal bulk of data.gov.uk pages with OKFN producing 'data' page in CKAN

Measuring success
Stats: users, number of datasets, per department, big wins: Ordnance Survey, Coins, top public salaries
Creation of visualisations, apps, linked data, news stories, companies - £6bn
CKAN – similar goals
What do you think?

Open Data and CKAN Data Catalogues

by david-read on Oct 27, 2011

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Open Data and CKAN Data Catalogues Presentation Transcript