This document summarizes a presentation on CKAN, an open-source data management system. It discusses CKAN's features for publishing, finding, and managing datasets. These include adding metadata and data, filtering datasets, previewing data types, and customizing CKAN. It also covers harvesting data from external sources, installing CKAN, and common issues. The goal of CKAN is to make data open and accessible on the web according to the 5 star open data model.

1. CKAN Tutorial
@ odw2013
Presenter: 李承錱 Cheng-Jen Lee (Sol)
Email: cjlee AT iis.sinica.edu.tw
This work is licensed under a
Creative Commons Attribution-ShareAlike 3.0 Taiwan License.

2. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

3. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

4. 2013/11/9
About CKAN
● The Comprehensive Knowledge Archive
Network
● A powerful data management system
● Publishing
● Sharing
● Finding
● Using Data

5. 2013/11/9
About CKAN
61 instances

6. 2013/11/9
CKAN 2

7. 2013/11/9
Feature Tour

8. 2013/11/9
Demo Site
demo.ckan.org

9. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Add Dataset Basic
Information

10. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Add Data
Under the Dataset

11. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Add Metadata
About the Dataset

12. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Filter By
Keywords

13. 2013/11/9
Feature Tour (1)
Publish & Find Datasets
Filter By
Geographical Features

14. 2013/11/9
Feature Tour (2)
Store & Manage Data
Data Explorer:
recline_preview (csv, xls)
json_preview
pdf_preview
ckanext-spatial

15. 2013/11/9
Feature Tour (2)
Store & Manage Data
Graphing data

16. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Share

17. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Organization

18. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Manage Users of
an Organization

19. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Manage Role of Members
Admin: edit datasets & members
Editor: edit datasets
Viewer: view (private) datasets
Note: Public datasets are visible to
everyone

20. 2013/11/9
Feature Tour (3)
Engage with Users & Others
Harvest and Federation

21. 2013/11/9
Feature Tour (3)
Engage with Users & Others
History

22. 2013/11/9
Feature Tour (4)
Customise & Extend
● RESTful JSON APIs
– The Action API
– The DataStore API
– The FileStore API...
● Extensions (over 60)
– ckanext-harvest
– ckanext-spatial
Open source is good!
● Themable
● Integrates with other CMS (ex. Drupal)

23. CKAN and 5 ★ Open Data
2013/11/9

24. CKAN and 5 ★ Open Data
● ★ Make your stuff available on the Web (whatever
format) under an open license
2013/11/9

25. CKAN and 5 ★ Open Data
● ★★ Make it available as structured data (e.g.,
Excel instead of image scan of a table)
★★★ Use non-proprietary formats (e.g., CSV
instead of Excel)
– Accept any data format
– Beautiful data demonstration
2013/11/9

26. CKAN and 5 ★ Open Data
● ★★★★ Use URIs to denote things, so that people
can point at your stuff
– Permanent link for each dataset
– Get Dataset URI through API
● ★★★★★ Link your data to other data to provide
context
– Linked data and RDF for metadata
2013/11/9

27. CKAN and 5 ★ Open Data
● RDF for metadata
– DCAT and Dublin Core
– curl -L -H "Accept:application/rdf+xml"
http://thedatahub.org/dataset/gold-prices
2013/11/9

28. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

29. 2013/11/9
United Kingdom
DATA.GOV.UK

30. 2013/11/9
United States
DATA.GOV

31. 2013/11/9
Brazil
DADOS.GOV.BR

32. 2013/11/9
European Union
PUBLICDATA.EU

33. 2013/11/9
Geospatial Data Explorer:
Lat/Long field

34. 2013/11/9
Geospatial Data Explorer:
GeoJSON

35. 2013/11/9
Geospatial Data Explorer:
WMS

36. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

37. 2013/11/9
System Architecture

38. What You Should Know
● Python, Pylons
● ckan plugins toolkit
● SQLAlchemy, SQL
● HTML, JavaScript
● Babel
● Web Server (Nginx, uWSGI...)
2013/11/9

39. 2013/11/9
Install from Source
● Virtual environment
● Checkout the source (via GIT)
– https://github.com/okfn/ckan
● Create a CKAN config file
● Setup Jetty & Solr
● Initialize Database (user, db)
● Link to who.ini
● Create a Sysadmin User
● Deployment (nginx + uWSGI)
● Install other extensions...

40. 2013/11/9
Installation Notes
● https://ckan-docs-tw.readthedocs.org/

41. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

42. 2013/11/9
Harvesters
● ckanext-harvest
– Remote harvesting extension
– https://github.com/okfn/ckanext-harvest
● Source Type
– CKAN (built-in)
– CSW
– WAF
– Custom (csv/xls/website… etc)

43. 2013/11/9
Harvested from TGOS
CSW service

44. 2013/11/9
Harvesters
http://Mydomain.com/harvest

45. 2013/11/9
Harvesters
Add a new harvest source

46. 2013/11/9
Harvesters
Create a harvest job

47. 2013/11/9
Harvesters
Overview of harvested datasets

48. 2013/11/9
Harvesters
Background Process
● Manually
– (pyenv) $ paster --plugin=ckanext-harvest
harvester gather_consumer -c
/etc/ckan/default/production.ini
– (pyenv) $ paster --plugin=ckanext-harvest
harvester fetch_consumer -c
/etc/ckan/default/production.ini
– (pyenv) $ paster --plugin=ckanext-harvest
harvester run -c
/etc/ckan/default/production.ini

49. 2013/11/9
Harvesters
Background Process
● Automatically
– Supervisor (for gather & fetch consumer)
– Cron (for run)

50. 2013/11/9
Harvesters
Custom harvester
● Implement the harvester interface
to perform harvesting operations
● Three stages
● gather: get the identification
● fetch: fetch the contents
● import: create ckan package(dataset)

51. 2013/11/9
Harvesters
The harvesting interface
from base import HarvesterBase
class SRDAHarvester(HarvesterBase):
def _set_config(self,config_str):
def info(self):
...
def gather_stage(self, harvest_job):
def fetch_stage(self, harvest_object):
def import_stage(self, harvest_object):
See the extension site for details
An example (SRDA): http://goo.gl/ZMnND7

52. 2013/11/9
Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Installation
● Harvesters
● Issues

53. 2013/11/9
Issues
● CJK Support
– CJK Search
– Some broken translations
– Frequent updates
● Extensions compatibilities
● Tons of tweaks needed
● Performance Issue
● Complicated Architecture

54. ● Official Documents:
– http://docs.ckan.org/en/latest/
● Installation Notes (in Chinese):
– https://ckan-docs-tw.readthedocs.org/
● CKAN Development Discussions:
– http://lists.okfn.org/mailman/listinfo/ckan-dev
● CKAN Taiwan Interest Group:
– https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group
2013/11/9
Resources

55. Thanks for your attention!
2013/11/9
Any Q?
Email: u10313335 AT citi.sinica.edu.tw
http://about.me/sollee