This document outlines an agenda and presentation on CKAN, an open-source data management system. The presentation covers an introduction to CKAN, a tour of its features for publishing, finding, and managing data, how it supports open data principles, examples of CKAN instances, issues, and installation and harvesting topics.

1. CKAN 2 Introduction
Presenter: 李承錱 Cheng-Jen Lee (Sol)
Email: cjlee AT iis.sinica.edu.tw
This work is licensed under a
Creative Commons Attribution-ShareAlike 3.0 Taiwan License.

2. Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Issues
● Extended Topic 1: Installation
● Extended Topic 2: Harvesters
2014/5/22 2

3. Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Issues
● Extended Topic 1: Installation
● Extended Topic 2: Harvesters
2014/5/22 3

4. About CKAN
● The Comprehensive Knowledge Archive
Network
● A powerful data management system
● Publishing
● Sharing
● Finding
● Using Data
2014/5/22 4

5. About CKAN
83 instances
around the world
in May 2014
2014/5/22 5

6. CKAN 2
2014/5/22 6

7. Feature Tour
2014/5/22 7

8. Demo Site
demo.ckan.org
2014/5/22 8

9. Feature Tour (1)
Publish & Find Datasets
Add Dataset Basic
Information
2014/5/22 9

10. Feature Tour (1)
Publish & Find Datasets
Add Data
Under the Dataset
2014/5/22 10

11. Feature Tour (1)
Publish & Find Datasets
Add Metadata
About the Dataset
2014/5/22 11

12. Feature Tour (1)
Publish & Find Datasets
Filter By
Keywords
2014/5/22 12

13. Feature Tour (1)
Publish & Find Datasets
Filter By
Geographical Features
2014/5/22 13

14. Feature Tour (2)
Store & Manage Data
Data Explorer:
recline_preview (csv, xls)
json_preview
pdf_preview
ckanext-spatial
2014/5/22 14

15. Feature Tour (2)
Store & Manage Data
Graphing data
2014/5/22 15

16. Feature Tour (3)
Engage with Users & Others
Share
2014/5/22 16

17. Feature Tour (3)
Engage with Users & Others
Organization
2014/5/22 17

18. Feature Tour (3)
Engage with Users & Others
Manage Users of
an Organization
2014/5/22 18

19. Feature Tour (3)
Engage with Users & Others
Manage Role of Members
Admin: edit datasets & members
Editor: edit datasets
Viewer: view (private) datasets
Note: Public datasets are visible to
everyone
2014/5/22 19

20. Feature Tour (3)
Engage with Users & Others
Harvest and Federation
2014/5/22 20

21. Feature Tour (3)
Engage with Users & Others
History
2014/5/22 21

22. Feature Tour (4)
Customise & Extend
● RESTful JSON APIs
– The Action API
– The DataStore API
– The FileStore API...
● Extensions (over 60)
– ckanext-harvest
– ckanext-spatial
Open source is good!
● Themable
● Integrates with other CMS (ex. Drupal)
2014/5/22 22

23. CKAN and 5 ★ Open Data
2014/5/22 23

24. CKAN and 5 ★ Open Data
● ★ Make your stuff available on the Web (whatever
format) under an open license
2014/5/22 24

25. CKAN and 5 ★ Open Data
● ★★ Make it available as structured data (e.g.,
Excel instead of image scan of a table)
★★★ Use non-proprietary formats (e.g., CSV
instead of Excel)
– Accept any data format
– Beautiful data demonstration
– Datastore : Indexing for structure data
2014/5/22 25

26. CKAN and 5 ★ Open Data
● ★★★★ Use URIs to denote things, so that people
can point at your stuff
– Permanent link for each dataset
– Get Dataset URI through API
● ★★★★★ Link your data to other data to provide
context
– Linked data and RDF for metadata
2014/5/22 26

27. CKAN and 5 ★ Open Data
● RDF for metadata
– DCAT and Dublin Core
– curl -L -H "Accept:application/rdf+xml"
http://thedatahub.org/dataset/gold-prices
2014/5/22 27

28. Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Issues
● Extended Topic 1: Installation
● Extended Topic 2: Harvesters
2014/5/22 28

29. United Kingdom
DATA.GOV.UK
2014/5/22 29

30. United States
DATA.GOV
2014/5/22 30

31. Brazil
DADOS.GOV.BR
2014/5/22 31

32. European Union
PUBLICDATA.EU
2014/5/22 32

33. Geospatial Data Explorer:
Lat/Long field
2014/5/22 33

34. Geospatial Data Explorer:
GeoJSON
2014/5/22 34

35. Geospatial Data Explorer:
WMS
2014/5/22 35

36. Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Issues
● Extended Topic 1: Installation
● Extended Topic 2: Harvesters
2014/5/22 36

37. Issues
● CJK Support
– CJK Search
– Some broken translations
– Frequent updates
● Extensions compatibilities
● Tons of tweaks needed
● Performance Issue
● Complicated Architecture
2014/5/22 37

38. System Architecture
2014/5/22 38

39. Issues (Cont'd)
● What You Should Know
– Python & Pylons
– ckan plugins toolkit
– SQLAlchemy & SQL
– HTML, JavaScript
– Babel
– Web Server (Nginx, uWSGI...)
2014/5/22 39

40. Resources
● Official Documents:
– http://docs.ckan.org/en/latest/
● Installation Notes (in Chinese):
– https://ckan-docs-tw.readthedocs.org/
● CKAN Development Discussions:
– http://lists.okfn.org/mailman/listinfo/ckan-dev
● CKAN Taiwan Interest Group:
– https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group
2014/5/22 40

41. Thanks for your attention!
Any Q?
Email: u10313335 AT citi.sinica.edu.tw
http://about.me/sollee
2014/5/22 41

42. Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Issues
● Extended Topic 1: Installation
● Extended Topic 2: Harvesters
2014/5/22 42

43. Install from Source
● Virtual environment
● Checkout the source (via GIT)
– https://github.com/okfn/ckan
● Create a CKAN config file
● Setup Jetty & Solr
● Initialize Database (user, db)
● Link to who.ini
● Create a Sysadmin User
● Deployment (nginx + uWSGI)
● Install other extensions...
2014/5/22 43

44. Installation Notes
● https://ckan-docs-tw.readthedocs.org/
2014/5/22 44

45. Agenda
● About CKAN
● Feature Tour
– Publish & Find Datasets
– Store & Manage Data
– Engage with Users & Others
– Customise & Extend
● CKAN and 5 ★ Open Data
● Showcase
● Issues
● Extended Topic 1: Installation
● Extended Topic 2: Harvesters
2014/5/22 45

46. Harvesters
● ckanext-harvest
– Remote harvesting extension
– https://github.com/okfn/ckanext-harvest
● Source Type
– CKAN (built-in)
– CSW
– WAF
– Custom (csv/xls/website… etc)
2014/5/22 46

47. Harvested from TGOS
CSW service
2014/5/22 47

48. Harvesters
http://Mydomain.com/harvest
2014/5/22 48

49. Harvesters
Add a new harvest source
2014/5/22 49

50. Harvesters
Create a harvest job
2014/5/22 50

51. Harvesters
Overview of harvested datasets
2014/5/22 51

52. Harvesters
Background Process
● Manually
– (pyenv) $ paster --plugin=ckanext-harvest
harvester gather_consumer -c
/etc/ckan/default/production.ini
– (pyenv) $ paster --plugin=ckanext-harvest
harvester fetch_consumer -c
/etc/ckan/default/production.ini
– (pyenv) $ paster --plugin=ckanext-harvest
harvester run -c
/etc/ckan/default/production.ini
2014/5/22 52

53. Harvesters
Background Process
● Automatically
– Supervisor (for gather & fetch consumer)
– Cron (for run)
2014/5/22 53

54. Harvesters
Custom harvester
● Implement the harvester interface
to perform harvesting operations
● Three stages
● gather: get the identification
● fetch: fetch the contents
● import: create ckan package(dataset)
2014/5/22 54

55. Harvesters
The harvesting interface
from base import HarvesterBase
class SRDAHarvester(HarvesterBase):
def _set_config(self,config_str):
def info(self):
...
def gather_stage(self, harvest_job):
def fetch_stage(self, harvest_object):
def import_stage(self, harvest_object):
See the extension site for details
An example (SRDA): http://goo.gl/ZMnND7
2014/5/22 55