0
CKAN 2 Introduction 
Presenter: 李承錱 Cheng-Jen Lee (Sol) 
Email: cjlee AT iis.sinica.edu.tw 
This work is licensed under a ...
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
About CKAN 
● The Comprehensive Knowledge Archive 
Network 
● A powerful data management system 
● Publishing 
● Sharing 
...
About CKAN 
83 instances 
around the world 
in May 2014 
2014/5/22 5
CKAN 2 
2014/5/22 6
Feature Tour 
2014/5/22 7
Demo Site 
demo.ckan.org 
2014/5/22 8
Feature Tour (1) 
Publish & Find Datasets 
Add Dataset Basic 
Information 
2014/5/22 9
Feature Tour (1) 
Publish & Find Datasets 
Add Data 
Under the Dataset 
2014/5/22 10
Feature Tour (1) 
Publish & Find Datasets 
Add Metadata 
About the Dataset 
2014/5/22 11
Feature Tour (1) 
Publish & Find Datasets 
Filter By 
Keywords 
2014/5/22 12
Feature Tour (1) 
Publish & Find Datasets 
Filter By 
Geographical Features 
2014/5/22 13
Feature Tour (2) 
Store & Manage Data 
Data Explorer: 
recline_preview (csv, xls) 
json_preview 
pdf_preview 
ckanext-spat...
Feature Tour (2) 
Store & Manage Data 
Graphing data 
2014/5/22 15
Feature Tour (3) 
Engage with Users & Others 
Share 
2014/5/22 16
Feature Tour (3) 
Engage with Users & Others 
Organization 
2014/5/22 17
Feature Tour (3) 
Engage with Users & Others 
Manage Users of 
an Organization 
2014/5/22 18
Feature Tour (3) 
Engage with Users & Others 
Manage Role of Members 
Admin: edit datasets & members 
Editor: edit dataset...
Feature Tour (3) 
Engage with Users & Others 
Harvest and Federation 
2014/5/22 20
Feature Tour (3) 
Engage with Users & Others 
History 
2014/5/22 21
Feature Tour (4) 
Customise & Extend 
● RESTful JSON APIs 
– The Action API 
– The DataStore API 
– The FileStore API... 
...
CKAN and 5 ★ Open Data 
2014/5/22 23
CKAN and 5 ★ Open Data 
● ★ Make your stuff available on the Web (whatever 
format) under an open license 
2014/5/22 24
CKAN and 5 ★ Open Data 
● ★★ Make it available as structured data (e.g., 
Excel instead of image scan of a table) 
★★★ Use...
CKAN and 5 ★ Open Data 
● ★★★★ Use URIs to denote things, so that people 
can point at your stuff 
– Permanent link for ea...
CKAN and 5 ★ Open Data 
● RDF for metadata 
– DCAT and Dublin Core 
– curl -L -H "Accept:application/rdf+xml" 
http://thed...
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
United Kingdom 
DATA.GOV.UK 
2014/5/22 29
United States 
DATA.GOV 
2014/5/22 30
Brazil 
DADOS.GOV.BR 
2014/5/22 31
European Union 
PUBLICDATA.EU 
2014/5/22 32
Geospatial Data Explorer: 
Lat/Long field 
2014/5/22 33
Geospatial Data Explorer: 
GeoJSON 
2014/5/22 34
Geospatial Data Explorer: 
WMS 
2014/5/22 35
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
Issues 
● CJK Support 
– CJK Search 
– Some broken translations 
– Frequent updates 
● Extensions compatibilities 
● Tons ...
System Architecture 
2014/5/22 38
Issues (Cont'd) 
● What You Should Know 
– Python & Pylons 
– ckan plugins toolkit 
– SQLAlchemy & SQL 
– HTML, JavaScript...
Resources 
● Official Documents: 
– http://docs.ckan.org/en/latest/ 
● Installation Notes (in Chinese): 
– https://ckan-do...
Thanks for your attention! 
Any Q? 
Email: u10313335 AT citi.sinica.edu.tw 
http://about.me/sollee 
2014/5/22 41
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
Install from Source 
● Virtual environment 
● Checkout the source (via GIT) 
– https://github.com/okfn/ckan 
● Create a CK...
Installation Notes 
● https://ckan-docs-tw.readthedocs.org/ 
2014/5/22 44
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
Harvesters 
● ckanext-harvest 
– Remote harvesting extension 
– https://github.com/okfn/ckanext-harvest 
● Source Type 
– ...
Harvested from TGOS 
CSW service 
2014/5/22 47
Harvesters 
http://Mydomain.com/harvest 
2014/5/22 48
Harvesters 
Add a new harvest source 
2014/5/22 49
Harvesters 
Create a harvest job 
2014/5/22 50
Harvesters 
Overview of harvested datasets 
2014/5/22 51
Harvesters 
Background Process 
● Manually 
– (pyenv) $ paster --plugin=ckanext-harvest 
harvester gather_consumer -c 
/et...
Harvesters 
Background Process 
● Automatically 
– Supervisor (for gather & fetch consumer) 
– Cron (for run) 
2014/5/22 5...
Harvesters 
Custom harvester 
● Implement the harvester interface 
to perform harvesting operations 
● Three stages 
● gat...
Harvesters 
The harvesting interface 
from base import HarvesterBase 
class SRDAHarvester(HarvesterBase): 
def _set_config...
Upcoming SlideShare
Loading in...5
×

ckan 2.0 Introduction (20140522 updated)

210

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
210
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 由 Open Knowledge Foundation 支持
  • 由 Open Knowledge Foundation 支持
  • 公布網址
  • Data.gov.uk 即是 drupal+CKAN
  • Store the raw data and metadata. Visualise structured data with interactive tables, graphs and maps.
  • TGOS: 國土資訊圖資服務平台
  • Transcript of "ckan 2.0 Introduction (20140522 updated)"

    1. 1. CKAN 2 Introduction Presenter: 李承錱 Cheng-Jen Lee (Sol) Email: cjlee AT iis.sinica.edu.tw This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Taiwan License.
    2. 2. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 2
    3. 3. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 3
    4. 4. About CKAN ● The Comprehensive Knowledge Archive Network ● A powerful data management system ● Publishing ● Sharing ● Finding ● Using Data 2014/5/22 4
    5. 5. About CKAN 83 instances around the world in May 2014 2014/5/22 5
    6. 6. CKAN 2 2014/5/22 6
    7. 7. Feature Tour 2014/5/22 7
    8. 8. Demo Site demo.ckan.org 2014/5/22 8
    9. 9. Feature Tour (1) Publish & Find Datasets Add Dataset Basic Information 2014/5/22 9
    10. 10. Feature Tour (1) Publish & Find Datasets Add Data Under the Dataset 2014/5/22 10
    11. 11. Feature Tour (1) Publish & Find Datasets Add Metadata About the Dataset 2014/5/22 11
    12. 12. Feature Tour (1) Publish & Find Datasets Filter By Keywords 2014/5/22 12
    13. 13. Feature Tour (1) Publish & Find Datasets Filter By Geographical Features 2014/5/22 13
    14. 14. Feature Tour (2) Store & Manage Data Data Explorer: recline_preview (csv, xls) json_preview pdf_preview ckanext-spatial 2014/5/22 14
    15. 15. Feature Tour (2) Store & Manage Data Graphing data 2014/5/22 15
    16. 16. Feature Tour (3) Engage with Users & Others Share 2014/5/22 16
    17. 17. Feature Tour (3) Engage with Users & Others Organization 2014/5/22 17
    18. 18. Feature Tour (3) Engage with Users & Others Manage Users of an Organization 2014/5/22 18
    19. 19. Feature Tour (3) Engage with Users & Others Manage Role of Members Admin: edit datasets & members Editor: edit datasets Viewer: view (private) datasets Note: Public datasets are visible to everyone 2014/5/22 19
    20. 20. Feature Tour (3) Engage with Users & Others Harvest and Federation 2014/5/22 20
    21. 21. Feature Tour (3) Engage with Users & Others History 2014/5/22 21
    22. 22. Feature Tour (4) Customise & Extend ● RESTful JSON APIs – The Action API – The DataStore API – The FileStore API... ● Extensions (over 60) – ckanext-harvest – ckanext-spatial Open source is good! ● Themable ● Integrates with other CMS (ex. Drupal) 2014/5/22 22
    23. 23. CKAN and 5 ★ Open Data 2014/5/22 23
    24. 24. CKAN and 5 ★ Open Data ● ★ Make your stuff available on the Web (whatever format) under an open license 2014/5/22 24
    25. 25. CKAN and 5 ★ Open Data ● ★★ Make it available as structured data (e.g., Excel instead of image scan of a table) ★★★ Use non-proprietary formats (e.g., CSV instead of Excel) – Accept any data format – Beautiful data demonstration – Datastore : Indexing for structure data 2014/5/22 25
    26. 26. CKAN and 5 ★ Open Data ● ★★★★ Use URIs to denote things, so that people can point at your stuff – Permanent link for each dataset – Get Dataset URI through API ● ★★★★★ Link your data to other data to provide context – Linked data and RDF for metadata 2014/5/22 26
    27. 27. CKAN and 5 ★ Open Data ● RDF for metadata – DCAT and Dublin Core – curl -L -H "Accept:application/rdf+xml" http://thedatahub.org/dataset/gold-prices 2014/5/22 27
    28. 28. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 28
    29. 29. United Kingdom DATA.GOV.UK 2014/5/22 29
    30. 30. United States DATA.GOV 2014/5/22 30
    31. 31. Brazil DADOS.GOV.BR 2014/5/22 31
    32. 32. European Union PUBLICDATA.EU 2014/5/22 32
    33. 33. Geospatial Data Explorer: Lat/Long field 2014/5/22 33
    34. 34. Geospatial Data Explorer: GeoJSON 2014/5/22 34
    35. 35. Geospatial Data Explorer: WMS 2014/5/22 35
    36. 36. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 36
    37. 37. Issues ● CJK Support – CJK Search – Some broken translations – Frequent updates ● Extensions compatibilities ● Tons of tweaks needed ● Performance Issue ● Complicated Architecture 2014/5/22 37
    38. 38. System Architecture 2014/5/22 38
    39. 39. Issues (Cont'd) ● What You Should Know – Python & Pylons – ckan plugins toolkit – SQLAlchemy & SQL – HTML, JavaScript – Babel – Web Server (Nginx, uWSGI...) 2014/5/22 39
    40. 40. Resources ● Official Documents: – http://docs.ckan.org/en/latest/ ● Installation Notes (in Chinese): – https://ckan-docs-tw.readthedocs.org/ ● CKAN Development Discussions: – http://lists.okfn.org/mailman/listinfo/ckan-dev ● CKAN Taiwan Interest Group: – https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group 2014/5/22 40
    41. 41. Thanks for your attention! Any Q? Email: u10313335 AT citi.sinica.edu.tw http://about.me/sollee 2014/5/22 41
    42. 42. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 42
    43. 43. Install from Source ● Virtual environment ● Checkout the source (via GIT) – https://github.com/okfn/ckan ● Create a CKAN config file ● Setup Jetty & Solr ● Initialize Database (user, db) ● Link to who.ini ● Create a Sysadmin User ● Deployment (nginx + uWSGI) ● Install other extensions... 2014/5/22 43
    44. 44. Installation Notes ● https://ckan-docs-tw.readthedocs.org/ 2014/5/22 44
    45. 45. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 45
    46. 46. Harvesters ● ckanext-harvest – Remote harvesting extension – https://github.com/okfn/ckanext-harvest ● Source Type – CKAN (built-in) – CSW – WAF – Custom (csv/xls/website… etc) 2014/5/22 46
    47. 47. Harvested from TGOS CSW service 2014/5/22 47
    48. 48. Harvesters http://Mydomain.com/harvest 2014/5/22 48
    49. 49. Harvesters Add a new harvest source 2014/5/22 49
    50. 50. Harvesters Create a harvest job 2014/5/22 50
    51. 51. Harvesters Overview of harvested datasets 2014/5/22 51
    52. 52. Harvesters Background Process ● Manually – (pyenv) $ paster --plugin=ckanext-harvest harvester gather_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester fetch_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester run -c /etc/ckan/default/production.ini 2014/5/22 52
    53. 53. Harvesters Background Process ● Automatically – Supervisor (for gather & fetch consumer) – Cron (for run) 2014/5/22 53
    54. 54. Harvesters Custom harvester ● Implement the harvester interface to perform harvesting operations ● Three stages ● gather: get the identification ● fetch: fetch the contents ● import: create ckan package(dataset) 2014/5/22 54
    55. 55. Harvesters The harvesting interface from base import HarvesterBase class SRDAHarvester(HarvesterBase): def _set_config(self,config_str): def info(self): ... def gather_stage(self, harvest_job): def fetch_stage(self, harvest_object): def import_stage(self, harvest_object): See the extension site for details An example (SRDA): http://goo.gl/ZMnND7 2014/5/22 55
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×