CKAN 2 Introduction 
Presenter: 李承錱 Cheng-Jen Lee (Sol) 
Email: cjlee AT iis.sinica.edu.tw 
This work is licensed under a ...
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
About CKAN 
● The Comprehensive Knowledge Archive 
Network 
● A powerful data management system 
● Publishing 
● Sharing 
...
About CKAN 
83 instances 
around the world 
in May 2014 
2014/6/18 5
CKAN 2 
2014/6/18 6
Feature Tour 
2014/6/18 7
Demo Site 
demo.ckan.org 
2014/6/18 8
Feature Tour (1) 
Publish & Find Datasets 
Add Dataset Basic 
Information 
2014/6/18 9
Feature Tour (1) 
Publish & Find Datasets 
Add Data 
Under the Dataset 
2014/6/18 10
Feature Tour (1) 
Publish & Find Datasets 
Add Metadata 
About the Dataset 
2014/6/18 11
Feature Tour (1) 
Publish & Find Datasets 
Filter By 
Keywords 
2014/6/18 12
Feature Tour (1) 
Publish & Find Datasets 
Filter By 
Geographical Features 
2014/6/18 13
Feature Tour (2) 
Store & Manage Data 
Data Explorer: 
recline_preview (csv, xls) 
json_preview 
pdf_preview 
ckanext-spat...
Feature Tour (2) 
Store & Manage Data 
Graphing data 
2014/6/18 15
Feature Tour (3) 
Engage with Users & Others 
Share 
2014/6/18 16
Feature Tour (3) 
Engage with Users & Others 
Organization 
2014/6/18 17
Feature Tour (3) 
Engage with Users & Others 
Manage Users of 
an Organization 
2014/6/18 18
Feature Tour (3) 
Engage with Users & Others 
Manage Role of Members 
Admin (管理者): edit datasets & members 
Editor (編輯): e...
Feature Tour (3) 
Engage with Users & Others 
Harvest and Federation 
2014/6/18 20
Feature Tour (3) 
Engage with Users & Others 
History 
2014/6/18 21
Feature Tour (4) 
Customise & Extend 
● RESTful JSON APIs 
– The Action API 
– The DataStore API 
– The FileStore API... 
...
CKAN and 5 ★ Open Data 
2014/6/18 23
CKAN and 5 ★ Open Data 
● ★ Make your stuff available on the Web (whatever 
format) under an open license 
2014/6/18 24
CKAN and 5 ★ Open Data 
● ★★ Make it available as structured data (e.g., 
Excel instead of image scan of a table) 
★★★ Use...
CKAN and 5 ★ Open Data 
● ★★★★ Use URIs to denote things, so that people 
can point at your stuff 
– Permanent link for ea...
CKAN and 5 ★ Open Data 
● RDF for metadata 
– DCAT and Dublin Core 
– curl -L -H "Accept:application/rdf+xml" 
http://thed...
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
United Kingdom 
DATA.GOV.UK 
2014/6/18 29
United States 
DATA.GOV 
2014/6/18 30
Brazil 
DADOS.GOV.BR 
2014/6/18 31
European Union 
PUBLICDATA.EU 
2014/6/18 32
Japan 
DATA.GO.JP 
2014/6/18 33
Tainan 
DATA.TAINAN.GOV.TW (NEW!) 
2014/6/18 34
Geospatial Data Explorer: 
Lat/Long field 
2014/6/18 35
Geospatial Data Explorer: 
GeoJSON 
2014/6/18 36
Geospatial Data Explorer: 
WMS 
2014/6/18 37
Agenda 
● About CKAN 
● Feature Tour 
– Publish & Find Datasets 
– Store & Manage Data 
– Engage with Users & Others 
– Cu...
Issues 
● CJK Support 
– CJK Search 
– Some broken translations 
– File name 
● Extensions compatibilities 
● Tons of twea...
System Architecture 
2014/6/18 40
Issues (Cont'd) 
● What You Should Know 
– Python & Pylons 
– ckan plugins toolkit 
– SQLAlchemy & SQL 
– HTML, JavaScript...
Resources 
● Official Documents: 
– http://docs.ckan.org/en/latest/ 
● Installation Notes (in Chinese): 
– https://ckan-do...
Thanks for your attention! 
Any Q? 
Email: u10313335 AT citi.sinica.edu.tw 
http://about.me/sollee 
2014/6/18 43
CKAN 2: Additional 
Topics 
Presenter: 李承錱 Cheng-Jen Lee (Sol) 
Email: u10313335 AT citi.sinica.edu.tw 
This work is licen...
Agenda 
● Extended Topic 1: Installation 
● Extended Topic 2: Harvesters 
2014/6/18 45
Install from Source 
● Virtual environment 
● Checkout the source (via GIT) 
– https://github.com/okfn/ckan 
● Create a CK...
Installation Notes 
● https://ckan-docs-tw.readthedocs.org/ 
2014/6/18 47
Agenda 
● Extended Topic 1: Installation 
● Extended Topic 2: Harvesters 
2014/6/18 48
Harvesters 
● ckanext-harvest 
– Remote harvesting extension 
– https://github.com/okfn/ckanext-harvest 
● Source Type 
– ...
Harvested from TGOS 
CSW service 
2014/6/18 50
Harvesters 
http://Mydomain.com/harvest 
2014/6/18 51
Harvesters 
Add a new harvest source 
2014/6/18 52
Harvesters 
Create a harvest job 
2014/6/18 53
Harvesters 
Overview of harvested datasets 
2014/6/18 54
Harvesters 
Background Process 
● Manually 
– (pyenv) $ paster --plugin=ckanext-harvest 
harvester gather_consumer -c 
/et...
Harvesters 
Background Process 
● Automatically 
– Supervisor (for gather & fetch consumer) 
– Cron (for run) 
2014/6/18 5...
Harvesters 
Custom harvester 
● Implement the harvester interface 
to perform harvesting operations 
● Three stages 
● gat...
Harvesters 
The harvesting interface 
from base import HarvesterBase 
class SRDAHarvester(HarvesterBase): 
def _set_config...
Upcoming SlideShare
Loading in …5
×

ckan 2.0 Introduction (20140618 updated)

757 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
757
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
25
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • 地理資訊圖資雲服務平台
  • 由 Open Knowledge Foundation 支持
  • 由 Open Knowledge Foundation 支持
  • 公布網址
  • Data.gov.uk 即是 drupal+CKAN
  • Store the raw data and metadata. Visualise structured data with interactive tables, graphs and maps.
  • TGOS: 國土資訊圖資服務平台
  • ckan 2.0 Introduction (20140618 updated)

    1. 1. CKAN 2 Introduction Presenter: 李承錱 Cheng-Jen Lee (Sol) Email: cjlee AT iis.sinica.edu.tw This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Taiwan License.
    2. 2. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues 2014/6/18 2
    3. 3. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues 2014/6/18 3
    4. 4. About CKAN ● The Comprehensive Knowledge Archive Network ● A powerful data management system ● Publishing ● Sharing ● Finding ● Using Data 2014/6/18 4
    5. 5. About CKAN 83 instances around the world in May 2014 2014/6/18 5
    6. 6. CKAN 2 2014/6/18 6
    7. 7. Feature Tour 2014/6/18 7
    8. 8. Demo Site demo.ckan.org 2014/6/18 8
    9. 9. Feature Tour (1) Publish & Find Datasets Add Dataset Basic Information 2014/6/18 9
    10. 10. Feature Tour (1) Publish & Find Datasets Add Data Under the Dataset 2014/6/18 10
    11. 11. Feature Tour (1) Publish & Find Datasets Add Metadata About the Dataset 2014/6/18 11
    12. 12. Feature Tour (1) Publish & Find Datasets Filter By Keywords 2014/6/18 12
    13. 13. Feature Tour (1) Publish & Find Datasets Filter By Geographical Features 2014/6/18 13
    14. 14. Feature Tour (2) Store & Manage Data Data Explorer: recline_preview (csv, xls) json_preview pdf_preview ckanext-spatial 2014/6/18 14
    15. 15. Feature Tour (2) Store & Manage Data Graphing data 2014/6/18 15
    16. 16. Feature Tour (3) Engage with Users & Others Share 2014/6/18 16
    17. 17. Feature Tour (3) Engage with Users & Others Organization 2014/6/18 17
    18. 18. Feature Tour (3) Engage with Users & Others Manage Users of an Organization 2014/6/18 18
    19. 19. Feature Tour (3) Engage with Users & Others Manage Role of Members Admin (管理者): edit datasets & members Editor (編輯): edit datasets Viewer (成員): view (private) datasets Note: Public datasets are visible to everyone 2014/6/18 19
    20. 20. Feature Tour (3) Engage with Users & Others Harvest and Federation 2014/6/18 20
    21. 21. Feature Tour (3) Engage with Users & Others History 2014/6/18 21
    22. 22. Feature Tour (4) Customise & Extend ● RESTful JSON APIs – The Action API – The DataStore API – The FileStore API... ● Extensions (over 60) – ckanext-harvest – ckanext-spatial Open source is good! ● Themable ● Integrates with other CMS (ex. Drupal) 2014/6/18 22
    23. 23. CKAN and 5 ★ Open Data 2014/6/18 23
    24. 24. CKAN and 5 ★ Open Data ● ★ Make your stuff available on the Web (whatever format) under an open license 2014/6/18 24
    25. 25. CKAN and 5 ★ Open Data ● ★★ Make it available as structured data (e.g., Excel instead of image scan of a table) ★★★ Use non-proprietary formats (e.g., CSV instead of Excel) – Accept any data format – Beautiful data demonstration – Datastore : Indexing for structured data 2014/6/18 25
    26. 26. CKAN and 5 ★ Open Data ● ★★★★ Use URIs to denote things, so that people can point at your stuff – Permanent link for each dataset – Get Dataset URI through API ● ★★★★★ Link your data to other data to provide context – Linked data and RDF for metadata 2014/6/18 26
    27. 27. CKAN and 5 ★ Open Data ● RDF for metadata – DCAT and Dublin Core – curl -L -H "Accept:application/rdf+xml" http://thedatahub.org/dataset/gold-prices 2014/6/18 27
    28. 28. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues 2014/6/18 28
    29. 29. United Kingdom DATA.GOV.UK 2014/6/18 29
    30. 30. United States DATA.GOV 2014/6/18 30
    31. 31. Brazil DADOS.GOV.BR 2014/6/18 31
    32. 32. European Union PUBLICDATA.EU 2014/6/18 32
    33. 33. Japan DATA.GO.JP 2014/6/18 33
    34. 34. Tainan DATA.TAINAN.GOV.TW (NEW!) 2014/6/18 34
    35. 35. Geospatial Data Explorer: Lat/Long field 2014/6/18 35
    36. 36. Geospatial Data Explorer: GeoJSON 2014/6/18 36
    37. 37. Geospatial Data Explorer: WMS 2014/6/18 37
    38. 38. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues 2014/6/18 38
    39. 39. Issues ● CJK Support – CJK Search – Some broken translations – File name ● Extensions compatibilities ● Tons of tweaks needed ● Performance Issue ● Complicated Architecture 2014/6/18 39
    40. 40. System Architecture 2014/6/18 40
    41. 41. Issues (Cont'd) ● What You Should Know – Python & Pylons – ckan plugins toolkit – SQLAlchemy & SQL – HTML, JavaScript – Babel (Translation) – Web Server (UNIX, Apache, Nginx ...) 2014/6/18 41
    42. 42. Resources ● Official Documents: – http://docs.ckan.org/en/latest/ ● Installation Notes (in Chinese): – https://ckan-docs-tw.readthedocs.org/ ● CKAN Development Discussions: – http://lists.okfn.org/mailman/listinfo/ckan-dev ● CKAN Taiwan Interest Group: – https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group 2014/6/18 42
    43. 43. Thanks for your attention! Any Q? Email: u10313335 AT citi.sinica.edu.tw http://about.me/sollee 2014/6/18 43
    44. 44. CKAN 2: Additional Topics Presenter: 李承錱 Cheng-Jen Lee (Sol) Email: u10313335 AT citi.sinica.edu.tw This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Taiwan License.
    45. 45. Agenda ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/6/18 45
    46. 46. Install from Source ● Virtual environment ● Checkout the source (via GIT) – https://github.com/okfn/ckan ● Create a CKAN config file ● Setup Jetty & Solr ● Initialize Database (user, db) ● Link to who.ini ● Create a Sysadmin User ● Deployment (Apache + Nginx) ● Install other extensions... 2014/6/18 46
    47. 47. Installation Notes ● https://ckan-docs-tw.readthedocs.org/ 2014/6/18 47
    48. 48. Agenda ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/6/18 48
    49. 49. Harvesters ● ckanext-harvest – Remote harvesting extension – https://github.com/okfn/ckanext-harvest ● Source Type – CKAN (built-in) – CSW – WAF – Custom (csv/xls/website… etc) 2014/6/18 49
    50. 50. Harvested from TGOS CSW service 2014/6/18 50
    51. 51. Harvesters http://Mydomain.com/harvest 2014/6/18 51
    52. 52. Harvesters Add a new harvest source 2014/6/18 52
    53. 53. Harvesters Create a harvest job 2014/6/18 53
    54. 54. Harvesters Overview of harvested datasets 2014/6/18 54
    55. 55. Harvesters Background Process ● Manually – (pyenv) $ paster --plugin=ckanext-harvest harvester gather_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester fetch_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester run -c /etc/ckan/default/production.ini 2014/6/18 55
    56. 56. Harvesters Background Process ● Automatically – Supervisor (for gather & fetch consumer) – Cron (for run) 2014/6/18 56
    57. 57. Harvesters Custom harvester ● Implement the harvester interface to perform harvesting operations ● Three stages ● gather: get the identification ● fetch: fetch the contents ● import: create ckan package (dataset) 2014/6/18 57
    58. 58. Harvesters The harvesting interface from base import HarvesterBase class SRDAHarvester(HarvesterBase): def _set_config(self,config_str): def info(self): ... def gather_stage(self, harvest_job): def fetch_stage(self, harvest_object): def import_stage(self, harvest_object): See the extension site for details An example (SRDA): http://goo.gl/ZMnND7 2014/6/18 58

    ×