Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ckan tutorial odw2013 131109

2,153 views

Published on

Published in: Technology
  • Be the first to comment

Ckan tutorial odw2013 131109

  1. 1. CKAN Tutorial @ odw2013 Presenter: 李承錱 Cheng-Jen Lee (Sol) Email: cjlee AT iis.sinica.edu.tw This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Taiwan License.
  2. 2. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  3. 3. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  4. 4. 2013/11/9 About CKAN ● The Comprehensive Knowledge Archive Network ● A powerful data management system ● Publishing ● Sharing ● Finding ● Using Data
  5. 5. 2013/11/9 About CKAN 61 instances
  6. 6. 2013/11/9 CKAN 2
  7. 7. 2013/11/9 Feature Tour
  8. 8. 2013/11/9 Demo Site demo.ckan.org
  9. 9. 2013/11/9 Feature Tour (1) Publish & Find Datasets Add Dataset Basic Information
  10. 10. 2013/11/9 Feature Tour (1) Publish & Find Datasets Add Data Under the Dataset
  11. 11. 2013/11/9 Feature Tour (1) Publish & Find Datasets Add Metadata About the Dataset
  12. 12. 2013/11/9 Feature Tour (1) Publish & Find Datasets Filter By Keywords
  13. 13. 2013/11/9 Feature Tour (1) Publish & Find Datasets Filter By Geographical Features
  14. 14. 2013/11/9 Feature Tour (2) Store & Manage Data Data Explorer: recline_preview (csv, xls) json_preview pdf_preview ckanext-spatial
  15. 15. 2013/11/9 Feature Tour (2) Store & Manage Data Graphing data
  16. 16. 2013/11/9 Feature Tour (3) Engage with Users & Others Share
  17. 17. 2013/11/9 Feature Tour (3) Engage with Users & Others Organization
  18. 18. 2013/11/9 Feature Tour (3) Engage with Users & Others Manage Users of an Organization
  19. 19. 2013/11/9 Feature Tour (3) Engage with Users & Others Manage Role of Members Admin: edit datasets & members Editor: edit datasets Viewer: view (private) datasets Note: Public datasets are visible to everyone
  20. 20. 2013/11/9 Feature Tour (3) Engage with Users & Others Harvest and Federation
  21. 21. 2013/11/9 Feature Tour (3) Engage with Users & Others History
  22. 22. 2013/11/9 Feature Tour (4) Customise & Extend ● RESTful JSON APIs – The Action API – The DataStore API – The FileStore API... ● Extensions (over 60) – ckanext-harvest – ckanext-spatial Open source is good! ● Themable ● Integrates with other CMS (ex. Drupal)
  23. 23. CKAN and 5 ★ Open Data 2013/11/9
  24. 24. CKAN and 5 ★ Open Data ● ★ Make your stuff available on the Web (whatever format) under an open license 2013/11/9
  25. 25. CKAN and 5 ★ Open Data ● ★★ Make it available as structured data (e.g., Excel instead of image scan of a table) ★★★ Use non-proprietary formats (e.g., CSV instead of Excel) – Accept any data format – Beautiful data demonstration 2013/11/9
  26. 26. CKAN and 5 ★ Open Data ● ★★★★ Use URIs to denote things, so that people can point at your stuff – Permanent link for each dataset – Get Dataset URI through API ● ★★★★★ Link your data to other data to provide context – Linked data and RDF for metadata 2013/11/9
  27. 27. CKAN and 5 ★ Open Data ● RDF for metadata – DCAT and Dublin Core – curl -L -H "Accept:application/rdf+xml" http://thedatahub.org/dataset/gold-prices 2013/11/9
  28. 28. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  29. 29. 2013/11/9 United Kingdom DATA.GOV.UK
  30. 30. 2013/11/9 United States DATA.GOV
  31. 31. 2013/11/9 Brazil DADOS.GOV.BR
  32. 32. 2013/11/9 European Union PUBLICDATA.EU
  33. 33. 2013/11/9 Geospatial Data Explorer: Lat/Long field
  34. 34. 2013/11/9 Geospatial Data Explorer: GeoJSON
  35. 35. 2013/11/9 Geospatial Data Explorer: WMS
  36. 36. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  37. 37. 2013/11/9 System Architecture
  38. 38. What You Should Know ● Python, Pylons ● ckan plugins toolkit ● SQLAlchemy, SQL ● HTML, JavaScript ● Babel ● Web Server (Nginx, uWSGI...) 2013/11/9
  39. 39. 2013/11/9 Install from Source ● Virtual environment ● Checkout the source (via GIT) – https://github.com/okfn/ckan ● Create a CKAN config file ● Setup Jetty & Solr ● Initialize Database (user, db) ● Link to who.ini ● Create a Sysadmin User ● Deployment (nginx + uWSGI) ● Install other extensions...
  40. 40. 2013/11/9 Installation Notes ● https://ckan-docs-tw.readthedocs.org/
  41. 41. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  42. 42. 2013/11/9 Harvesters ● ckanext-harvest – Remote harvesting extension – https://github.com/okfn/ckanext-harvest ● Source Type – CKAN (built-in) – CSW – WAF – Custom (csv/xls/website… etc)
  43. 43. 2013/11/9 Harvested from TGOS CSW service
  44. 44. 2013/11/9 Harvesters http://Mydomain.com/harvest
  45. 45. 2013/11/9 Harvesters Add a new harvest source
  46. 46. 2013/11/9 Harvesters Create a harvest job
  47. 47. 2013/11/9 Harvesters Overview of harvested datasets
  48. 48. 2013/11/9 Harvesters Background Process ● Manually – (pyenv) $ paster --plugin=ckanext-harvest harvester gather_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester fetch_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester run -c /etc/ckan/default/production.ini
  49. 49. 2013/11/9 Harvesters Background Process ● Automatically – Supervisor (for gather & fetch consumer) – Cron (for run)
  50. 50. 2013/11/9 Harvesters Custom harvester ● Implement the harvester interface to perform harvesting operations ● Three stages ● gather: get the identification ● fetch: fetch the contents ● import: create ckan package(dataset)
  51. 51. 2013/11/9 Harvesters The harvesting interface from base import HarvesterBase class SRDAHarvester(HarvesterBase): def _set_config(self,config_str): def info(self): ... def gather_stage(self, harvest_job): def fetch_stage(self, harvest_object): def import_stage(self, harvest_object): See the extension site for details An example (SRDA): http://goo.gl/ZMnND7
  52. 52. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  53. 53. 2013/11/9 Issues ● CJK Support – CJK Search – Some broken translations – Frequent updates ● Extensions compatibilities ● Tons of tweaks needed ● Performance Issue ● Complicated Architecture
  54. 54. ● Official Documents: – http://docs.ckan.org/en/latest/ ● Installation Notes (in Chinese): – https://ckan-docs-tw.readthedocs.org/ ● CKAN Development Discussions: – http://lists.okfn.org/mailman/listinfo/ckan-dev ● CKAN Taiwan Interest Group: – https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group 2013/11/9 Resources
  55. 55. Thanks for your attention! 2013/11/9 Any Q? Email: u10313335 AT citi.sinica.edu.tw http://about.me/sollee

×