Ckan tutorial odw2013 131109

  • 533 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
533
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
17
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • 由 Open Knowledge Foundation 支持
  • 由 Open Knowledge Foundation 支持
  • 公布網址
  • Data.gov.uk 即是 drupal+CKAN
  • Store the raw data and metadata. Visualise structured data with interactive tables, graphs and maps.
  • TGOS: 國土資訊圖資服務平台

Transcript

  • 1. CKAN Tutorial @ odw2013 Presenter: 李承錱 Cheng-Jen Lee (Sol) Email: cjlee AT iis.sinica.edu.tw This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Taiwan License.
  • 2. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  • 3. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  • 4. 2013/11/9 About CKAN ● The Comprehensive Knowledge Archive Network ● A powerful data management system ● Publishing ● Sharing ● Finding ● Using Data
  • 5. 2013/11/9 About CKAN 61 instances
  • 6. 2013/11/9 CKAN 2
  • 7. 2013/11/9 Feature Tour
  • 8. 2013/11/9 Demo Site demo.ckan.org
  • 9. 2013/11/9 Feature Tour (1) Publish & Find Datasets Add Dataset Basic Information
  • 10. 2013/11/9 Feature Tour (1) Publish & Find Datasets Add Data Under the Dataset
  • 11. 2013/11/9 Feature Tour (1) Publish & Find Datasets Add Metadata About the Dataset
  • 12. 2013/11/9 Feature Tour (1) Publish & Find Datasets Filter By Keywords
  • 13. 2013/11/9 Feature Tour (1) Publish & Find Datasets Filter By Geographical Features
  • 14. 2013/11/9 Feature Tour (2) Store & Manage Data Data Explorer: recline_preview (csv, xls) json_preview pdf_preview ckanext-spatial
  • 15. 2013/11/9 Feature Tour (2) Store & Manage Data Graphing data
  • 16. 2013/11/9 Feature Tour (3) Engage with Users & Others Share
  • 17. 2013/11/9 Feature Tour (3) Engage with Users & Others Organization
  • 18. 2013/11/9 Feature Tour (3) Engage with Users & Others Manage Users of an Organization
  • 19. 2013/11/9 Feature Tour (3) Engage with Users & Others Manage Role of Members Admin: edit datasets & members Editor: edit datasets Viewer: view (private) datasets Note: Public datasets are visible to everyone
  • 20. 2013/11/9 Feature Tour (3) Engage with Users & Others Harvest and Federation
  • 21. 2013/11/9 Feature Tour (3) Engage with Users & Others History
  • 22. 2013/11/9 Feature Tour (4) Customise & Extend ● RESTful JSON APIs – The Action API – The DataStore API – The FileStore API... ● Extensions (over 60) – ckanext-harvest – ckanext-spatial Open source is good! ● Themable ● Integrates with other CMS (ex. Drupal)
  • 23. CKAN and 5 ★ Open Data 2013/11/9
  • 24. CKAN and 5 ★ Open Data ● ★ Make your stuff available on the Web (whatever format) under an open license 2013/11/9
  • 25. CKAN and 5 ★ Open Data ● ★★ Make it available as structured data (e.g., Excel instead of image scan of a table) ★★★ Use non-proprietary formats (e.g., CSV instead of Excel) – Accept any data format – Beautiful data demonstration 2013/11/9
  • 26. CKAN and 5 ★ Open Data ● ★★★★ Use URIs to denote things, so that people can point at your stuff – Permanent link for each dataset – Get Dataset URI through API ● ★★★★★ Link your data to other data to provide context – Linked data and RDF for metadata 2013/11/9
  • 27. CKAN and 5 ★ Open Data ● RDF for metadata – DCAT and Dublin Core – curl -L -H "Accept:application/rdf+xml" http://thedatahub.org/dataset/gold-prices 2013/11/9
  • 28. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  • 29. 2013/11/9 United Kingdom DATA.GOV.UK
  • 30. 2013/11/9 United States DATA.GOV
  • 31. 2013/11/9 Brazil DADOS.GOV.BR
  • 32. 2013/11/9 European Union PUBLICDATA.EU
  • 33. 2013/11/9 Geospatial Data Explorer: Lat/Long field
  • 34. 2013/11/9 Geospatial Data Explorer: GeoJSON
  • 35. 2013/11/9 Geospatial Data Explorer: WMS
  • 36. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  • 37. 2013/11/9 System Architecture
  • 38. What You Should Know ● Python, Pylons ● ckan plugins toolkit ● SQLAlchemy, SQL ● HTML, JavaScript ● Babel ● Web Server (Nginx, uWSGI...) 2013/11/9
  • 39. 2013/11/9 Install from Source ● Virtual environment ● Checkout the source (via GIT) – https://github.com/okfn/ckan ● Create a CKAN config file ● Setup Jetty & Solr ● Initialize Database (user, db) ● Link to who.ini ● Create a Sysadmin User ● Deployment (nginx + uWSGI) ● Install other extensions...
  • 40. 2013/11/9 Installation Notes ● https://ckan-docs-tw.readthedocs.org/
  • 41. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  • 42. 2013/11/9 Harvesters ● ckanext-harvest – Remote harvesting extension – https://github.com/okfn/ckanext-harvest ● Source Type – CKAN (built-in) – CSW – WAF – Custom (csv/xls/website… etc)
  • 43. 2013/11/9 Harvested from TGOS CSW service
  • 44. 2013/11/9 Harvesters http://Mydomain.com/harvest
  • 45. 2013/11/9 Harvesters Add a new harvest source
  • 46. 2013/11/9 Harvesters Create a harvest job
  • 47. 2013/11/9 Harvesters Overview of harvested datasets
  • 48. 2013/11/9 Harvesters Background Process ● Manually – (pyenv) $ paster --plugin=ckanext-harvest harvester gather_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester fetch_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester run -c /etc/ckan/default/production.ini
  • 49. 2013/11/9 Harvesters Background Process ● Automatically – Supervisor (for gather & fetch consumer) – Cron (for run)
  • 50. 2013/11/9 Harvesters Custom harvester ● Implement the harvester interface to perform harvesting operations ● Three stages ● gather: get the identification ● fetch: fetch the contents ● import: create ckan package(dataset)
  • 51. 2013/11/9 Harvesters The harvesting interface from base import HarvesterBase class SRDAHarvester(HarvesterBase): def _set_config(self,config_str): def info(self): ... def gather_stage(self, harvest_job): def fetch_stage(self, harvest_object): def import_stage(self, harvest_object): See the extension site for details An example (SRDA): http://goo.gl/ZMnND7
  • 52. 2013/11/9 Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Installation ● Harvesters ● Issues
  • 53. 2013/11/9 Issues ● CJK Support – CJK Search – Some broken translations – Frequent updates ● Extensions compatibilities ● Tons of tweaks needed ● Performance Issue ● Complicated Architecture
  • 54. ● Official Documents: – http://docs.ckan.org/en/latest/ ● Installation Notes (in Chinese): – https://ckan-docs-tw.readthedocs.org/ ● CKAN Development Discussions: – http://lists.okfn.org/mailman/listinfo/ckan-dev ● CKAN Taiwan Interest Group: – https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group 2013/11/9 Resources
  • 55. Thanks for your attention! 2013/11/9 Any Q? Email: u10313335 AT citi.sinica.edu.tw http://about.me/sollee