ckan 2.0 Introduction (20140522 updated)

482 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
482
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 由 Open Knowledge Foundation 支持
  • 由 Open Knowledge Foundation 支持
  • 公布網址
  • Data.gov.uk 即是 drupal+CKAN
  • Store the raw data and metadata. Visualise structured data with interactive tables, graphs and maps.
  • TGOS: 國土資訊圖資服務平台
  • ckan 2.0 Introduction (20140522 updated)

    1. 1. CKAN 2 Introduction Presenter: 李承錱 Cheng-Jen Lee (Sol) Email: cjlee AT iis.sinica.edu.tw This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Taiwan License.
    2. 2. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 2
    3. 3. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 3
    4. 4. About CKAN ● The Comprehensive Knowledge Archive Network ● A powerful data management system ● Publishing ● Sharing ● Finding ● Using Data 2014/5/22 4
    5. 5. About CKAN 83 instances around the world in May 2014 2014/5/22 5
    6. 6. CKAN 2 2014/5/22 6
    7. 7. Feature Tour 2014/5/22 7
    8. 8. Demo Site demo.ckan.org 2014/5/22 8
    9. 9. Feature Tour (1) Publish & Find Datasets Add Dataset Basic Information 2014/5/22 9
    10. 10. Feature Tour (1) Publish & Find Datasets Add Data Under the Dataset 2014/5/22 10
    11. 11. Feature Tour (1) Publish & Find Datasets Add Metadata About the Dataset 2014/5/22 11
    12. 12. Feature Tour (1) Publish & Find Datasets Filter By Keywords 2014/5/22 12
    13. 13. Feature Tour (1) Publish & Find Datasets Filter By Geographical Features 2014/5/22 13
    14. 14. Feature Tour (2) Store & Manage Data Data Explorer: recline_preview (csv, xls) json_preview pdf_preview ckanext-spatial 2014/5/22 14
    15. 15. Feature Tour (2) Store & Manage Data Graphing data 2014/5/22 15
    16. 16. Feature Tour (3) Engage with Users & Others Share 2014/5/22 16
    17. 17. Feature Tour (3) Engage with Users & Others Organization 2014/5/22 17
    18. 18. Feature Tour (3) Engage with Users & Others Manage Users of an Organization 2014/5/22 18
    19. 19. Feature Tour (3) Engage with Users & Others Manage Role of Members Admin: edit datasets & members Editor: edit datasets Viewer: view (private) datasets Note: Public datasets are visible to everyone 2014/5/22 19
    20. 20. Feature Tour (3) Engage with Users & Others Harvest and Federation 2014/5/22 20
    21. 21. Feature Tour (3) Engage with Users & Others History 2014/5/22 21
    22. 22. Feature Tour (4) Customise & Extend ● RESTful JSON APIs – The Action API – The DataStore API – The FileStore API... ● Extensions (over 60) – ckanext-harvest – ckanext-spatial Open source is good! ● Themable ● Integrates with other CMS (ex. Drupal) 2014/5/22 22
    23. 23. CKAN and 5 ★ Open Data 2014/5/22 23
    24. 24. CKAN and 5 ★ Open Data ● ★ Make your stuff available on the Web (whatever format) under an open license 2014/5/22 24
    25. 25. CKAN and 5 ★ Open Data ● ★★ Make it available as structured data (e.g., Excel instead of image scan of a table) ★★★ Use non-proprietary formats (e.g., CSV instead of Excel) – Accept any data format – Beautiful data demonstration – Datastore : Indexing for structure data 2014/5/22 25
    26. 26. CKAN and 5 ★ Open Data ● ★★★★ Use URIs to denote things, so that people can point at your stuff – Permanent link for each dataset – Get Dataset URI through API ● ★★★★★ Link your data to other data to provide context – Linked data and RDF for metadata 2014/5/22 26
    27. 27. CKAN and 5 ★ Open Data ● RDF for metadata – DCAT and Dublin Core – curl -L -H "Accept:application/rdf+xml" http://thedatahub.org/dataset/gold-prices 2014/5/22 27
    28. 28. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 28
    29. 29. United Kingdom DATA.GOV.UK 2014/5/22 29
    30. 30. United States DATA.GOV 2014/5/22 30
    31. 31. Brazil DADOS.GOV.BR 2014/5/22 31
    32. 32. European Union PUBLICDATA.EU 2014/5/22 32
    33. 33. Geospatial Data Explorer: Lat/Long field 2014/5/22 33
    34. 34. Geospatial Data Explorer: GeoJSON 2014/5/22 34
    35. 35. Geospatial Data Explorer: WMS 2014/5/22 35
    36. 36. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 36
    37. 37. Issues ● CJK Support – CJK Search – Some broken translations – Frequent updates ● Extensions compatibilities ● Tons of tweaks needed ● Performance Issue ● Complicated Architecture 2014/5/22 37
    38. 38. System Architecture 2014/5/22 38
    39. 39. Issues (Cont'd) ● What You Should Know – Python & Pylons – ckan plugins toolkit – SQLAlchemy & SQL – HTML, JavaScript – Babel – Web Server (Nginx, uWSGI...) 2014/5/22 39
    40. 40. Resources ● Official Documents: – http://docs.ckan.org/en/latest/ ● Installation Notes (in Chinese): – https://ckan-docs-tw.readthedocs.org/ ● CKAN Development Discussions: – http://lists.okfn.org/mailman/listinfo/ckan-dev ● CKAN Taiwan Interest Group: – https://groups.google.com/forum/#!forum/ckan-taiwan-interest-group 2014/5/22 40
    41. 41. Thanks for your attention! Any Q? Email: u10313335 AT citi.sinica.edu.tw http://about.me/sollee 2014/5/22 41
    42. 42. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 42
    43. 43. Install from Source ● Virtual environment ● Checkout the source (via GIT) – https://github.com/okfn/ckan ● Create a CKAN config file ● Setup Jetty & Solr ● Initialize Database (user, db) ● Link to who.ini ● Create a Sysadmin User ● Deployment (nginx + uWSGI) ● Install other extensions... 2014/5/22 43
    44. 44. Installation Notes ● https://ckan-docs-tw.readthedocs.org/ 2014/5/22 44
    45. 45. Agenda ● About CKAN ● Feature Tour – Publish & Find Datasets – Store & Manage Data – Engage with Users & Others – Customise & Extend ● CKAN and 5 ★ Open Data ● Showcase ● Issues ● Extended Topic 1: Installation ● Extended Topic 2: Harvesters 2014/5/22 45
    46. 46. Harvesters ● ckanext-harvest – Remote harvesting extension – https://github.com/okfn/ckanext-harvest ● Source Type – CKAN (built-in) – CSW – WAF – Custom (csv/xls/website… etc) 2014/5/22 46
    47. 47. Harvested from TGOS CSW service 2014/5/22 47
    48. 48. Harvesters http://Mydomain.com/harvest 2014/5/22 48
    49. 49. Harvesters Add a new harvest source 2014/5/22 49
    50. 50. Harvesters Create a harvest job 2014/5/22 50
    51. 51. Harvesters Overview of harvested datasets 2014/5/22 51
    52. 52. Harvesters Background Process ● Manually – (pyenv) $ paster --plugin=ckanext-harvest harvester gather_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester fetch_consumer -c /etc/ckan/default/production.ini – (pyenv) $ paster --plugin=ckanext-harvest harvester run -c /etc/ckan/default/production.ini 2014/5/22 52
    53. 53. Harvesters Background Process ● Automatically – Supervisor (for gather & fetch consumer) – Cron (for run) 2014/5/22 53
    54. 54. Harvesters Custom harvester ● Implement the harvester interface to perform harvesting operations ● Three stages ● gather: get the identification ● fetch: fetch the contents ● import: create ckan package(dataset) 2014/5/22 54
    55. 55. Harvesters The harvesting interface from base import HarvesterBase class SRDAHarvester(HarvesterBase): def _set_config(self,config_str): def info(self): ... def gather_stage(self, harvest_job): def fetch_stage(self, harvest_object): def import_stage(self, harvest_object): See the extension site for details An example (SRDA): http://goo.gl/ZMnND7 2014/5/22 55

    ×