Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Getting to Know CKAN, 24 June 2015, Singapore


Published on

Presented in Singapore on 24 June 2015 as part of the Infocomm Development Authority Data 101 series.

Provides an overview on what CKAN is and what organisations are using it for. The session also covered a number of topics related to the organisation of published data.

Published in: Data & Analytics
  • Controversial method reveals inner psychology of techniques you can use to get your Ex back! See it now! ➤➤
    Are you sure you want to  Yes  No
    Your message goes here
  • good. thanks
    Are you sure you want to  Yes  No
    Your message goes here

Getting to Know CKAN, 24 June 2015, Singapore

  2. 2. Introduction 3 Link Digital Executive Director CKAN Association Steering Group Member Open Knowledge Australia Co-Secretary & Treasurer Data Shades Founder GovHack National Organiser AWS User Group CBR Organiser
  3. 3. STEVEN DE COSTA Steering Group Member @STARL3N
  4. 4. STEVEN DE COSTA Steering Group Member @STARL3N
  5. 5. Data portal software: 1. Open Source 2. Large and expanding installation base within Government worldwide 3. Expanding use cases in the wider data ecosystem 4. Python web app, PostgreSQL DB 5. Built for machines, custodians and end users WHAT IS CKAN?
  8. 8. 1. >> Organisations (optionally with sub-organisations) 2. >> >> Datasets 3. >> >> >> Resources 4. >> Platform Custodian 5. >> >> Organisation Custodian, Editor or Member 6. >> Published or Private datasets CKAN STRUCTURE
  9. 9. 1. >> Constitution 2. >> >> Parliamentary Legislation and Acts (Jurisdiction = Platform) 3. >> >> >> Ministries (Organisation) 4. >> >> >> >> Programs (Sub-Organisations) 5. >> >> >> >> >> Projects (Datasets) 6. >> >> >> >> >> >> Outcomes (Resources) CKAN USE CASE PARADIGM
  10. 10. 1. User registration 2. User management 3. Custodian workflows (manage datasets and data resources) 4. Directory Browse by organisation or group 5. Faceted search for multiple fields (supporting end user discovery) 6. Resource views to preview data (a recently improved feature) 7. Metadata view CKAN UI
  11. 11. 1. Create an organistation (usually done by platform owner) 2. Login as member of organisation 3. Click ‘add dataset’ 4. Step 1: Add a title, description and other metadata 5. Step 2: Add resources (links to data or upload data files for hosting) 6. Step 3: Add any additional info CKAN CUSTODIAN WORKFLOW
  12. 12. 1. Title 2. Description: Using markdown as required 3. Keywords: some keywords (or tags) that describe your data. 4. License (required): a dropdown of available licenses for (the default is Creative Commons Attribution 3.0 Australia) 5. Organisation: a dropdown of organisations you can publish to. Most users can only publish to a single organisation. This will be automatically filled in. 6. Visibility: whether the dataset will be viewable to all users once complete. The default is private. 7. Geospatial Coverage (required): inherited from organisation metadata this is the area which the data covers. It can be; a point/polygon (Well-known text); an administrative boundary API; or, a reference URL (website address) from the National Gazatteer. Gazetteer reference URLs can be found by searching for a place at then clicking through to the most appropriate location "Reference ID", and then copying and pasting the URL from the page into the Geospatial field in CKAN METADATA FOR DATA.GOV.AU
  13. 13. 8. Temporal Coverage From / To (required): the span of time from/to which the data is applicable. If the data applies only to a single point in time you should only fill in the Temporal Coverage From field. 9. Language: the language in which the dataset is published. The default is English. 10. Data Status (required): the status of the data with regard to whether it is kept updated (active, yes) or historic (inactive, no). 11. Update Frequency (required): how often the dataset is updated. Eg: Daily, Weekly, Never. (for remote machine readable files this field will be used to fetch new versions of this data) 12. Expose User Contact Information: display additional contact information for the dataset. 13. AGIFT Function/Theme: the AGIFT top level government function to which the dataset relates. 14. Publisher: name of Agency/publishing organisation. The default is set to the organisation’s name. 15. Jurisdiction: name of the jurisdiction in which the dataset belongs. The default is set to the organisation’s jurisdiction. CKAN METADATA FOR DATA.GOV.AU
  14. 14. 1. Join the dev mailing list (monitored by tech team): 2. Search Stack overflow under CKAN: 3. Check the roadmap on 4. Join a tech team meeting: WHERE TO GET HELP
  16. 16. 1. Get JSON-formatted lists of a site’s datasets, groups or other CKAN objects 2. Get a full JSON representation of a dataset, resource or other object 3. Search for packages or resources matching a query 4. Create, update and delete datasets, resources and other objects 5. Get an activity stream of recently changed datasets on a site CKAN API
  17. 17. 1. CKAN as an Information Asset Register 2. FileStore – For hosting of data and resources 3. DataStore - provides a database for structured storage of data together with a powerful Web- accessible Data API 4. License Selection (machine ready?) 5. Harvesting A FEW MORE POINTS
  18. 18. Delivering the world’s best open data management system The purpose of the CKAN Association is to support sustainable growth and development of CKAN while also protecting the interests of the CKAN community. The Association values a healthy and thriving community which continues to deliver the best open data management system in the world. CKAN ASSOCIATION: STATEMENT OF PURPOSE
  19. 19. Community interests, or needs, can be generalised as: 1. Users need an enterprise level open data management system tailored to meet their needs now and into the future. 2. Individual contributors need a project that is rewarding to work for, inclusive and active. COMMUNITY INTERESTS
  20. 20. Who owns or directly manages the CKAN project? The project, its releases and future direction are cooperatively managed by its community of users and contributors. Association delegates, staff and office holders may be active within the project but will exercise no more or less influence than any other contributor or user. More information about CKAN and how to contribute can be found at DELEGATION TO SERVE AND PROTECT
  21. 21. What is the current structure of the CKAN Association? See The steering group carries on a number of business activities. This includes raising revenue, managing resources and directing projects or programs of activity relevant to the CKAN Association’s statement of purpose. COMPLETENESS OF PURPOSE
  23. 23. It's easy to find people willing to fund or work on the flashier parts of CKAN, and there is some funding for core work such as meetings and preparing releases. But there are a few important things that aren't getting addressed. Fix Security Issues There are a small number of known issues that high profile sites work around by implementing workflows and tight controls over users and data published. Everyone else would benefit from a more-secure-by-default CKAN. Clear Technical Debt CKAN, has some parts that should be replaced or removed. This work doesn't add any new features and worse, threatens to break upgrades for existing sites and extensions. It is necessary work though. Unless performance and technical limitations are addressed they will grow and start turning users away from CKAN. Ref: - Raised by Ian Ward, Tech Team Lead HOW MEMBERSHIPS BENEFIT CKAN
  24. 24. 1. Large installation base of significant platforms 2. Open Source projects favoured by technical influencers, driving adoption 3. Historical association with Open Knowledge adds significant ‘open’ credibility 4. Support via Open Knowledge opens direct and indirect funding support 5. Global awareness of CKAN due to many years in the market 6. CKAN Association is established with strong representatives from three continents 7. Established, experienced and professional technical team 8. Extension model allows for customisations 9. CKAN can be integrated with WordPress or Drupal 10. Free to use 11. A number of private companies now specialise in CKAN services 12. Public roadmap for future features OUR STRENGTHS
  25. 25. 1. Current reliance on Open Knowledge funded contributors 2. Technical debt (python framework needs replacement) 3. Relatively* low activity on roadmap feature development 4. Community and Communications Team lacks resources 5. Technical team is relatively* small 6. Relatively* small number of private companies specialise in CKAN services * Compare with projects with such significant adoption in enterprises OUR WEAKNESSES
  26. 26. 1. Partnerships within data and civic app sector 2. Many markets where CKAN is being considered 3. Markets are growing 4. CKAN is a technically interesting project for potential contributors 5. Attractive to internal technical teams within Government 6. Use case ‘distributions’ to meet new verticals 7. Use case extensions can increase the value of current installations 8. Regular update releases supporting a vendor community 9. Vendor community can grow awareness, demand and installations of CKAN OUR OPPORTUNITIES
  27. 27. 10. An effective communications effort can grow awareness of the project 11. A modified license would allow CKAN to be integrated with commercial applications 12. Revenue from formal membership tiers can fund business activities 13. In-kind contributions from formal membership can advance business activities 14. Domain specific working groups can align the project with various global activities and market segments 15. Ensuring broad interoperability to widen use cases and adoption within various solution 16. Direct alignment with OK initiatives, such as OpenSpending, for increasing the value of CKAN OUR OPPORTUNITIES
  28. 28. 1. Alternatives to CKAN could gain in popularity, degrading contributor interest 2. Project is overly reliant on support from Open Knowledge 3. Possibility of user churn out, degrading the user base 4. Supply of enterprise services fails to meet demand for services, killing adoption 5. Project could fork in too many directions if doesn’t evolve to meet user needs over the long term. 6. Contributors churn out, leaving the community without new contributors entering OUR THREATS
  30. 30. The DataCo Vision Today 31 DataCo brings together the supply and demand sides of the market place, which stimulates the private sector and ensures access by the public to geographical information. Part of our role is to help the contributor’s disparate data resources mesh, cooperate and function uniformly to create high-quality products.
  31. 31. Challenges to The DataCo Vision 32 • Public conversation on data has moved beyond enterprise ICT and large monopolistic supply chains. • The convergence between public cloud, IoT and analytics is raising the public awareness of Big Data’s value to the economy as a platform for innovation. • The convergence between open source and open government has raised awareness surrounding open data as a driver for economic redistributions of value from the few to the many. • Data is a special kind of economic good which people are learning to take for granted as open by default.
  32. 32. The DataCo Vision Tomorrow 33 DataCo’s role is to help disparate data resources mesh, cooperate and function uniformly to create high-quality products. We supply this data under a variety of commercial arrangements which enable our customers to develop products which they can rely on for high quality geographical information.
  33. 33. Foundations of Open Government 34 My Administration is committed to creating an unprecedented level of openness in Government. We will work together to ensure the public trust and establish a system of transparency, public participation, and collaboration. Openness will strengthen our democracy and promote efficiency and effectiveness in Government. Sources: Hope Poster by Shepard Fairey:
  34. 34. The role of Government in Australia 35 It should support civil society and its multiplicity of voices and activities. It should provide the economic framework and the essential infrastructure for public and private enterprise. Source:
  35. 35. Governance and Provenance in Singapore 36 Constitution: established 1965 Parliament: Policy, Legislation and Administrative Acts Ministries: Prime Minister’s Office Entities and Programs: Smart Nation Projects: Data 101 Outcomes: Attendees
  36. 36. Collaboration is essential 37 Why is collaboration essential for: innovation? < multidisciplinary skills and experience digital transactions? < stakeholder and user participation data classification? < early identification and ongoing governance security? < peer review and bright lights
  37. 37. Business Cases: Theory of the firm & Transaction costs 38
  38. 38. Where is there work to do with Open Data? 39
  39. 39. 40Link Digital’s Network Map
  40. 40. Published Data Handbook 41
  41. 41. Published Data Handbook 42 Recipe Name What you’ll need Ingredients Method Perfect for…
  42. 42. 43Establishing Data Publishing Policy
  43. 43. 44Establishing Data Management Systems
  44. 44. 45Educating Data Publishers
  45. 45. 46Facilitation within tech communities
  46. 46. Establishing enterprise solution architecture 47
  47. 47. 48Extract, Transform and Load Connectors
  48. 48. 49Cleaning of Messy Data
  49. 49. 50Geospatial Integration
  50. 50. 51Resource View Integration
  51. 51. 52Enriched Data and Analytics
  52. 52. Types of business models 53 Specialist services Managed Services Product Integration with monetisation options established with the platform owner
  53. 53. 54The perfect storm
  54. 54. 55Drupal interface
  55. 55. 56CKAN interface
  56. 56. 57What the DFMP does
  57. 57. 58Launched Globally as CKAN Galleries
  58. 58. 59Launched Globally as CKAN Galleries
  59. 59. 60Government as an API - Digital Transformation Office
  60. 60. on CKAN, Drupal and AWS
  61. 61. Classifying data 62 The three tiers are: internal whole of Government open
  62. 62. data classification
  63. 63. 64The ‘go. no go’ gates for going open
  64. 64. Take Note: What is NOT good 65
  65. 65. Take Note: What is best… 66