1
The Shift to {Open|Big|Linked} Data
Or, how I learned to stop worrying and love the data
Pia Waugh
Director of Coordination and Gov 2.0
Technology and Procurement Division
Department of Finance
22
Great expectations
http://www.flickr.com/photos/zebble/8212264/
33
[Open|Big|Government] Data
Credit: Joel Gurin (with permission)
44
http://xkcd.com/898/
The emergence of the technocracy
55
Key Benefits to Government in Opening Data
• Improved services delivery
• Efficiency gains
• Better policy outcomes
• More consistency across government(s)
• Improved opportunities to leverage innovation and
collaboration
• Opportunities to improve data quality through
verifiable public contributions
66
Key Benefits to Community in Opening Data
Economic
• Creates opportunities for industry to value-add to government data
• New services, systems and industries
• New opportunities and innovation in industry, research, civil society
Accountability
• Visibility to government spending, projects, effectiveness, etc
• Increases incentive to follow evidence based approach
Better policy and programs
• Enables greater participation in policy planning and implementation
• More informed public → better decision making
• Improvements to data → better policy and decisions
77
The APS eGov and Open Data Policy Landscape
Others:
• Publishing Public Sector Information & National Standards Framework
• Declaration of Open Government
• Gov 2.0 Taskforce Report
• Statement of IP Principles for Government (CC-BY)
• Ahead of the Game
• Digital Transition Policy (Archives) & Accessibility Policy
• Emerging Open Research Policies
• Open Public Sector Information: From Principles to Practice Report
88
State and Territory Policies
99
Policies Components
APS policies in aggregate:
• Permissive copyright – CC-BY as the default
• Open by default, machine readable accessible data
• Support reuse and innovation
• More public engagement
• Better use of data for government policy and service development
States/Territories add:
• Procurement – open by design
• Reporting – dashboards
• Departmental strategies
• Declaration of Open Data
1010
Policy and Implementation in the Commonwealth
Department
of Finance
(CTO & AGIMO)
Department of
Communications
(Spatial Policy Branch)
Data Efficiency
Working Group
(Project 4)
Project 4
Implementation
Sub-group
Open Data
Delivery Network
(CTO & Spatial Policy Branch)
• Data.gov.au
• FIND
• NationalMap
• NEII
• ABS data delivery
• Identify datasets
• Release datasets
• Identify shared services
• Develop private/public
partnerships
• Spatial Policy
• Open/Big/Spatial
strategic planning
• Open & Big data policy
• Open/Big/Spatial
strategic planning
ImplementationPolicy and Planning
Finance
Communications
Joint
Open Data
Community Forum
(CTO coordinated)
• Cross-jurisdiction collaboration
• Data.gov.au publishing community
• Bi-monthly meetings & training
• Online forum
1111
History of open data in Australia
Gov 2.0 Taskforce Report 2009 (based on PoIT UK). Led to:
1) Declaration of Open Government
2) CC-BY as default
3) Information Commissioner
4) data.gov.au and social media support/policy
5) Cloud/shared services
Myriad supporting tech and copyright policies over time
States/Territories, Federal and Local now largely aligned on open data
1212
Data Portals
Local Portals:
• City of Melbourne
1313
Open Data Discovery Model
Analysis
& Policy
Visusalisation
& Maps
Application
development
ServicesValue Creation
Fed Discovery
Fed Data
Full Discovery
1414
data.gov.au
Free, cloud, scalable API enabled platform for hosting government data.
Staged approach
1. Publishing (2013 – mid 2014)
Improving the functionality and ease of
publishing for agencies with training and
documentation
2. Value realisation (Late 2014)
Providing useful front end tools for data.gov.au
including data visualisation and analysis tools.
Publishing quality data a pre-requisite.
3. Data quality (Late 2014)
Looking at ways to provide agencies the ability
to accept iterative data improvements in a
verifiable way
Features
• Federated search for discoverability
• Manual and automated publishing options
• API access to government data
• Easy to publish, download & interact
• Basic data visualisation capability
• Use cases and site/data/org analytics
• Data Request Site
In Planning
• 5 star quality plugin
• Data model registry
• Selective crowdsourcing for updates
• National Map integration
1515
NationalMap
1616
1717
The Government Data Landscape (latest version online)
1818
Other Data Projects
• Administrative
• Spatial
• Geosciences
• Research
• Sensor
• Realtime (eg Transport)
• Census/Statistics
• Cultural
• Data about government
• Fiscal
1919
Open by Design – drawing a line in the sand
Building proactive publishing into:
• Systems
• Processes
• Procurement
• Planning
• Records management
Leveraging open data through:
• Public APIs
• Analysis tools and datavis
• Internal processes looking for external sources
2020
New and Old Skills Required
• Publishing and Automation
• Project management, reporting
• Metadata/linked data
• API development and serving
• Plumbing between systems
• Data and info visualisation
• Analysis and statistics
• Policy development
• Public consultation and engagement
• Online community management
2121
Some Challenges
• Education
• Legislative
• Culture
• Systems
• Privacy and anonymisation
• Reactive vs proactive
• Metadata/semantic context
• Too much data
• Real time vs historic
• Definitions and common references
• Limited skills and over specialisation
2222
Hypothetical: Government as an API
Care of fedAPI.gov
2323
The future is here....
And it is already widely distributed
http://www.flickr.com/photos/mr_matt/35688926
22/

Open Data Presentation v1.3 - Nov 2014

  • 1.
    1 The Shift to{Open|Big|Linked} Data Or, how I learned to stop worrying and love the data Pia Waugh Director of Coordination and Gov 2.0 Technology and Procurement Division Department of Finance
  • 2.
  • 3.
  • 4.
  • 5.
    55 Key Benefits toGovernment in Opening Data • Improved services delivery • Efficiency gains • Better policy outcomes • More consistency across government(s) • Improved opportunities to leverage innovation and collaboration • Opportunities to improve data quality through verifiable public contributions
  • 6.
    66 Key Benefits toCommunity in Opening Data Economic • Creates opportunities for industry to value-add to government data • New services, systems and industries • New opportunities and innovation in industry, research, civil society Accountability • Visibility to government spending, projects, effectiveness, etc • Increases incentive to follow evidence based approach Better policy and programs • Enables greater participation in policy planning and implementation • More informed public → better decision making • Improvements to data → better policy and decisions
  • 7.
    77 The APS eGovand Open Data Policy Landscape Others: • Publishing Public Sector Information & National Standards Framework • Declaration of Open Government • Gov 2.0 Taskforce Report • Statement of IP Principles for Government (CC-BY) • Ahead of the Game • Digital Transition Policy (Archives) & Accessibility Policy • Emerging Open Research Policies • Open Public Sector Information: From Principles to Practice Report
  • 8.
  • 9.
    99 Policies Components APS policiesin aggregate: • Permissive copyright – CC-BY as the default • Open by default, machine readable accessible data • Support reuse and innovation • More public engagement • Better use of data for government policy and service development States/Territories add: • Procurement – open by design • Reporting – dashboards • Departmental strategies • Declaration of Open Data
  • 10.
    1010 Policy and Implementationin the Commonwealth Department of Finance (CTO & AGIMO) Department of Communications (Spatial Policy Branch) Data Efficiency Working Group (Project 4) Project 4 Implementation Sub-group Open Data Delivery Network (CTO & Spatial Policy Branch) • Data.gov.au • FIND • NationalMap • NEII • ABS data delivery • Identify datasets • Release datasets • Identify shared services • Develop private/public partnerships • Spatial Policy • Open/Big/Spatial strategic planning • Open & Big data policy • Open/Big/Spatial strategic planning ImplementationPolicy and Planning Finance Communications Joint Open Data Community Forum (CTO coordinated) • Cross-jurisdiction collaboration • Data.gov.au publishing community • Bi-monthly meetings & training • Online forum
  • 11.
    1111 History of opendata in Australia Gov 2.0 Taskforce Report 2009 (based on PoIT UK). Led to: 1) Declaration of Open Government 2) CC-BY as default 3) Information Commissioner 4) data.gov.au and social media support/policy 5) Cloud/shared services Myriad supporting tech and copyright policies over time States/Territories, Federal and Local now largely aligned on open data
  • 12.
  • 13.
    1313 Open Data DiscoveryModel Analysis & Policy Visusalisation & Maps Application development ServicesValue Creation Fed Discovery Fed Data Full Discovery
  • 14.
    1414 data.gov.au Free, cloud, scalableAPI enabled platform for hosting government data. Staged approach 1. Publishing (2013 – mid 2014) Improving the functionality and ease of publishing for agencies with training and documentation 2. Value realisation (Late 2014) Providing useful front end tools for data.gov.au including data visualisation and analysis tools. Publishing quality data a pre-requisite. 3. Data quality (Late 2014) Looking at ways to provide agencies the ability to accept iterative data improvements in a verifiable way Features • Federated search for discoverability • Manual and automated publishing options • API access to government data • Easy to publish, download & interact • Basic data visualisation capability • Use cases and site/data/org analytics • Data Request Site In Planning • 5 star quality plugin • Data model registry • Selective crowdsourcing for updates • National Map integration
  • 15.
  • 16.
  • 17.
    1717 The Government DataLandscape (latest version online)
  • 18.
    1818 Other Data Projects •Administrative • Spatial • Geosciences • Research • Sensor • Realtime (eg Transport) • Census/Statistics • Cultural • Data about government • Fiscal
  • 19.
    1919 Open by Design– drawing a line in the sand Building proactive publishing into: • Systems • Processes • Procurement • Planning • Records management Leveraging open data through: • Public APIs • Analysis tools and datavis • Internal processes looking for external sources
  • 20.
    2020 New and OldSkills Required • Publishing and Automation • Project management, reporting • Metadata/linked data • API development and serving • Plumbing between systems • Data and info visualisation • Analysis and statistics • Policy development • Public consultation and engagement • Online community management
  • 21.
    2121 Some Challenges • Education •Legislative • Culture • Systems • Privacy and anonymisation • Reactive vs proactive • Metadata/semantic context • Too much data • Real time vs historic • Definitions and common references • Limited skills and over specialisation
  • 22.
    2222 Hypothetical: Government asan API Care of fedAPI.gov
  • 23.
    2323 The future ishere.... And it is already widely distributed http://www.flickr.com/photos/mr_matt/35688926 22/