Download as KEY, PPTX
Uploaded byChris Taggart
Open local data: challenges and opportunities
The document discusses challenges and opportunities around open local data in the UK. It notes that while central government data has become more open, local data remains difficult to access and use. Local data is sporadically published, hosted on inaccessible council websites, and the legal status is unclear. Obtaining even basic information like councilor details and budgets can be challenging. The document advocates for more transparent and open local data to empower citizens and enable innovative uses of the data. It presents the OpenlyLocal project, which scrapes and standardizes local council data with the goal of making it freely available and usable.
More Related Content
Similar to Open local data: challenges and opportunities
Open local data: challenges and opportunities
- 1. Open Local Data Challenges & opportunities WMRO Open Data Event, Birmingham, July 15, 2010
- 2. It’s been agood year for open government data data.gov.uk, Ordnance Survey, MPs expenses...
- 3. It’s been agood year for open government data data.gov.uk, Ordnance Survey, MPs expenses...
- 4. It’s been agood year for open government data data.gov.uk, Ordnance Survey, MPs expenses...
- 5. But what aboutlocal data?
- 6. But what aboutlocal data? • Frankly it’s a mess
- 7. But what aboutlocal data? • Frankly it’s a mess • Sporadically published by central government
- 8. But what aboutlocal data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites
- 9. But what aboutlocal data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites • Opaque local public bodies and quangos (goodbye RDAs, hello LEPs)
- 10. But what aboutlocal data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites • Opaque local public bodies and quangos (goodbye RDAs, hello LEPs) • At best unclear & at worst unusable legal situation
- 11. But what aboutlocal data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites • Opaque local public bodies and quangos (goodbye RDAs, hello LEPs) • At best unclear & at worst unusable legal situation • Start with the basics. Who are the councillors, where do they represent and what committees do they sit on?
- 12. But what aboutlocal data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites • Opaque local public bodies and quangos (goodbye RDAs, hello LEPs) • At best unclear & at worst unusable legal situation • Start with the basics. Who are the councillors, where do they represent and what committees do they sit on? • How easy it that information to find & reuse?
- 15. Simple questions, hardto answer
- 16. Simple questions, hardto answer • How does the budget of my council compare with similar ones?
- 17. Simple questions, hardto answer • How does the budget of my council compare with similar ones? • What’s the background of the chief executive of my council?
- 18. Simple questions, hardto answer • How does the budget of my council compare with similar ones? • What’s the background of the chief executive of my council? • Is my route to work going to affected by roadworks next week?
- 19. Simple questions, hardto answer • How does the budget of my council compare with similar ones? • What’s the background of the chief executive of my council? • Is my route to work going to affected by roadworks next week? • What are the connections between the companies awarded contracts and the councillors & senior management?
- 20. Simple questions, hardto answer • How does the budget of my council compare with similar ones? • What’s the background of the chief executive of my council? • Is my route to work going to affected by roadworks next week? • What are the connections between the companies awarded contracts and the councillors & senior management? • How will cuts in services be decided and what can I do to influence the decisions?
- 21.
- 22. Transparency A brief example. • A story in Private Eye
- 23. Transparency A brief example. • A story in Private Eye
- 24. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor
- 25. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website...
- 26. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website...
- 27. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading
- 28. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading
- 29. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading • Accessible only as a PDF of a scan of a document
- 30. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading • Accessible only as a PDF of a scan of a document
- 31. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading • Accessible only as a PDF of a scan of a document • Is it any wonder suspicions are raised?
- 32.
- 33. Engagement • Local Authorities now have a duty to engage. This means • Reducing barriers to understanding and involvement
- 34. Engagement • Local Authorities now have a duty to engage. This means • Reducing barriers to understanding and involvement • Accessible for all
- 35. Engagement • Local Authorities now have a duty to engage. This means • Reducing barriers to understanding and involvement • Accessible for all • Available to use when, where & how we want to
- 36. Engagement • Local Authorities now have a duty to engage. This means • Reducing barriers to understanding and involvement • Accessible for all • Available to use when, where & how we want to • Publishing as data means easy to be repurposed – for mobile, for mashups, for offline use (e.g. Postcode Paper)
- 37.
- 38. Equality • At the moment, all this information is available... at a cost
- 39. Equality ££ £ • At the moment, all this information is available... at a cost
- 40. Equality ££ £ • At the moment, all this information is available... at a cost • Huge asymmetry of information (and thus power)
- 41. Equality ££ £ • At the moment, all this information is available... at a cost • Huge asymmetry of information (and thus power) • Raises the barriers to involvement, and, critically, to challenge
- 42. Efficiency Now: cumbersome, wasteful,opaque, error prone Typical local data flows
- 43. Efficiency Now: cumbersome, wasteful,opaque, error prone Typical local data flows Council
- 44. Efficiency Now: cumbersome, wasteful,opaque, error prone Typical local data flows Govt Dept A email web Council Govt Dept B upload Govt Dept C
- 45. Efficiency Now: cumbersome, wasteful,opaque, error prone Typical local data flows Govt Dept A public (maybe) email web Council Govt Dept B upload Govt Dept C
- 46.
- 47. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.
- 48. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. • Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434
- 49. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. • Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434 • Pulls info from about local councils, plus ONS, OS, NPIA, CLG...
- 50. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. • Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434 • Pulls info from about local councils, plus ONS, OS, NPIA, CLG... • Increasingly working with local authorities to help them publish their information as open data
- 51. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. • Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434 • Pulls info from about local councils, plus ONS, OS, NPIA, CLG... • Increasingly working with local authorities to help them publish their information as open data • All open data – free for reuse, including commercial reuse
- 52. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. • Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434 • Pulls info from about local councils, plus ONS, OS, NPIA, CLG... • Increasingly working with local authorities to help them publish their information as open data • All open data – free for reuse, including commercial reuse • 100% accessible. Data first; bells & whistles later
- 57.
- 58.
- 59. RDF JSON XML
- 60.
- 61. OpenlyLocal: where next? • More data.
- 62. OpenlyLocal: where next? • More data. • Election data
- 63. OpenlyLocal: where next? • More data. • Election data ✓
- 64. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data
- 65. OpenlyLocal: where next? • More data. • ✓ Election data • Spending data ✓
- 66. OpenlyLocal: where next? • More data. • ✓ Election data • Spending data ✓ • Where I live...
- 67. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data ✓ • Where I live... ✓
- 68. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data ✓ • Where I live... ✓ • Cabinet members
- 69. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data ✓ • Where I live... ✓ • Cabinet members • Mayors
- 70. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data ✓ • Where I live... ✓ • Cabinet members • Mayors • Police authorities
- 71. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data ✓ • Where I live... ✓ • Cabinet members • Mayors • Police authorities • More councils.
- 72. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data ✓ • Where I live... ✓ • Cabinet members • Mayors • Police authorities • More councils. • More connections.
- 73. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data ✓ • Where I live... ✓ • Cabinet members • Mayors • Police authorities • More councils. • More connections. • More ways for councils to give us data.
- 74. OpenlyLocal: where next? • More data. • Election data ✓ • Spending data ✓ • Where I live... ✓ • Cabinet members • Mayors • Police authorities • More councils. • More connections. • More ways for councils to give us data. • More tie-ups with hyperlocal news/community sites.
- 75.
- 76. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way
- 77. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward
- 78. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward • No public database of local election results, only a commercial one (subsidized by the Electoral Commission)
- 79. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward • No public database of local election results, only a commercial one (subsidized by the Electoral Commission) • Allows even those with no prior knowledge of linked data/RDF/ semantic web (choose fave buzzword here) to take part
- 80. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward • No public database of local election results, only a commercial one (subsidized by the Electoral Commission) • Allows even those with no prior knowledge of linked data/RDF/ semantic web (choose fave buzzword here) to take part • Just need HTML competence
- 81. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward • No public database of local election results, only a commercial one (subsidized by the Electoral Commission) • Allows even those with no prior knowledge of linked data/RDF/ semantic web (choose fave buzzword here) to take part • Just need HTML competence • Over 20 councils took part, and invaluable lessons learned (see report)
- 82. The future forlocal data? Local government for local people <tr rel='openelection:candidacy'> public <!-- Here we are saying the candidate has given names of interest/ Keith William and a familyName Cockroft --> <th scope="row" rel="openelection:candidate"><span media/ public property="foaf:givenName">Keith William</span> <span property="foaf:familyName">Cockroft</span></th> community/ <td rel="openelection:party" resource="http:// openelectiondata.org/id/parties/6"><span commercial property="rdfs:label">Labour</span></td> sites <!-- The candidacy got 330 votes --> <td property="openelection:candidateVoteCount" open data datatype="xsd:integer">330</td> <td>16.6%</td> Govt Dept A Council maybe <td property="openelection:elected" datatype="xsd:boolean" content="false">No</td> </tr> <tr rel='openelection:candidacy'> <th scope="row" rel="openelection:candidate"> Govt Dept B <span typeof="openelection:Candidate" property="foaf:name">Brenda Lilian Constable</span> <span rel="openelection:address"> <span property="v:street-address">37 Morley Road</ span>, Govt Dept C <span property="v:locality">Burntwood</span>, <span property="v:region">Staffordshire</span> <span property="v:postal-code">WS7 2DE</span> </span> </th> Publish once, consume many times
- 83.
- 84. Open data opportunities ✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks
- 85. Open data opportunities ✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks ✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens)
- 86. Open data opportunities ✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks ✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens) ✤ Improved online services for citizens at zero cost (provided by websites/apps that work with any council)
- 87. Open data opportunities ✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks ✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens) ✤ Improved online services for citizens at zero cost (provided by websites/apps that work with any council) ✤ Increased transparency
- 88. Open data opportunities ✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks ✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens) ✤ Improved online services for citizens at zero cost (provided by websites/apps that work with any council) ✤ Increased transparency ✤ More freedom from central government targets, auditing
- 89.
- 90. Challenges ✤ Lack of ‘corporate’ awareness/understanding of open data issues
- 91. Challenges ✤ Lack of ‘corporate’ awareness/understanding of open data issues ✤ There is a lack of even basic web skills at some councils
- 92. Challenges ✤ Lack of ‘corporate’ awareness/understanding of open data issues ✤ There is a lack of even basic web skills at some councils ✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing
- 93. Challenges ✤ Lack of ‘corporate’ awareness/understanding of open data issues ✤ There is a lack of even basic web skills at some councils ✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing ✤ The understanding of even the basics of linked data and the steps to publishing public data in this way is very, very limited, even outside of the public sector
- 94. Challenges ✤ Lack of ‘corporate’ awareness/understanding of open data issues ✤ There is a lack of even basic web skills at some councils ✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing ✤ The understanding of even the basics of linked data and the steps to publishing public data in this way is very, very limited, even outside of the public sector ✤ Getting councils to publish data in open, linked formats requires dedication, and a range of resources and a mix of skills
- 95. Challenges ✤ Lack of ‘corporate’ awareness/understanding of open data issues ✤ There is a lack of even basic web skills at some councils ✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing ✤ The understanding of even the basics of linked data and the steps to publishing public data in this way is very, very limited, even outside of the public sector ✤ Getting councils to publish data in open, linked formats requires dedication, and a range of resources and a mix of skills ✤ Outsourced web publishing contracts often make it difficult and expensive to publish open, linked data
- 96.
- 97. More Challenges • IDs – councils don’t generally use them; central government uses many... inconsistently
- 98. More Challenges • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?)
- 99. More Challenges • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?) • Legacy systems. That nobody now understands
- 100. More Challenges • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?) • Legacy systems. That nobody now understands • Private companies/JVs. No FoI. No access to data
- 101. More Challenges • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?) • Legacy systems. That nobody now understands • Private companies/JVs. No FoI. No access to data • Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers)
- 102. More Challenges • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?) • Legacy systems. That nobody now understands • Private companies/JVs. No FoI. No access to data • Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers) • Silos, especially of budgets. How do you get money from one budget to another (even if the whole benefits)
- 103.
- 104. Yet More Challenges ✤ The size of the cuts mean they will be blunt, eliminating useful services, sparing wasteful ones. Who & what will be left & what will it mean for local public services?
- 105. Yet More Challenges ✤ The size of the cuts mean they will be blunt, eliminating useful services, sparing wasteful ones. Who & what will be left & what will it mean for local public services? ✤ Few resources to invest in doing this, and even less understanding of where to invest it. Danger of using usual procedures, suppliers (procurement, Capita, Serco etc)
- 106. Yet More Challenges ✤ The size of the cuts mean they will be blunt, eliminating useful services, sparing wasteful ones. Who & what will be left & what will it mean for local public services? ✤ Few resources to invest in doing this, and even less understanding of where to invest it. Danger of using usual procedures, suppliers (procurement, Capita, Serco etc) ✤ Little incentive for those organisations and people threatened by open data – for them it is a Darwinian fight for survival
- 107. An example dataset:local spending data (deadline jan 2011)
- 108. An example dataset:local spending data (deadline jan 2011) ✤ Technical challenge: how to extract the data from the accounts system without including non-suppliers (e.g. foster parents, children) ✤ How many councils are working on this (hopefully all of them) ✤ How many have blogged, posted the steps/issues/solutions – zero? ✤ What are the key fields that MUST be included, and do these need to be listed somewhere?
- 109. An example dataset:local spending data (deadline jan 2011) ✤ Technical challenge: how to extract the data from the accounts system without including non-suppliers (e.g. foster parents, children) ✤ How many councils are working on this (hopefully all of them) ✤ How many have blogged, posted the steps/issues/solutions – zero? ✤ What are the key fields that MUST be included, and do these need to be listed somewhere? ✤ Organizational challenge ✤ What is this going show about our spending/processes/contracts? ✤ How will we compare with other similar bodies? ✤ What will be the consequences (FoI requests, changed relationship with suppliers, unpopular services)
- 110. An example dataset:local spending data (deadline jan 2011) ✤ Technical challenge: how to extract the data from the accounts system without including non-suppliers (e.g. foster parents, children) ✤ How many councils are working on this (hopefully all of them) ✤ How many have blogged, posted the steps/issues/solutions – zero? ✤ What are the key fields that MUST be included, and do these need to be listed somewhere? ✤ Organizational challenge ✤ What is this going show about our spending/processes/contracts? ✤ How will we compare with other similar bodies? ✤ What will be the consequences (FoI requests, changed relationship with suppliers, unpopular services) ✤ Systemic challenge ✤ How will this affect the relationship with the citizens? ✤ What are the implications for our structure, processes, power...?
- 111. Power in thepublic sector was... ✤ Access to people ✤ Access to information ✤ Understanding structure/how things worked
- 112. Power in thepublic sector was... ocial net working Access to peopleby s dermined ✤ un ✤ Access to information ✤ Understanding structure/how things worked
- 113. Power in thepublic sector was... ocial net working Access to peopleby s dermined ✤ un Access to information o y pen data ratized b ✤ democ ✤ Understanding structure/how things worked
- 114. Power in thepublic sector was... ocial net working Access to peopleby s dermined ✤ un Access to information o y pen data ratized b ✤ democ ✤ Understanding urced? b ypassed? worked structure/how things crowdso
- 115. All this isa proxy for the real battle: relevance
- 116. All this isa proxy for the real battle: relevance ✤ How relevant is your local government to you?
- 117. All this isa proxy for the real battle: relevance ✤ How relevant is your local government to you? ✤ How well do the ‘proper channels’ work for you?
- 118. All this isa proxy for the real battle: relevance ✤ How relevant is your local government to you? ✤ How well do the ‘proper channels’ work for you? ✤ What is your ability to achieve things outside the existing structures?
- 119. All this isa proxy for the real battle: relevance ✤ How relevant is your local government to you? ✤ How well do the ‘proper channels’ work for you? ✤ What is your ability to achieve things outside the existing structures? ✤ What difference does your council make to your life?
- 120. Open data islocal government’s chance to stay relevant to their citizens/economy (but it’ll have its work cut out)