Open local data presentation for okcon

1,590 views

Published on

Presentation given to OKCON 2010 (Open Knowledge Foundation Conference), held at ULU, London, April 24. Note this is a slightly updated and reworked version of the presentation given to the Manchester Social Media Cafe on April 6

Published in: News & Politics
  • Be the first to comment

Open local data presentation for okcon

  1. 1. Open Local Data, OpenlyLocal & the Open Election Data project Open Knowledge Conference, London, April 24, 2010
  2. 2. It’s been a good year for open government data data.gov.uk, Ordnance Survey, MPs expenses...
  3. 3. It’s been a good year for open government data data.gov.uk, Ordnance Survey, MPs expenses...
  4. 4. It’s been a good year for open government data data.gov.uk, Ordnance Survey, MPs expenses...
  5. 5. But what about local data?
  6. 6. But what about local data? • Frankly it’s a mess
  7. 7. But what about local data? • Frankly it’s a mess • Sporadically published by central government
  8. 8. But what about local data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites
  9. 9. But what about local data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites • Opaque local public bodies and NDPBs (quangos)
  10. 10. But what about local data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites • Opaque local public bodies and NDPBs (quangos) • At best unclear & at worst unusable legal situation
  11. 11. But what about local data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites • Opaque local public bodies and NDPBs (quangos) • At best unclear & at worst unusable legal situation • Start with the basics. Who are the councillors, where do they represent and what committees do they sit on?
  12. 12. But what about local data? • Frankly it’s a mess • Sporadically published by central government • Inaccessible & impenetrable council websites • Opaque local public bodies and NDPBs (quangos) • At best unclear & at worst unusable legal situation • Start with the basics. Who are the councillors, where do they represent and what committees do they sit on? • How easy it that information to find & reuse?
  13. 13. Enter OpenlyLocal...
  14. 14. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. First code written 1 year ago
  15. 15. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. First code written 1 year ago • Screen-scrapes council websites. Now over 140 councils done, with basic information on all 434 in the UK
  16. 16. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. First code written 1 year ago • Screen-scrapes council websites. Now over 140 councils done, with basic information on all 434 in the UK • Pulls info from about 150 local councils, plus ONS, OS, NPIA, CLG...
  17. 17. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. First code written 1 year ago • Screen-scrapes council websites. Now over 140 councils done, with basic information on all 434 in the UK • Pulls info from about 150 local councils, plus ONS, OS, NPIA, CLG... • Building a graph of connections – councillors, committees, social networking, police forces, pension funds
  18. 18. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. First code written 1 year ago • Screen-scrapes council websites. Now over 140 councils done, with basic information on all 434 in the UK • Pulls info from about 150 local councils, plus ONS, OS, NPIA, CLG... • Building a graph of connections – councillors, committees, social networking, police forces, pension funds • All open data – free for reuse, including commercial reuse
  19. 19. Enter OpenlyLocal... • Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem. First code written 1 year ago • Screen-scrapes council websites. Now over 140 councils done, with basic information on all 434 in the UK • Pulls info from about 150 local councils, plus ONS, OS, NPIA, CLG... • Building a graph of connections – councillors, committees, social networking, police forces, pension funds • All open data – free for reuse, including commercial reuse • 100% accessible. Data first; bells & whistles later
  20. 20. RDF
  21. 21. RDF JSON
  22. 22. RDF JSON XML
  23. 23. OpenlyLocal: where next?
  24. 24. OpenlyLocal: where next? • More data. More councils. More connections.
  25. 25. OpenlyLocal: where next? • More data. More councils. More connections. • Election data, cabinet members, mayors, police authorities, health authorities, pension funds, etc... it’s all about the network
  26. 26. OpenlyLocal: where next? • More data. More councils. More connections. • Election data, cabinet members, mayors, police authorities, health authorities, pension funds, etc... it’s all about the network • Data used by Hyperlocal sites (inc Guardian). Ning app. Google Gadget. Widget. And now starting to get info in the other direction (i.e. connecting blog posts to councillors)
  27. 27. OpenlyLocal: where next? • More data. More councils. More connections. • Election data, cabinet members, mayors, police authorities, health authorities, pension funds, etc... it’s all about the network • Data used by Hyperlocal sites (inc Guardian). Ning app. Google Gadget. Widget. And now starting to get info in the other direction (i.e. connecting blog posts to councillors) • Some things can’t be done programmatically. Need crowdsourcing tools & help with tying info to Wikipedia, other sources
  28. 28. OpenlyLocal: where next? • More data. More councils. More connections. • Election data, cabinet members, mayors, police authorities, health authorities, pension funds, etc... it’s all about the network • Data used by Hyperlocal sites (inc Guardian). Ning app. Google Gadget. Widget. And now starting to get info in the other direction (i.e. connecting blog posts to councillors) • Some things can’t be done programmatically. Need crowdsourcing tools & help with tying info to Wikipedia, other sources • Some good visualisations done, but lots more potential, particularly now that we have OS geo data
  29. 29. Why does open local data matter?
  30. 30. Why does open local data matter? • Transparency – can we see what’s going on (and understand it too)
  31. 31. Why does open local data matter? • Transparency – can we see what’s going on (and understand it too) • Engagement – reducing barriers to getting involved
  32. 32. Why does open local data matter? • Transparency – can we see what’s going on (and understand it too) • Engagement – reducing barriers to getting involved • Equality of access to information (and thus power). Much of this data is already available... for a price
  33. 33. Why does open local data matter? • Transparency – can we see what’s going on (and understand it too) • Engagement – reducing barriers to getting involved • Equality of access to information (and thus power). Much of this data is already available... for a price • Efficiency – Don’t let five different branches of government needlessly do the same thing (aka keeping it DRY)
  34. 34. Simple questions, hard to answer
  35. 35. Simple questions, hard to answer • How does the budget of my council compare with similar ones?
  36. 36. Simple questions, hard to answer • How does the budget of my council compare with similar ones? • What’s the background of the chief executive of my council?
  37. 37. Simple questions, hard to answer • How does the budget of my council compare with similar ones? • What’s the background of the chief executive of my council? • Is my route to work going to affected by roadworks next week?
  38. 38. Simple questions, hard to answer • How does the budget of my council compare with similar ones? • What’s the background of the chief executive of my council? • Is my route to work going to affected by roadworks next week? • What are the connections between the companies awarded contracts and the councillors & senior management?
  39. 39. Simple questions, hard to answer • How does the budget of my council compare with similar ones? • What’s the background of the chief executive of my council? • Is my route to work going to affected by roadworks next week? • What are the connections between the companies awarded contracts and the councillors & senior management? • How will cuts in services be decided and what can I do to influence the decisions?
  40. 40. Transparency A brief example.
  41. 41. Transparency A brief example. • A story in Private Eye
  42. 42. Transparency A brief example. • A story in Private Eye
  43. 43. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor
  44. 44. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website...
  45. 45. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website...
  46. 46. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading
  47. 47. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading
  48. 48. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading • Accessible only as a PDF of a scan of a document
  49. 49. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading • Accessible only as a PDF of a scan of a document
  50. 50. Transparency A brief example. • A story in Private Eye • An investigation by the District Auditor • A report buried in the nether recesses of the website... • ...with no relevant heading • Accessible only as a PDF of a scan of a document • Is it any wonder suspicions are raised?
  51. 51. Engagement
  52. 52. Engagement • Local Authorities now have a duty to engage. This means • Reducing barriers to understanding and involvement
  53. 53. Engagement • Local Authorities now have a duty to engage. This means • Reducing barriers to understanding and involvement • Accessible for all
  54. 54. Engagement • Local Authorities now have a duty to engage. This means • Reducing barriers to understanding and involvement • Accessible for all • Available to use when, where & how we want to
  55. 55. Engagement • Local Authorities now have a duty to engage. This means • Reducing barriers to understanding and involvement • Accessible for all • Available to use when, where & how we want to • Publishing as data means easy to be repurposed – for mobile, for mashups, for offline use (e.g. Postcode Paper)
  56. 56. Equality
  57. 57. Equality • At the moment, all this information is available... at a cost
  58. 58. Equality ££ £ • At the moment, all this information is available... at a cost
  59. 59. Equality ££ £ • At the moment, all this information is available... at a cost • Huge asymmetry of information (and thus power)
  60. 60. Equality ££ £ • At the moment, all this information is available... at a cost • Huge asymmetry of information (and thus power) • Raises the barriers to involvement, and, critically, to challenge
  61. 61. Efficiency Now: cumbersome, wasteful, opaque, error prone Typical local data flows
  62. 62. Efficiency Now: cumbersome, wasteful, opaque, error prone Typical local data flows Council
  63. 63. Efficiency Now: cumbersome, wasteful, opaque, error prone Typical local data flows Govt Dept A email web Council Govt Dept B upload Govt Dept C
  64. 64. Efficiency Now: cumbersome, wasteful, opaque, error prone Typical local data flows Govt Dept A public (maybe) email web Council Govt Dept B upload Govt Dept C
  65. 65. Efficiency A better way? Council
  66. 66. Efficiency A better way? open data Council
  67. 67. Efficiency A better way?  <tr rel='openelection:candidacy'>         <!-- Here we are saying the candidate has given names of Keith William and a familyName Cockroft -->         <th scope="row" rel="openelection:candidate"><span property="foaf:givenName">Keith William</span> <span property="foaf:familyName">Cockroft</span></th>             <td rel="openelection:party" resource="http:// openelectiondata.org/id/parties/6"><span property="rdfs:label">Labour</span></td>         <!-- The candidacy got 330 votes -->        <td property="openelection:candidateVoteCount" datatype="xsd:integer">330</td>         <td>16.6%</td> open data Council          <td property="openelection:elected" datatype="xsd:boolean" content="false">No</td>       </tr>      <tr rel='openelection:candidacy'>         <th scope="row" rel="openelection:candidate">           <span typeof="openelection:Candidate" property="foaf:name">Brenda Lilian Constable</span>           <span rel="openelection:address">             <span property="v:street-address">37 Morley Road</ span>,             <span property="v:locality">Burntwood</span>,             <span property="v:region">Staffordshire</span>             <span property="v:postal-code">WS7 2DE</span>           </span>         </th>        
  68. 68. Efficiency A better way?  <tr rel='openelection:candidacy'>         <!-- Here we are saying the candidate has given names of Keith William and a familyName Cockroft -->         <th scope="row" rel="openelection:candidate"><span public property="foaf:givenName">Keith William</span> <span property="foaf:familyName">Cockroft</span></th>             <td rel="openelection:party" resource="http:// openelectiondata.org/id/parties/6"><span property="rdfs:label">Labour</span></td>         <!-- The candidacy got 330 votes -->        <td property="openelection:candidateVoteCount" open data datatype="xsd:integer">330</td>         <td>16.6%</td> Govt Dept A Council          <td property="openelection:elected" datatype="xsd:boolean" content="false">No</td>       </tr>      <tr rel='openelection:candidacy'>         <th scope="row" rel="openelection:candidate"> Govt Dept B           <span typeof="openelection:Candidate" property="foaf:name">Brenda Lilian Constable</span>           <span rel="openelection:address">             <span property="v:street-address">37 Morley Road</ span>, Govt Dept C             <span property="v:locality">Burntwood</span>,             <span property="v:region">Staffordshire</span>             <span property="v:postal-code">WS7 2DE</span>           </span>         </th>        
  69. 69. Efficiency A better way?  <tr rel='openelection:candidacy'>         <!-- Here we are saying the candidate has given names of Keith William and a familyName Cockroft -->         <th scope="row" rel="openelection:candidate"><span public property="foaf:givenName">Keith William</span> <span property="foaf:familyName">Cockroft</span></th>             <td rel="openelection:party" resource="http:// openelectiondata.org/id/parties/6"><span property="rdfs:label">Labour</span></td>         <!-- The candidacy got 330 votes -->        <td property="openelection:candidateVoteCount" open data datatype="xsd:integer">330</td>         <td>16.6%</td> Govt Dept A Council          <td property="openelection:elected" datatype="xsd:boolean" content="false">No</td>       </tr>      <tr rel='openelection:candidacy'>         <th scope="row" rel="openelection:candidate"> Govt Dept B           <span typeof="openelection:Candidate" property="foaf:name">Brenda Lilian Constable</span>           <span rel="openelection:address">             <span property="v:street-address">37 Morley Road</ span>, Govt Dept C             <span property="v:locality">Burntwood</span>,             <span property="v:region">Staffordshire</span>             <span property="v:postal-code">WS7 2DE</span>           </span>         </th>         Publish once, consume many times
  70. 70. Problems
  71. 71. Problems • IDs – councils don’t generally use them; central government uses many... inconsistently
  72. 72. Problems • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?)
  73. 73. Problems • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?) • Legacy systems. That nobody now understands
  74. 74. Problems • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?) • Legacy systems. That nobody now understands • Private companies/JVs. No FoI. No access to data
  75. 75. Problems • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?) • Legacy systems. That nobody now understands • Private companies/JVs. No FoI. No access to data • Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers)
  76. 76. Problems • IDs – councils don’t generally use them; central government uses many... inconsistently • Data tied up in PDFs (will we ever get this back?) • Legacy systems. That nobody now understands • Private companies/JVs. No FoI. No access to data • Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers) • Silos, especially of budgets. How do you get money from one budget to another (even if the whole benefits)
  77. 77. The OpenElectionData project
  78. 78. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way
  79. 79. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward
  80. 80. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward • No public database of local election results, only a commercial one (subsidized by the Electoral Commission)
  81. 81. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward • No public database of local election results, only a commercial one (subsidized by the Electoral Commission) • Allows even those with no prior knowledge of linked data/RDF/ semantic web(choose fave buzzword here) to take part
  82. 82. The OpenElectionData project • Tackling the open local data problem, one set at a time & learning lessons on the way • Succeed, or fail forward • No public database of local election results, only a commercial one (subsidized by the Electoral Commission) • Allows even those with no prior knowledge of linked data/RDF/ semantic web(choose fave buzzword here) to take part • Just need HTML competence
  83. 83. Open Election Data project How it works • Instead of publishing their election results as arbitrary HTML they publish it as HTML that has been semantically marked up. • e.g. <tr rel='openelection:candidacy'>         <th scope="row" rel="openelection:candidate"><span typeof="openelection:Candidate" property="foaf:name">Helen Elizabeth Fisher</span></th>      <td rel="openelection:party" resource="http://openelectiondata.org/id/parties/25"><span property="rdfs:label">Conservative</span></td>      <td property="openelection:candidateVoteCount" datatype="xsd:integer">655</td>         <td>33.0%</td>      <td property="openelection:elected" datatype="xsd:boolean" content="true">Yes</td> </tr> instead of <tr class='zebraBandOdd'>     <th scope="row" class='darkGreyCell'><span class='boldEntry'>Helen Elizabeth Fisher</span></th>     <td class='lightCell'>Conservative</td>     <td class='lightCell'>655</td>     <td class='lightCell'>33.0%</td>     <td class='lightCell'>Yes</td> </tr>
  84. 84. Open Election Data project How it works
  85. 85. Open Election Data project How it works • Allows:
  86. 86. Open Election Data project How it works • Allows: • data to be pulled from the page, and consumed by RDF reader
  87. 87. Open Election Data project How it works • Allows: • data to be pulled from the page, and consumed by RDF reader • Local Authorities to develop an understanding of the key issues of open linked data - e.g. URIs as identifiers
  88. 88. Open Election Data project How it works • Allows: • data to be pulled from the page, and consumed by RDF reader • Local Authorities to develop an understanding of the key issues of open linked data - e.g. URIs as identifiers • the beginnings of a complete database of local election results
  89. 89. Open Election Data project How it works • Allows: • data to be pulled from the page, and consumed by RDF reader • Local Authorities to develop an understanding of the key issues of open linked data - e.g. URIs as identifiers • the beginnings of a complete database of local election results • a concrete way of identifying the blockers & pain points when moving towards open local data
  90. 90. How’s it going? Too early to say but
  91. 91. How’s it going? Too early to say but • After the May 6 election (local elections at 166 councils) will be somewhere between 12 and 40 councils publishing their data this way (too early to give more accurate figure yet, lots joining at last minute)
  92. 92. How’s it going? Too early to say but • After the May 6 election (local elections at 166 councils) will be somewhere between 12 and 40 councils publishing their data this way (too early to give more accurate figure yet, lots joining at last minute) • Already we have increased knowledge and publishing of open data by councils by an order of magnitude, and have established informal network of people who get it
  93. 93. How’s it going? Too early to say but • After the May 6 election (local elections at 166 councils) will be somewhere between 12 and 40 councils publishing their data this way (too early to give more accurate figure yet, lots joining at last minute) • Already we have increased knowledge and publishing of open data by councils by an order of magnitude, and have established informal network of people who get it • Council people have begun to help each other, solving problems, writing scripts for internal CMSs, and it now has its own momentum
  94. 94. How’s it going? Too early to say but • After the May 6 election (local elections at 166 councils) will be somewhere between 12 and 40 councils publishing their data this way (too early to give more accurate figure yet, lots joining at last minute) • Already we have increased knowledge and publishing of open data by councils by an order of magnitude, and have established informal network of people who get it • Council people have begun to help each other, solving problems, writing scripts for internal CMSs, and it now has its own momentum • Core blockers already becoming clear: lack of basic skills (even HTML); intransigent middle managers; inflexible IT systems
  95. 95. Where next?
  96. 96. Where next? • A Freedom of Data act – too many different bodies
  97. 97. Where next? • A Freedom of Data act – too many different bodies • A recasting of how Government (central & local) does IT. Avoiding the big #fail
  98. 98. Where next? • A Freedom of Data act – too many different bodies • A recasting of how Government (central & local) does IT. Avoiding the big #fail • A change in relationship between central & local government
  99. 99. Where next? • A Freedom of Data act – too many different bodies • A recasting of how Government (central & local) does IT. Avoiding the big #fail • A change in relationship between central & local government • A change between between government and citizen
  100. 100. Where next? • A Freedom of Data act – too many different bodies • A recasting of how Government (central & local) does IT. Avoiding the big #fail • A change in relationship between central & local government • A change between between government and citizen • Some new business models – to support hyperlocal sites, innovative and supportive suppliers, public-interest groups
  101. 101. Where next? • A Freedom of Data act – too many different bodies • A recasting of how Government (central & local) does IT. Avoiding the big #fail • A change in relationship between central & local government • A change between between government and citizen • Some new business models – to support hyperlocal sites, innovative and supportive suppliers, public-interest groups • Develop the meme: enabler or blocker

×