The document discusses the importance of open data, its benefits for engagement, equality, efficiency, and transparency, while highlighting the challenges faced by local authorities in publishing such data. It emphasizes that fostering an open data culture can help shift incentives towards innovation and learning from failures. The document also suggests a need to reframe the conversation around failure, advocating for continuous improvement and transparency in public services.

1. Open Data
&
The Rewards
of Failure
Future Everything Conference, Manchester, May 23 2010

2. Me: Chris Taggart
@countculture

3. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com

4. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com

5. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form

6. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites

7. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites

8. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites
✤ All open data

9. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites RDF JSON XML
✤ All open data

10. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites RDF JSON XML
✤ All open data

11. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites RDF JSON XML
✤ All open data
✤ Open Election Data project

12. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites RDF JSON XML
✤ All open data
✤ Open Election Data project

13. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites RDF JSON XML
✤ All open data
✤ Open Election Data project
✤ Helping local authorities publish their
election results as open data

14. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites RDF JSON XML
✤ All open data
✤ Open Election Data project
✤ Helping local authorities publish their
election results as open data
✤ Tackling the open local data problem,
one set at a time & learning lessons on
the way

15. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites RDF JSON XML
✤ All open data
✤ Open Election Data project
✤ Helping local authorities publish their
election results as open data
✤ Tackling the open local data problem,
one set at a time & learning lessons on
the way

16. Me: Chris Taggart
@countculture
✤ Developer of OpenlyLocal.com
✤ Makes local data (esp govt) available
in consistent and accessible form
✤ Scrapes council and govt websites RDF JSON XML
✤ All open data
✤ Open Election Data project
✤ Helping local authorities publish their
election results as open data
✤ Tackling the open local data problem,
one set at a time & learning lessons on
the way
✤ Member of Local Public Data Panel

17. Open data: why do we want it?

18. Open data: why do we want it?
✤ Stupid question: It’s our data

19. Open data: why do we want it?
✤ Stupid question: It’s our data

20. Open data: why do we want it?
✤ Stupid question: It’s our data
✤ Engagement – reducing barriers to
getting involved

21. Open data: why do we want it?
✤ Stupid question: It’s our data
✤ Engagement – reducing barriers to
getting involved

22. Open data: why do we want it?
✤ Stupid question: It’s our data
✤ Engagement – reducing barriers to
getting involved
✤ Equality of access to information (and
thus power). Much of this data is
already available... for a price

23. Open data: why do we want it?
✤ Stupid question: It’s our data
✤ Engagement – reducing barriers to
getting involved
✤ Equality of access to information (and
thus power). Much of this data is
already available... for a price

24. Open data: why do we want it?
✤ Stupid question: It’s our data
✤ Engagement – reducing barriers to
getting involved
✤ Equality of access to information (and
thus power). Much of this data is
already available... for a price
✤ Efficiency – Endless amount of
duplication in all levels of government
because they can’t/won’t share data

25. Open data: why do we want it?
✤ Stupid question: It’s our data
✤ Engagement – reducing barriers to
getting involved
✤ Equality of access to information (and
thus power). Much of this data is
already available... for a price
✤ Efficiency – Endless amount of
duplication in all levels of government
because they can’t/won’t share data
✤ but most important of all...

26. Transparency
so that we see what’s going
on (and understand it too)

27. Transparency
so that we see what’s going
on (and understand it too)

28. Transparency
so that we see what’s going
on (and understand it too)

29. Transparency
so that we see what’s going
on (and understand it too)

30. Transparency
so that we see what’s going
on (and understand it too)

31. But wait a minute...
Lord Mandelson photo: http://www.flickr.com/photos/bisgovuk/4382508152/

32. But wait a minute...
Lord Mandelson photo: http://www.flickr.com/photos/bisgovuk/4382508152/
To some people and
companies these are all
reasons for not doing it.
From their perspective,
it’s all downside

33. But wait a minute...
Lord Mandelson photo: http://www.flickr.com/photos/bisgovuk/4382508152/
To some people and
companies these are all
reasons for not doing it.
From their perspective,
it’s all downside

34. But wait a minute...
Lord Mandelson photo: http://www.flickr.com/photos/bisgovuk/4382508152/
To some people and
companies these are all
reasons for not doing it.
From their perspective,
it’s all downside

35. But wait a minute...
Lord Mandelson photo: http://www.flickr.com/photos/bisgovuk/4382508152/
To some people and
companies these are all
reasons for not doing it.
From their perspective,
it’s all downside

36. Open data: why do we want it?
✤ Stupid question: It’s our data
✤ Engagement – reducing barriers to
getting involved
✤ Equality of access to information (and
thus power). Much of this data is
already available... for a price
✤ Efficiency – Endless amount of
duplication in all levels of government
because they can’t/won’t share data
✤ but most important of all...

37. don’t the
Open data: why do we want tit?ve it?
o ha
y want you
✤ Stupid question: It’s our data
✤ Engagement – reducing barriers to
getting involved
✤ Equality of access to information (and
thus power). Much of this data is
already available... for a price
✤ Efficiency – Endless amount of
duplication in all levels of government
because they can’t/won’t share data
✤ but most important of all...

38. don’t the
Open data: why do we want tit?ve it?
o ha
y want you
t; we collected it,
✤ No it’s no It’svourus power
Stupid question:t gi es data
it’s wha
✤ Engagement – reducing barriers to
getting involved
✤ Equality of access to information (and
thus power). Much of this data is
already available... for a price
✤ Efficiency – Endless amount of
duplication in all levels of government
because they can’t/won’t share data
✤ but most important of all...

39. don’t the
Open data: why do we want tit?ve it?
o ha
y want you
t; we collected it,
✤ No it’s no It’svourus power
Stupid question:t gi es data
it’s wha
✤ Engagement – duces power of to
Rereducing barriers olitical/
p
getting involvedpublic service elite
✤ Equality of access to information (and
thus power). Much of this data is
already available... for a price
✤ Efficiency – Endless amount of
duplication in all levels of government
because they can’t/won’t share data
✤ but most important of all...

40. don’t the
Open data: why do we want tit?ve it?
o ha
y want you
t; we collected it,
✤ No it’s no It’svourus power
Stupid question:t gi es data
it’s wha
✤ Engagement – duces power of to
Rereducing barriers olitical/
p
getting involvedpublic service elite
✤ Equality of access to information (and
thus power).goes the competitive
Bang Much of this data is
already available... antage
adv for a price
✤ Efficiency – Endless amount of
duplication in all levels of government
because they can’t/won’t share data
✤ but most important of all...

41. don’t the
Open data: why do we want tit?ve it?
o ha
y want you
t; we collected it,
✤ No it’s no It’svourus power
Stupid question:t gi es data
it’s wha
✤ Engagement – duces power of to
Rereducing barriers olitical/
p
getting involvedpublic service elite
✤ Equality of access to information (and
thus power).goes the competitive
Bang Much of this data is
already available... antage
adv for a price
✤ Efficiency – Endless amount of
Fewer people = difficult d
ecisions,
duplication in all levels of government
less power, less prestige
because they can’t/won’t share data
✤ but most important of all...

42. don’t the
Open data: why do we want tit?ve it?
o ha
y want you
t; we collected it,
✤ No it’s no It’svourus power
Stupid question:t gi es data
it’s wha
✤ Engagement – duces power of to
Rereducing barriers olitical/
p
getting involvedpublic service elite
✤ Equality of access to information (and
thus power).goes the competitive
Bang Much of this data is
already available... antage
adv for a price
✤ Efficiency – Endless amount of
Fewer people = difficult d
ecisions,
duplication in all levels of government
less power, less prestige
because they can’t/won’t share data
✤ but most important of all...
scary

43. Transparency
so that we see what’s going
on (and understand it too)

44. Transparency
an that jus ho what’s going
so d see wetseew truly screwed
on (and up things are. it too)
understand

45. The rewards of failure today

46. The rewards of failure today
✤ Big projects/policy initiatives bring media attention, power &
influence (and the key individuals will have moved on by the time
its failure is publicly realized)

47. The rewards of failure today
✤ Big projects/policy initiatives bring media attention, power &
influence (and the key individuals will have moved on by the time
its failure is publicly realized)
✤ Spread the risk by emphasizing process over results, scoping
everything out to the nth degree (you get very little benefit from
doing good in public service; lots of grief by visibly doing bad)

48. The rewards of failure today
✤ Big projects/policy initiatives bring media attention, power &
influence (and the key individuals will have moved on by the time
its failure is publicly realized)
✤ Spread the risk by emphasizing process over results, scoping
everything out to the nth degree (you get very little benefit from
doing good in public service; lots of grief by visibly doing bad)
✤ Don’t do small, quick, innovative projects (it’s too easy to see whether
they have succeeded... or failed, and difficult to spread the blame)

49. The rewards of failure today
✤ Big projects/policy initiatives bring media attention, power &
influence (and the key individuals will have moved on by the time
its failure is publicly realized)
✤ Spread the risk by emphasizing process over results, scoping
everything out to the nth degree (you get very little benefit from
doing good in public service; lots of grief by visibly doing bad)
✤ Don’t do small, quick, innovative projects (it’s too easy to see whether
they have succeeded... or failed, and difficult to spread the blame)
✤ Outsource everything (see ‘process over results’ above. Bonus:
‘commercial confidentiality’ means the facts can kept secret)

50. no better
The private sector’s
http://www.flickr.com/photos/pingnews/2935021856/

51. http://www.flickr.com/photos/pingnews/2935021856/
The private sector’s
no better
✤ Finance industry incentivized to take
risks, reap short-term rewards and let others bear the consequences

52. http://www.flickr.com/photos/pingnews/2935021856/
The private sector’s
no better
✤ Finance industry incentivized to take
risks, reap short-term rewards and let others bear the consequences
✤ Bigger, more opaque and more complex schemes give protection to
people & companies (until it all falls apart)

53. http://www.flickr.com/photos/pingnews/2935021856/
The private sector’s
no better
✤ Finance industry incentivized to take
risks, reap short-term rewards and let others bear the consequences
✤ Bigger, more opaque and more complex schemes give protection to
people & companies (until it all falls apart)
✤ Senior executives insulated from shareholders by diverse and
diffuse nature of the reporting/ownership system

54. http://www.flickr.com/photos/pingnews/2935021856/
The private sector’s
no better
✤ Finance industry incentivized to take
risks, reap short-term rewards and let others bear the consequences
✤ Bigger, more opaque and more complex schemes give protection to
people & companies (until it all falls apart)
✤ Senior executives insulated from shareholders by diverse and
diffuse nature of the reporting/ownership system
✤ People at the top are incentivized to make big changes (because
intelligent comparison becomes difficult); people at the bottom
incentivized to do as they’re told

55. The rewards of failure done right

56. The rewards of failure done right
✤ The open-source movement has been founded
on failure and fixing that failure, with
amazing results (and amazing stability)

57. The rewards of failure done right
✤ The open-source movement has been founded
on failure and fixing that failure, with
amazing results (and amazing stability)
✤ More than that, it’s failure in public.
Makes all bugs shallow; all faults visible

58. The rewards of failure done right
✤ The open-source movement has been founded
on failure and fixing that failure, with
amazing results (and amazing stability)
✤ More than that, it’s failure in public.
Makes all bugs shallow; all faults visible
✤ It also allows a meritocracy to develop,
incentivizing innovation and quality,
encouraging people to do their best work

59. The rewards of failure done right
✤ The open-source movement has been founded
on failure and fixing that failure, with
amazing results (and amazing stability)
✤ More than that, it’s failure in public.
Makes all bugs shallow; all faults visible
✤ It also allows a meritocracy to develop,
incentivizing innovation and quality,
encouraging people to do their best work
✤ The internet and the web are not only
accepting of failure, the inevitability of
failure is at the heart

60. How can Open Data help?

61. How can Open Data help?
✤ It helps change the incentives, from Fail Big, Fail Slow to
Fail Small, Fail Fast, and Fail Forward (i.e. building on the failure)

62. How can Open Data help?
✤ It helps change the incentives, from Fail Big, Fail Slow to
Fail Small, Fail Fast, and Fail Forward (i.e. building on the failure)
✤ It helps surface issues/problems making them less likely to be
repeated (learning from other people’s failures)

63. How can Open Data help?
✤ It helps change the incentives, from Fail Big, Fail Slow to
Fail Small, Fail Fast, and Fail Forward (i.e. building on the failure)
✤ It helps surface issues/problems making them less likely to be
repeated (learning from other people’s failures)
✤ It allows you to stand on the shoulders of giants. Don’t reinvent the
wheel – use or improve someone else’s

64. How can Open Data help?
✤ It helps change the incentives, from Fail Big, Fail Slow to
Fail Small, Fail Fast, and Fail Forward (i.e. building on the failure)
✤ It helps surface issues/problems making them less likely to be
repeated (learning from other people’s failures)
✤ It allows you to stand on the shoulders of giants. Don’t reinvent the
wheel – use or improve someone else’s
✤ It increases the quality of data – reuse by other people and combining
with other data identifies (and helps solve) quality issues far more
quickly and cheaply than internal checking

65. We need to recast the debate

66. We need to recast the debate
✤ Perhaps we need to talk about continuous improvement rather than
failure; evolution (which has failure built in) rather than revolution

67. We need to recast the debate
✤ Perhaps we need to talk about continuous improvement rather than
failure; evolution (which has failure built in) rather than revolution
✤ Be upfront about problems, knowledge gaps, issues

68. We need to recast the debate
✤ Perhaps we need to talk about continuous improvement rather than
failure; evolution (which has failure built in) rather than revolution
✤ Be upfront about problems, knowledge gaps, issues
✤ Less centralised complexity; more distributed innovation

69. We need to recast the debate
✤ Perhaps we need to talk about continuous improvement rather than
failure; evolution (which has failure built in) rather than revolution
✤ Be upfront about problems, knowledge gaps, issues
✤ Less centralised complexity; more distributed innovation
✤ Make public data open by default (and include outsourced work/
data), particularly anything to do with spending money

70. We need to recast the debate
✤ Perhaps we need to talk about continuous improvement rather than
failure; evolution (which has failure built in) rather than revolution
✤ Be upfront about problems, knowledge gaps, issues
✤ Less centralised complexity; more distributed innovation
✤ Make public data open by default (and include outsourced work/
data), particularly anything to do with spending money
✤ Outsource commodity tasks, not skills and knowledge (and avoid the
used-car buyer information trap). Stop the vendoritis

71. We need to recast the debate
✤ Perhaps we need to talk about continuous improvement rather than
failure; evolution (which has failure built in) rather than revolution
✤ Be upfront about problems, knowledge gaps, issues
✤ Less centralised complexity; more distributed innovation
✤ Make public data open by default (and include outsourced work/
data), particularly anything to do with spending money
✤ Outsource commodity tasks, not skills and knowledge (and avoid the
used-car buyer information trap). Stop the vendoritis
✤ Reward small-scale innovation and spending less money (not big
projects and spending more money)

72. In short, less this...

73. ...and more this
5 years of Linux kernel development in Git