The paper aims at exploring the consequences of the gradually increasing availability of Open Data for evaluation as we know it. Using concepts from the literature on evaluation and democracy, it contends that new technologies both require a new behavior by evaluators and open up possibilities in the very framework in which evaluation is done. The pressure to open up data changes the way governments and public sector offices conceptualize, produce, and disseminate data. Responding to this demand requires that internal procedures change in fundamental, still partially unexplored ways. Issues arise also for citizens seeking information. They face a rapid growth of internet-based sources, which both creates opportunities for research and difficulties in assessing data quality, credibility, and usability. It also implies that public interventions--be they programmes, projects, or services--are open to public scrutiny of a new, more informed type. It increasingly involves expert, non-expert, and differently-expert scrutiny. It is highly unlikely that Open Data will ever provide all--or even most--information needed for an evaluation. There is a risk that, in addition to opening up new research avenues and framing new evaluation questions by new actors, the availability of great masses of data on public policies obscures the need to directly observe effects and to build credible theories about phenomena. The very existence of open data, and the possibilities they open up to public scrutiny call into question the role of internal and external evaluators. This is even more so when thinking of the opportunities opened by the ability to conjure collective intelligence in evaluation processes--using concepts already developed in the participation tradition. The paper explores these themes based on an on-going research project. The two authors are involved in the Open Data movement in Italy and will advance their research during the next months through their work, research on existing literature, and holding workshops (e.g. within the Sapienza Seminar on Classic Evaluation Theorists). 10th EES Biennial Conference

1. Swarm Intelligence or Lost in the
Crowd. Open Data, Public Scrutiny of
Public Action, and How Evaluations
Change
Aline Pennisi e Laura Tagle
EES Annual Conference
Helsinki, October 2012

2. Outline
• State of the art

3. the increasing use of ICT in all domains
• Increasingly citizens/businesses are making their own
information on the internet and consuming information made
by others
• The internet is increasing the value of information created by
government
There is economic and social use of all this information…
…. and new opportunities to evaluate government policies

4. what open data is
“A piece of content or data is open if anyone is free to use, reuse, and
redistribute it —without restrictions from copyright, patents or other
mechanisms of control.”
• collected by governments while performing its tasks (open
government data)
• created and shared by users
• Restrictions: technological or legal features
Focus: how open data changes the way the public
sector relates to the external world and works
internally

5. how open data and ICT affect the way the
public sector works
• Dissemination of raw administrative data (while dissemination of
statistics is not a novelty ...) and accessibility to anyone
• Interaction with a huge number of external bodies (citizens,
businesses, other public authorities) on a much wider range of
possibilities, with fewer filters
• The connection/integration between your data and the
data produced by other parties ... public but also non-public
• Crowd-sourcing and collaborative data collection/validation
• The increased uncertainty about official/truth, given a less
marked border between "certified" information and not
• The possibility that others provide a service that was previously
the sole prerogative of the Government / public sector (in competition,
replacement or in addition)

6. Open government and e-government
• Overlapping yet not the same
• Different approach to use of ICT
• Both share similar limitations
– Digital divides (territorial, age, socio-economic)
– New vulnerabilities

7. E- government
• Stress on technology: use of ICT to provide services
• Allows, but not necessarily requires:
– new services
– collaboration
– openness
– Scrutiny
Frustratingly interacting with an automated system from
home not intrinsically better than walking to the counter
to frustratingly interact with a human

8. Open government
• Stress on activities
• Data as public good: push for publication of data changes
internal functioning of government
• Openness relates to
– Possible collaboration: accomodates or invites interactions with
other public and private entities
– New services: no fixed bundle of services, but invites and
accomodates creative use of existing public goods (among which
data) to provide new services
– Multi-centric, democratic governance
– Scrutiny of government action

9. Minnesota Tagged Fish
http://www.dnr.state.mn.us/fisheries/tagged_fish_reporting/index.htm
l
9

10. FixMYStreet
http://www.fixmystreet.com/
10

11. OpenCoesione
http://www.opencoesione.gov.it/
11

12. first claim on benefits of open data / ICT
• efficiency: with open data the role of Government/public
administration in informing the citizens of public affairs and in providing
utilities can be largely reduced (e.g., Robinson & Yu, 2010)
o other (private intermediaries, profit or not-) can do this better,
because they are on the "user’s side“ and having to compete every
day in the market they need to continuously innovate and do it better
o others can do this at lower cost, extrapolating data from more and
more numerous sources, and giving it more meaning (through
research, representation, processing, data update---beyond simple
data delivery)
But some questions are left unresolved:
who is responsible for ensuring these services and making sure they are fair/
for everyone?
are we really willing to pay "public" services supplied by third parties at a cost?
does “everyone” really mean everyone? How many people are digitally literate?

13. second claim on benefits of open data / ICT
• democracy: Open data allows you to re-invent the relationship
between rulers and ruled, between public service providers and citizens
— in favor of the latter (e.g., Tim O'Reilly, 2009; Maier-Rabler & Huber,
2011)
o Government/public administrations can offer an unbiased platform
where all actors interact (government as a platform ... from Donald
Kettle's "vending machine" to Eric Raymond's "Bazaar")
o everyone can monitor the actual conduct of the
Government/Administration
But some questions are left unresolved:
It is not clear whether we have enough or the right data to determine accountability
If there isn’t a first “authentic/explicit” interpretation of data will we actually know
which are the objectives of policies?
If there is no official info who will lead the citizen in choosing between alternatives?
Which groups are more active on the internet? How are vulnerable groups/territories
reaching out on the internet?

14. third claim on the benefits of open data / ICT
• effectiveness: Open data improves the services provided and
supports better policy decisions (e.g.,)
o It reduces the information asymmetry and citizens/businesses can
more easily report about the situation on the ground
o allows you to rebuild confidence between the parties and share
goals, and then to generate co-operative behaviors between policy
makers and policy beneficiaries
o dissemination, extreme detail and speed of information facilitates
coordination among the various policy actors
o it can provide valuable data for informing the choices of citizens and
economic operators
o You can create a stronger pressure towards results
however: is voice enough to identify policies "that work“? how to deal with moral
hazard? How to select among all the information available?

15. opportunities for evaluation
o increased availability of (often free!) data on the programs, projects or
services that you want to evaluate
o possibility to have many more reviews from many more evaluators
(since they all have the same chances to have access to crucial data)
o open government data is changing the way to collect and make
available data on social phenomena (e.g., urban decor, environment,
crime)
o and it pushes the observation of social phenomena towards more
detailed territorial level (using more administrative archives and fewer
samples/estimates)
MORE ?

16. challenges for evaluations
• Input data overwhelm the discourse already
• Government-produced information on programs, projects and services are not
necessarily sufficient for the assessment of effects ..
• Open government data can only concern secondary data (created or collected or
archived by public sector for its purposes)
• Difficulties in discerning sense, reliability, biases in primary data—not exactly new
issues, but with NEW technical features and at a (potentially) much larger scale
• Judgment criteria multiply exponentially
• Speaking truth to power has a new meaning: main client is not anymore public sector
but the public
• Independence becomes a must: reputation vis-à-vis the masses, not only the policy or
professional community
• Quality control not only Steering Groups or advisory groups but the collective
intelligence
Evaluation needs primary data

17. challenges for evaluators
• Skills
• Demand: will open data stimulate demand for evaluation? Or will the
availability of multiple analyses fulfill the need for evaluation—in a
time of shrinking resources?
• Status

18. expert, non-expert, and differently-
expert scrutiny
o how is the data going to be interpreted now that everyone is
legitimated to do it?
o Myriads of analyses and new “evaluators”
Who needs to pay an evaluator when there are thousands of self-
appointed analysts?
Where do we acquire the new skills?

19. New frontiers
• opportunities opened by the ability to conjure collective intelligence in
evaluation processes--using concepts already developed in the
participation tradition
• evaluators also now have new tools to gather information, which can
change the relationship with what is observed or evaluated
• Evaluation as expert knowledge, as an application of methods by
“evaluators”, questioned.
• Clearer what has always been there:
• evaluation is one among many practices of public scrutiny,
• New, different expertise come into the play-field
• People with no voice & power can have their say.

20. Which one are we going for?