Public Data. data.gov.no?

Hidden data treasures. How we can make more government data available for re-use without sacrificing the privacy of citizens? How to contact me: Harald Groven Web developer (actually database guy) [email_address] twitter.com/kongharald

CC Suzannelong

Practical and historic reasons why the most interesting government data sources are only published as useless not too useful aggregate numbers.
New web services may be built by giving access to source data.
One example of newly developed service from utdanning.no
Constraint: Avoid sacrificing privacy. (+ cost & incompetence)
Legal, political and bureaucratic barriers to sharing source data.

Public data: 150 years of no development? This is how public data was presented more than a century ago: Source: Eilert Sundt Om dødeligheden i Norge 1855

How the kind of data set is being presented 150 years later... Any progress? (except change of font?)

Government data published after 175 years! Census of 1801

Mashup of household register of 1865, 1886 and 2005 red dots = 2005 map = Friis 1861 New technology makes it possible to do visualizations and analyses not imaginable when data was collected

Recommended reading: The cultural & democratic impact of statistics on the public sphere Sarah Igo The Averaged American: Surveys, Citizens, and the Making of a Mass Public. Harvard UP 2007 Usefulness / uselessness of aggregates ? In the rest of the talk, I will argue that aggregates are mostly useless, but in fact they are not... good starting point

Usefulness of disaggregation Key concepts of data warehousing Rollup = Aggregate up one level Drill down = Disaggregate down one level Slice = Change variable Visual example thanks Obama & Vivek! Public spending in the US Text example thanks to NSD, Bergen % of students not passing exam 2005-09

Practical reason for not publishing disaggregated source data in pre-computer age: Space and cost ! Image: CC Harald Groven

Finding the needle in haystack of unaggregated data: Data warehouse. Data cube, visualized in 3D

Example: 1. Career opportunities according to category of highest education. 2. What kind of education do workers within an occupation category usually have?
Data sources:
The Norwegian Labour Force Survey (large sample)
Norwegian employees register, (all employed Norwegians)

Standards
Standard classification of occupations (STYRK98)
Norwegian Standard Classification of Education (NUS2000)
Service was developed in 2009
Will be publicly launched on utdanning.no in ?? 2010
Example

Raw data from Statistics Norway

Act relating to official statistics and Statistics Norway
§ 2-5. The use of information (1) Information collected in accordance with any prescribed obligation to provide information, or which is given voluntarily, may only be used for the production of official statistics ... If information is handed over, the obligation of secrecy pursuant ... shall also apply to the recipient of the information. When particular grounds so indicate, the Data Inspectorate may nevertheless make exceptions to such obligation of secrecy for certain types of information. § 2-6. The publication of information Information collected in accordance with any prescribed obligation to provide information, or which is given voluntarily, shall under no circumstances be published in such a way that it may be traced back to the supplier of any data or to any other identifiable individual to the detriment of the person concerned , or to the unreasonable detriment of the latter if the supplier of the data or the individual is an undertaking of the kind mentioned in § 5-1 third paragraph or a public organisation. § 2-7. Cessation of the obligation of secrecy The obligation of secrecy ... shall cease after 100 years .
Source: Act No. 54 of June 16 1989

What kind of statistical data are published? High level aggregates: Accessible Medium level aggregates (e.g. municipality level): Sometimes Low level aggregates, untraceable to identifiable persons: " grey zone ", accessible but largely unknown Anonymized raw data: Inaccessible for 100 years, some cases available for research! Raw data, unanonymized: Inaccessible for 100 years!

Statistics Norway Nov 6th 2008

[What you are requesting] Is not statistics, so won't be published .
Publishing data with ID field is problematic, and we will suggest you to collect this information from each organization .
"Dette er ikke statistikk, og er derfor ikke aktuelt for SSB å publisere. [...]
Det er problematisk for oss å levere ut data med ID, jeg vil derfor foreslå at dere henter inn disse opplysningene fra den enkelte undervisningsinstitusjon."
Rejection letter #1

From: Norwegian Social Science Data Services (NSD) May 13th 2009
"Your request for data to be published at our your website has been discussed by our board of directors. Our board expressed their concern regarding an data exchange partly because it may lead to ambiguity regarding the origin and context of the data , which is our organization's web portal."

"Forespørselen om utlevering av DBH-data til bruk for Utdanning.no sine sider er nå diskutert av Styringsgruppen for DBH. Styringsgruppen uttrykte skepsis til en slik datautveksling blant annet fordi det kan skape uklarheter omkring dataenes opprinnelse og kontekst , som er DBH-portalen. Styringsgruppen kom med en del innspill på hvordan dataene kan nå bedre ut til deres målgruppe, studenter og potensielle studenter. Blant annet bør det være en link til DBH på utdanning.no sine sider, og her kan være en tekst med informasjon om interessante data med for eksempel link til DBHs rapporter om karakterer eller andre data. Ellers er dette en sak som også er opp til KD å avgjøre. Når det gjelder NUS-koder så har vi dem komplett i vår database, men det er SSB sitt kodeverk. Derfor henviser vi til SSB for utlevering eller avtale om utlevering av disse dataene "
Rejection letter #2

Project: Display wage statistics on 500 of our web pages

Rejection letter #3
Statistics Norway Aug 20 2009

"Statistics Norway won't do it. Practically it would mean creating brand new wage statistics, and we do not have capacity or time to do it within reasonable constraints "

"SSB har ikke mulighet til å ta på seg dette oppdraget. I praksis vil det gå ut på å etablere en helt ny lønnsstatistikk, og det har vi verken kapasitet eller muligheter til å gjennomfør innen rimelighetens rammer . Det er i utgangspunktet to hovedårsaker til at SSB presenterer lønnsstatistikk på den måten vi gjør per i dag, med relativt grove inndelinger etter næring og yrke: 1. Statistikken er utvalgsbasert . Det trekkes årlige utvalg av foretak for hver næring i privat sektor for å minske oppgavebelastningen på næringslivet. Mao er det ikke registerinformasjon vi sitter på for privat sektor. I offentlig sektor er det fulltelling. Mer om utvalg i lønnsstatistikken, se http://www.ssb.no/emner/06/05/notat_200374/notat_200374.pdf. Dette legger klare begrensninger på hvor detaljert statistikk vi kan presentere. 2. Det vil bestandig være en viss usikkerhet knyttet til informasjon innhentet fra næringslivet, også yrkesinformasjon. Av den grunn må SSB være forsiktige med å publisere for detaljert statistikk. "

Creating a data.gov.no ? Require government agencies to publish anonymised data - Statistician's method: 1%, 10% samples? - CS method: Randomize variables so that the data set have the same statistical aggregates - Easy method: Publish data sets or each value, with a threshold value (e.g. 3 persons) to avoid tracing ID. - Use categories, not uncoded values.

CC Macwagon Government resources for making use of public data?

Public Data. data.gov.no?

by Harald Groven

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 27

Statistics