Linked Census Data

Linked Census Data
Rinke Hoekstra
CEDAR Kickoﬀ, 26 January 2012

donderdag 26 januari 12

Overview
“Can Linked Data make a difference for historical analysis?”

Problem

Procedure (as I understand it)

Step-by-step

Vocabularies, tools

Conclusion


Problem
~519 Excel spreadsheets (more?... I heard 1200)

Want to do analysis over time and space, but...

Structure

Excel sheets cannot be readily imported in a database

Contents

Excel sheets are not normalised (age) nor harmonised (occupations/places)

Excel sheets contain errors (both original and data-entry)

Want to preserve all stages of data cleansing/harmonisation


Procedure
Verbatim import of sheets to
Archiving database/triple store

Correcting/ Add missing information (headers)

Documenting
Interpreting Add corrected information (data)

Normalising Interpret and correct objective information

Link information across sheets
Harmonising Link information to other datasets (e.g. locations)

Visualising Build (generic) visualisations of results


... a bit about Linked Data

“Just another Data Model”
RDF ≠ Ontology (OWL)
RDF ≠ Taxonomy (RDFS/SKOS)

Globally Unique Identiﬁers (URI) for all entities

Dereferencable on the Web (URI = URL)

HTTP-accessible databases (triple stores, SPARQL)

Triples all the way <subject,
predicate,
object>


Spreadsheet ≠ Database

Primary Keys are entities

Column names are attributes

Cell values are attribute values

Secondary keys are relations to
other entities


Spreadsheet ≠ Database
No Primary Keys!

Anything can be an entity

Column headers are “types”

Row headers are “types”

Hierarchies!

Cell values are entity “values”

No relations to other entities


Anatomy of a Spreadsheet

Workbook

Cell Cell Cell

Sheet Cell Cell Cell

Cell Cell Cell

Cell Cell Cell

Sheet Cell Cell Cell

Cell Cell Cell



Workbook1.xls

Sheet1:A1 Sheet1:B1 Sheet1:C1

Sheet1 Sheet1:A2 Sheet1:B2 Sheet1:C2

... ... ...

Sheet2:A1 Sheet2:B1 Sheet2:C1

Sheet2 Sheet2:A2 Sheet2:B2 Sheet2:C2

... ... ...



Workbook1.xls

workers agriculture 12

Sheet1 industry 6

... ...

diamond
A 34
cutters

Sheet2 B 67

... ... ...



Workbook1.xls

workers agriculture 12

Sheet1 industry 6

... ...

diamond
A 34
cutters

Sheet2 B 67

... ... ...

NB: all URIs scoped to sheet!


Data Cube

How to best represent numeric data, in a ﬂexible way?

SDMX (Eurostat, World Bank, CBS, etc.)

Every data item is an observation

Every observation has a value

Every observation has one or more dimensions


Data Cube

How to best represent numeric data, in a ﬂexible way? 12

1878

SDMX (Eurostat, World Bank, CBS, etc.) M

O
I
leeftijd

nummer der beroepsklasse geboortejaar

Every data item is an observation geslacht

huwelijkse staat

E pannenbakkers

Every observation has a value beroep

positie
D 1

Every observation has one or more dimensions letter der beroepsklasse


Data Cube

How to best represent numeric data, in a ﬂexible way? 12

1878

SDMX (Eurostat, World Bank, CBS, etc.) M

O
I
leeftijd ?
nummer der beroepsklasse ?
geboortejaar

Every data item is an observation ?
geslacht
?
huwelijkse staat

E pannenbakkers

Every observation has a value beroep

positie
D 1

Every observation has one or more dimensions letter der beroepsklasse



Properties Headers

RowHeaders Data



Properties Headers

RowHeaders Data

http://github.com/Data2Semantics/TabLinker

:I
"1"^^xsd:int

skos:broader :Nummer_der_beroepsklasse d2s:populationSize

:I/E :Letter__Onderdeel_beroepsklasse_ _:x d2s:dimension :14--15_1875--1874

d2s:dimension
skos:broader
:M

:BENAMING_van_de_onderdeelen_der_onderscheidene_beroepsklassen__met_de_daartoe_behoorende_beroepen d2s:dimension

:O
:Positie_in_het_beroep__aangeduid_met_A__B__C_of_D
Sheet1:I/E/Fabricage_van_dakpannen__pannenbakkers

:D

Sheet1:D15


d2s:HierarchicalRowHeader d2s:DataCell d2s:Header

rdf:type rdf:type rdf:type
rdf:type rdf:type
rdf:type rdf:type

Sheet1:E15 Sheet1:C14 Sheet1:B8 Sheet1:L15 Sheet1:L3 Sheet1:L4 Sheet1:L5

d2s:isDimension

:I
d2s:isDimension "1"^^xsd:int
d2s:isObservation d2s:isDimension

skos:broader :Nummer_der_beroepsklasse d2s:isDimension
d2s:populationSize

d2s:isDimension :I/E :Letter__Onderdeel_beroepsklasse_ _:x d2s:dimension :14--15_1875--1874 d2s:isDimension

d2s:dimension
skos:broader
:M

:BENAMING_van_de_onderdeelen_der_onderscheidene_beroepsklassen__met_de_daartoe_behoorende_beroepen d2s:dimension
:Regelnummer
:O
:Positie_in_het_beroep__aangeduid_met_A__B__C_of_D d2s:dimension
Sheet1:I/E/Fabricage_van_dakpannen__pannenbakkers

:D :5 :10

d2s:isDimension d2s:isDimension d2s:isDimension

Sheet1:F15 Sheet1:D15 Sheet1:L6

rdf:type rdf:type rdf:type

d2s:RowHeader d2s:Metadata


What TabLinker can’t do
Annotations
“footnote”-style on separate sheet

Interpret functions
e.g. automatic sums

Integrate/harmonise across sheets/ﬁles

Additional useful functionality:

“checksum” functionality

Export to database tables


Normalising & Correcting

"1"^^xsd:int

d2s:populationSize

_:x

d2s:dimension

:14--15_1875--1874


Normalising & Correcting

"1"^^xsd:int "1"^^xsd:int "11"^^xsd:int

d2s:populationSize d2s:populationSize d2s:populationSize
"1889"^^xsd:int
d2s:censusYear
_:x _:x
d2s:birthYears
:1875--1874
d2s:gemeente

d2s:dimension d2s:dimension d2s:ageGroup

:Assendelft

:14--15_1875--1874 :14--15_1875--1874 :14-15


Documenting

<http://example.com/workbook1/sheet1> <http://example.com/workbook1/sheet1/corrected> provo:Activity
rdf:type
:curation20120126
"1"^^xsd:int "11"^^xsd:int
provo:wasGeneratedBy provo:hadAgent

provo:startedAt
d2s:populationSize d2s:populationSize provo:endedAt
"1889"^^xsd:int :RinkeHoekstra
d2s:censusYear
_:x
d2s:birthYears
:1875--1874 _:b _:a
d2s:gemeente
d2s:dimension d2s:ageGroup
time:inXSDDateTime time:inXSDDateTime
:Assendelft

:14--15_1875--1874 :14-15
"20120126T09:00:00" "20120126T08:30:00"

http://www.w3.org/TR/prov-o/


Harmonising

I

skos:broader
skos:broader
skos:broader

D E A

skos:broader skos:broader skos:broader
skos:broader

Fabricage van
Fabricage van steen aardewerk (incl.
Fabricage van Fabricage van dakpannen
(molensteen, steenbakkers, porcelein, terracotta,
kalk (pannenbakkers)
tegelbakkers) kachelbakkers,
pottenbakkers, enz.)


Harmonising
I

skos:broader
skos:broader
skos:broader

D E A

skos:broader

Fabricage van

skos:exactMatch skos:broadMatch skos:broadMatch skos:closeMatch
skos:exactMatch skos:exactMatch
skos:exactMatch

HISCO:23811 HISCO:25281 HISCO:25281 HISCO:26345

HISCO:23810 HISCO:25281 HISCO:26340


Harmonising
I

skos:broader
skos:broader
skos:broader

D E A

skos:broader

Fabricage van

Sheet1:I

skos:broader skos:broader
skos:broader

Sheet1:D Sheet1:E Sheet1:A

skos:broader

Sheet1:Fabricage van
Sheet1:Fabricage van steen Sheet1:Fabricage van aardewerk (incl.
Sheet1:Fabricage
(molensteen, steenbakkers, dakpannen porcelein, terracotta,
van kalk
tegelbakkers) (pannenbakkers) kachelbakkers,


I

skos:broader
skos:broader
skos:broader

D E A

1889 skos:broader

Fabricage van

skos:narrowMatch I skos:closeMatch

skos:exactMatch
skos:narrowMatch
skos:broader
skos:broader
skos:broader

D E A

skos:broader skos:broader skos:broader 1899
skos:broader

Fabricage van
(steenbakkers, porcelein,


I
Is SKOS sufﬁcient?
skos:broader
skos:broader
skos:broader

D E A

1889 skos:broader

Fabricage van

skos:narrowMatch I skos:closeMatch

skos:exactMatch
skos:narrowMatch
skos:broader
skos:broader
skos:broader

D E A

skos:broader skos:broader skos:broader 1899
skos:broader

Fabricage van
(steenbakkers, porcelein,

NB: These are not strings, but globally unique URIs, scoped within their spreadsheet (graph!) of origin.


Vocabularies, Tools
Vocabularies
Data Cube, SKOS, W3C Time, PROV-O

Excel + TabLinker
Semi-automatic conversion of Excel sheets to RDF

ProvTracer
Create PROV-O provenance trail for shell/python scripts

Visualization Prototype
SGVizler (SPARQL + Google Graph API)


Discussion
Advantages of Linked Data approach

Straightforward transformation from spreadsheets

Seamless integration of original, corrected and harmonised data

Ingestion of external (linked) data

Powerful documentation (provenance)

Everything is transparently query-able (SPARQL)

.... on the Web


Discussion

Disadvantages of Linked Data approach (subject to research)

Size? (300k * 519 sheets = 156M triples)

Only rudimentary support for arithmetical operations in queries

No dynamic/conditional ‘view’-like graphs


SPARQL vs. SQL?

Middle ground?

Expose database through D2RQ


Fin


Linked Census Data

Recommended

Recommended

More Related Content

More from Rinke Hoekstra

More from Rinke Hoekstra (20)

Recently uploaded

Recently uploaded (20)

Linked Census Data