Open Source, Open Data
Kirrily Robert
Florida Linux Show, 2009

From Open Source
to Open Data

1993
Me in 1993 My Linux desktop
looked like this

1993
• I started using Linux in 1993
• I was very excited by it, even
though it was quite primitive at the
time
• Other people thought I was a little
crazy

1999
Image: Wikipedia Image: Engadget
Jar Jar in 1999 Google’s servers in 1999

1999
• By 1999 Linux + open source was
starting to take off
• Companies using and building
services on Linux etc.
• We were calling it “Open Source” -
a more marketable term for Free
Software

Four Software Freedoms
http://www.gnu.org/philosophy/free-sw.html
• Freedom to run the program
• Freedom to study the program and
modify it for your own use
• Freedom to redistribute verbatim
copies
• Freedom to improve the program,
and release your improvements

Free Culture
• A similar movement
• Make cultural works freely
available
• Mostly over the Internet

Free Culture

Free Culture

Free Culture

Free Culture
http://wiki.freeculture.org/Free_Culture_Definition
• Freedom to use the work
• Freedom to study the work and to
apply knowledge acquired from it
• Freedom to make and redistribute
copies
• Freedom to make changes and
improvements, and to distribute
derivative works

Image: masternewmedia.org

What is Open Data?

Data

Image: himmelskratzer @ Flickr

What is data?
• Ones and zeroes (obviously)
• But also filing cabinets, research
archives, and other offline
resources
• It’s not OPEN data unless you can
get at it

Open Data Freedoms
• Freedom to use the data
• Freedom to study the data and
modify it for your own use
• Freedom to make and share
verbatim copies
• Freedom to improve the data and
redistribute the results

Data availability
• Digital
• Online
• Well formatted

Open Data Projects

public.resource.org
• Created 2007
by Carl Malamud
• “Making Government
Information More
Accessible”

public.resource.org
• SEC EDGAR records
• Patents database
• Copyright database
• Congressional records
• Legal decisions
• Fedflix

Data.gov
• Founded 2008
• “Increase public
access to high value,
machine readable
datasets generated
by the Executive
Branch of the Federal
Government.”

OpenStreetMap

Compare...

OpenStreetMap

Open Library Project

• CD data
• Tracks, artists, releases...
• CC license

Flickr
• Images
• Metadata
• tags, timestamps, geolocations, etc.
• Range of CC licenses and
permissive TOS

Infochimps
• Large data sets
• Various licenses
• Tools for transformation

• Open data about “everything”
• 8.5m concepts
• CC-BY license
• API and data dumps

2,416,683 books

16,608 ships

488 cheeses

Structured data
{
"name": "Asiago cheese"
"id": "/en/asiago_cheese",
"region": [{
"id": "/en/asiago",
"name": "Asiago",
"type" : "/location/location"
}],
"source_of_milk": [{
"id": "/en/cattle",
"name": "Cow",
"type" : "/biology/organism_classification"
}]
}

Open Data Apps

• Apps for America competition
• Open source and open data
• Round 1: various data sources
• Round 2: Data.gov

Legistalker

Filibusted

Where the money goes

Open Source for Open Data

What can open source do?
Input Scrape
Processing Munge
Output Visualise

Scraping data
• APIs
• XML, RSS, JSON...
• Downloadable data sets
• XML, Excel, CSV, triple dumps...
• Beautiful Soup (Python)
• http://www.crummy.com/software/

Munging data
• Perl
• http://perl.org/
• R (statistical analysis)
• http://r-project.org/
• Hadoop (parallel data processing)
• http://hadoop.apache.org/

Visualisations
• MIT Simile
• http://simile.mit.edu/
• Processing
• http://processing.org/

http://itoworld.com

Semantic Web
• Describe meaning, not markup
• Triples: subject, predicate, object
• Expression: RDF

Linked Open Data

Semantic web tools
• Triple stores
• Sesame, BigData, Virtuoso...
• Libraries
• RDFLib (Python), Redland RDf (librdf)...

Freebase Acre

Open source for open data
• Low barrier to entry
• Hooks in to Freebase data
• Share and clone apps
• Apps are BSD licensed

FMDB

Gendered names app

Query editor

Clone!

http://freebase.com/developer

Where next?

Open Data: Issues
• License clarity
• Govt + Corporate acceptance
• Developer literacy
• What do we DO with it?

What do we do with it?

What do we do with it?
• 10 years ago we were asking the
same questions of Open Source
• With Open Data, we are just
starting to realise its potential
• Please join us!

Keep in touch
• Email
• kirrily@metaweb.com
• Freebase blog
• http://blog.freebase.com/
• Twitter
• @fbase