Open Source, Open Data

Open Source, Open Data Kirrily Robert Florida Linux Show, 2009

From Open Source to Open Data

1993 Me in 1993 My Linux desktop looked like this

1993 • I started using Linux in 1993 • I was very excited by it, even though it was quite primitive at the time • Other people thought I was a little crazy

1999 Image: Wikipedia Image: Engadget Jar Jar in 1999 Google’s servers in 1999

1999 • By 1999 Linux + open source was starting to take off • Companies using and building services on Linux etc. • We were calling it “Open Source” - a more marketable term for Free Software

Four Software Freedoms http://www.gnu.org/philosophy/free-sw.html • Freedom to run the program • Freedom to study the program and modify it for your own use • Freedom to redistribute verbatim copies • Freedom to improve the program, and release your improvements

Free Culture • A similar movement • Make cultural works freely available • Mostly over the Internet

Free Culture

Free Culture http://wiki.freeculture.org/Free_Culture_Deﬁnition • Freedom to use the work • Freedom to study the work and to apply knowledge acquired from it • Freedom to make and redistribute copies • Freedom to make changes and improvements, and to distribute derivative works

Image: masternewmedia.org

What is Open Data?

Image: himmelskratzer @ Flickr

What is data? • Ones and zeroes (obviously) • But also ﬁling cabinets, research archives, and other ofﬂine resources • It’s not OPEN data unless you can get at it

Open Data Freedoms • Freedom to use the data • Freedom to study the data and modify it for your own use • Freedom to make and share verbatim copies • Freedom to improve the data and redistribute the results

Data availability • Digital • Online • Well formatted

Open Data Projects

public.resource.org • Created 2007 by Carl Malamud • “Making Government Information More Accessible”

public.resource.org • SEC EDGAR records • Patents database • Copyright database • Congressional records • Legal decisions • Fedﬂix

Data.gov • Founded 2008 • “Increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government.”

OpenStreetMap

Compare...

OpenStreetMap

Open Library Project

• CD data • Tracks, artists, releases... • CC license

Flickr • Images • Metadata • tags, timestamps, geolocations, etc. • Range of CC licenses and permissive TOS

Infochimps • Large data sets • Various licenses • Tools for transformation

• Open data about “everything” • 8.5m concepts • CC-BY license • API and data dumps

2,416,683 books

16,608 ships

488 cheeses

Structured data { "name": "Asiago cheese" "id": "/en/asiago_cheese", "region": [{ "id": "/en/asiago", "name": "Asiago", "type" : "/location/location" }], "source_of_milk": [{ "id": "/en/cattle", "name": "Cow", "type" : "/biology/organism_classification" }] }

Open Data Apps

• Apps for America competition • Open source and open data • Round 1: various data sources • Round 2: Data.gov

Legistalker

Filibusted

Where the money goes

Open Source for Open Data

What can open source do? Input Scrape Processing Munge Output Visualise

Scraping data • APIs • XML, RSS, JSON... • Downloadable data sets • XML, Excel, CSV, triple dumps... • Beautiful Soup (Python) • http://www.crummy.com/software/

Munging data • Perl • http://perl.org/ • R (statistical analysis) • http://r-project.org/ • Hadoop (parallel data processing) • http://hadoop.apache.org/

Visualisations • MIT Simile • http://simile.mit.edu/ • Processing • http://processing.org/

http://itoworld.com

Semantic Web • Describe meaning, not markup • Triples: subject, predicate, object • Expression: RDF

Linked Open Data

Semantic web tools • Triple stores • Sesame, BigData, Virtuoso... • Libraries • RDFLib (Python), Redland RDf (librdf)...

Freebase Acre

Open source for open data • Low barrier to entry • Hooks in to Freebase data • Share and clone apps • Apps are BSD licensed

Gendered names app

Query editor

Clone!

http://freebase.com/developer

Where next?

Open Data: Issues • License clarity • Govt + Corporate acceptance • Developer literacy • What do we DO with it?

What do we do with it?

What do we do with it? • 10 years ago we were asking the same questions of Open Source • With Open Data, we are just starting to realise its potential • Please join us!

Keep in touch • Email • kirrily@metaweb.com • Freebase blog • http://blog.freebase.com/ • Twitter • @fbase

http://infotrope.net	114
http://geothink.tistory.com	64
http://www.slideshare.net	54
http://xrefer.blogspot.com	18
http://geothink.textcube.com	7
http://www.neogeography.kr	3
http://embedded.dreamwidth.net	1
http://twitter.com	1
http://static.slidesharecdn.com	1
http://www.owf.vm	1

Open Source, Open Data

by Kirrily Robert on Oct 24, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

10 Embeds 264

Statistics

Open Source, Open Data — Presentation Transcript