Open Source, Open Data
Kirrily Robert
Florida Linux Show, 2009
From Open Source
   to Open Data
1993




Me in 1993     My Linux desktop
                looked like this
1993

•   I started using Linux in 1993
•   I was very excited by it, even
    though it was quite primitive at the
    ti...
1999




      Image: Wikipedia                     Image: Engadget

Jar Jar in 1999           Google’s servers in 1999
1999

•   By 1999 Linux + open source was
    starting to take off
•   Companies using and building
    services on Linux ...
Four Software Freedoms
    http://www.gnu.org/philosophy/free-sw.html


•   Freedom to run the program
•   Freedom to stud...
Free Culture


•   A similar movement
•   Make cultural works freely
    available
•   Mostly over the Internet
Free Culture
Free Culture
Free Culture
Free Culture
http://wiki.freeculture.org/Free_Culture_Definition

•   Freedom to use the work
•   Freedom to study the work...
Image: masternewmedia.org
What is Open Data?
Data
Image: himmelskratzer @ Flickr
What is data?

•   Ones and zeroes (obviously)
•   But also filing cabinets, research
    archives, and other offline
    re...
Open Data Freedoms
•   Freedom to use the data
•   Freedom to study the data and
    modify it for your own use
•   Freedo...
Data availability



•   Digital
•   Online
•   Well formatted
Open Data Projects
public.resource.org


       •   Created 2007
           by Carl Malamud
       •   “Making Government
           Informat...
public.resource.org

•   SEC EDGAR records
•   Patents database
•   Copyright database
•   Congressional records
•   Legal...
Data.gov
  •   Founded 2008
  •   “Increase public
      access to high value,
      machine readable
      datasets gener...
OpenStreetMap
Compare...
OpenStreetMap
Open Library Project
•   CD data
•   Tracks, artists, releases...
•   CC license
Flickr
•   Images
•   Metadata
    •   tags, timestamps, geolocations, etc.

•   Range of CC licenses and
    permissive T...
Infochimps

•   Large data sets
•   Various licenses
•   Tools for transformation
•   Open data about “everything”
•   8.5m concepts
•   CC-BY license
•   API and data dumps
2,416,683 books
16,608 ships
488 cheeses
Structured data
{
    "name": "Asiago cheese"
    "id": "/en/asiago_cheese",
    "region": [{
      "id":    "/en/asiago",...
Open Data Apps
•   Apps for America competition
•   Open source and open data
•   Round 1: various data sources
•   Round 2: Data.gov
Legistalker
Filibusted
Where the money goes
Open Source for Open Data
What can open source do?
    Input      Scrape




  Processing   Munge




   Output      Visualise
Scraping data

•   APIs
    •   XML, RSS, JSON...

•   Downloadable data sets
    •   XML, Excel, CSV, triple dumps...

• ...
Munging data

•   Perl
    •   http://perl.org/

•   R (statistical analysis)
    •   http://r-project.org/

•   Hadoop (p...
Visualisations


•   MIT Simile
    •   http://simile.mit.edu/

•   Processing
    •   http://processing.org/
http://itoworld.com
Semantic Web



•   Describe meaning, not markup
•   Triples: subject, predicate, object
•   Expression: RDF
Linked Open Data
Semantic web tools


•   Triple stores
    •   Sesame, BigData, Virtuoso...

•   Libraries
    •   RDFLib (Python), Redlan...
Freebase Acre
Open source for open data


•   Low barrier to entry
•   Hooks in to Freebase data
•   Share and clone apps
•   Apps are B...
FMDB
Gendered names app
Query editor
Clone!
http://freebase.com/developer
Where next?
Open Data: Issues


•   License clarity
•   Govt + Corporate acceptance
•   Developer literacy
•   What do we DO with it?
What do we do with it?
What do we do with it?


•   10 years ago we were asking the
    same questions of Open Source
•   With Open Data, we are ...
Keep in touch

•   Email
    •   kirrily@metaweb.com

•   Freebase blog
    •   http://blog.freebase.com/

•   Twitter
   ...
Open Source, Open Data
Open Source, Open Data
Open Source, Open Data
Open Source, Open Data
Open Source, Open Data
Upcoming SlideShare
Loading in...5
×

Open Source, Open Data

7,217

Published on

My presentation from Florida Linux Show 2009. Find out how open source's principles are being used outside of software, and how open source and open data can work together to change the world.

Published in: Technology, News & Politics
1 Comment
27 Likes
Statistics
Notes
  • great slide
    ---
    http://shine.yahoo.com/channel/none/muscle-relaxers-soma-carisoprodol-pain-medication-1231574/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
7,217
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
411
Comments
1
Likes
27
Embeds 0
No embeds

No notes for slide

Open Source, Open Data

  1. 1. Open Source, Open Data Kirrily Robert Florida Linux Show, 2009
  2. 2. From Open Source to Open Data
  3. 3. 1993 Me in 1993 My Linux desktop looked like this
  4. 4. 1993 • I started using Linux in 1993 • I was very excited by it, even though it was quite primitive at the time • Other people thought I was a little crazy
  5. 5. 1999 Image: Wikipedia Image: Engadget Jar Jar in 1999 Google’s servers in 1999
  6. 6. 1999 • By 1999 Linux + open source was starting to take off • Companies using and building services on Linux etc. • We were calling it “Open Source” - a more marketable term for Free Software
  7. 7. Four Software Freedoms http://www.gnu.org/philosophy/free-sw.html • Freedom to run the program • Freedom to study the program and modify it for your own use • Freedom to redistribute verbatim copies • Freedom to improve the program, and release your improvements
  8. 8. Free Culture • A similar movement • Make cultural works freely available • Mostly over the Internet
  9. 9. Free Culture
  10. 10. Free Culture
  11. 11. Free Culture
  12. 12. Free Culture http://wiki.freeculture.org/Free_Culture_Definition • Freedom to use the work • Freedom to study the work and to apply knowledge acquired from it • Freedom to make and redistribute copies • Freedom to make changes and improvements, and to distribute derivative works
  13. 13. Image: masternewmedia.org
  14. 14. What is Open Data?
  15. 15. Data
  16. 16. Image: himmelskratzer @ Flickr
  17. 17. What is data? • Ones and zeroes (obviously) • But also filing cabinets, research archives, and other offline resources • It’s not OPEN data unless you can get at it
  18. 18. Open Data Freedoms • Freedom to use the data • Freedom to study the data and modify it for your own use • Freedom to make and share verbatim copies • Freedom to improve the data and redistribute the results
  19. 19. Data availability • Digital • Online • Well formatted
  20. 20. Open Data Projects
  21. 21. public.resource.org • Created 2007 by Carl Malamud • “Making Government Information More Accessible”
  22. 22. public.resource.org • SEC EDGAR records • Patents database • Copyright database • Congressional records • Legal decisions • Fedflix
  23. 23. Data.gov • Founded 2008 • “Increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government.”
  24. 24. OpenStreetMap
  25. 25. Compare...
  26. 26. OpenStreetMap
  27. 27. Open Library Project
  28. 28. • CD data • Tracks, artists, releases... • CC license
  29. 29. Flickr • Images • Metadata • tags, timestamps, geolocations, etc. • Range of CC licenses and permissive TOS
  30. 30. Infochimps • Large data sets • Various licenses • Tools for transformation
  31. 31. • Open data about “everything” • 8.5m concepts • CC-BY license • API and data dumps
  32. 32. 2,416,683 books
  33. 33. 16,608 ships
  34. 34. 488 cheeses
  35. 35. Structured data { "name": "Asiago cheese" "id": "/en/asiago_cheese", "region": [{ "id": "/en/asiago", "name": "Asiago", "type" : "/location/location" }], "source_of_milk": [{ "id": "/en/cattle", "name": "Cow", "type" : "/biology/organism_classification" }] }
  36. 36. Open Data Apps
  37. 37. • Apps for America competition • Open source and open data • Round 1: various data sources • Round 2: Data.gov
  38. 38. Legistalker
  39. 39. Filibusted
  40. 40. Where the money goes
  41. 41. Open Source for Open Data
  42. 42. What can open source do? Input Scrape Processing Munge Output Visualise
  43. 43. Scraping data • APIs • XML, RSS, JSON... • Downloadable data sets • XML, Excel, CSV, triple dumps... • Beautiful Soup (Python) • http://www.crummy.com/software/
  44. 44. Munging data • Perl • http://perl.org/ • R (statistical analysis) • http://r-project.org/ • Hadoop (parallel data processing) • http://hadoop.apache.org/
  45. 45. Visualisations • MIT Simile • http://simile.mit.edu/ • Processing • http://processing.org/
  46. 46. http://itoworld.com
  47. 47. Semantic Web • Describe meaning, not markup • Triples: subject, predicate, object • Expression: RDF
  48. 48. Linked Open Data
  49. 49. Semantic web tools • Triple stores • Sesame, BigData, Virtuoso... • Libraries • RDFLib (Python), Redland RDf (librdf)...
  50. 50. Freebase Acre
  51. 51. Open source for open data • Low barrier to entry • Hooks in to Freebase data • Share and clone apps • Apps are BSD licensed
  52. 52. FMDB
  53. 53. Gendered names app
  54. 54. Query editor
  55. 55. Clone!
  56. 56. http://freebase.com/developer
  57. 57. Where next?
  58. 58. Open Data: Issues • License clarity • Govt + Corporate acceptance • Developer literacy • What do we DO with it?
  59. 59. What do we do with it?
  60. 60. What do we do with it? • 10 years ago we were asking the same questions of Open Source • With Open Data, we are just starting to realise its potential • Please join us!
  61. 61. Keep in touch • Email • kirrily@metaweb.com • Freebase blog • http://blog.freebase.com/ • Twitter • @fbase
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×