Making the gov data more open

1,499 views

Published on

http://spring2011.drupalcamp.se/schedule/making-government-data-open-drupal-and-other-tools

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,499
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
13
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Making the gov data more open

  1. 1. may 2 0 1 1MAKING THE GOV DATA OPEN MAREK SOTAK | ATOMIC ANT  www.atomicant.co.uk
  2. 2. OH HAI!ABOUT ME & ATOMIC ANTMarek Sotak • Web designer, developer • From Prague, Czech Republic • Over 5 years with Drupal - since v4.6 • Rootcandy admin theme • Organising events - Drupal Design Camp, Local Meet-ups • @sotak on twitter • http://sotak.co.uk - personal blog/experiments 6 : 0 2 : 1atomicant.co.uk #justsaying ;)
  3. 3. OH HAI!ABOUT ME & ATOMIC ANT• Based in London & Prague• Human interface design, training, branding, development• Clients all over the world• http://atomicant.co.uk
  4. 4. OPEN DATA?HUH? What is OPEN DATA?atomicant.co.uk
  5. 5. OPEN DATA?HUH?Wikileaks Iraq war logs: every death mapped http://bit.ly/iraqwarlogsatomicant.co.uk
  6. 6. OPEN DATA?HUH?Dont eat at ____ http://donteat.atatomicant.co.uk
  7. 7. OPEN DATA?HUH?Dont eat at - http://donteat.at/atomicant.co.uk
  8. 8. DATA MINING - SCRAPINGLETS GET DIRTYBigClean.org – Pragueatomicant.co.uk
  9. 9. DATA MINING - SCRAPINGLETS GET DIRTYTheres a lot of data laying around on the internet that can beuseful → Crime reports, government reports, statistics,missing pets register, current affairsHowever sometimes they are in a format such as PDF, html,etc... something you cant really take and performcalculations, visualizations, filtering, etc... on.Is it really that hard to publish something in a CSV, XML,.. ?atomicant.co.uk
  10. 10. DATA MINING - SCRAPINGLETS GET DIRTYMinistry of the interior – Czech RepublicPublic Collections- open what?atomicant.co.uk
  11. 11. DATA MINING - SCRAPINGLETS GET DIRTYatomicant.co.uk
  12. 12. DATA MINING - SCRAPINGLETS GET DIRTYatomicant.co.uk
  13. 13. DATA MINING - SCRAPINGLETS GET DIRTYatomicant.co.uk
  14. 14. DATA MINING - SCRAPINGLETS GET DIRTYatomicant.co.uk
  15. 15. DATA MINING - SCRAPINGLETS GET DIRTY Request a site/content Run through the html – DOM - selectors Do whatever you want with the data Save the dataatomicant.co.uk
  16. 16. SCRAPERWIKIREFINE AND SCRAPE DATAatomicant.co.uk
  17. 17. SCRAPERWIKIWHAT IS IT? HOW TO USE ITScrape and link data using Ruby, Python and PHP scriptsthat run maintenance-free in the cloud. Request data forscoops and better decisions.atomicant.co.uk
  18. 18. DATA MINING - SCRAPINGLETS GET DIRTY
  19. 19. SCRAPERWIKIWHAT IS IT? HOW TO USE ITatomicant.co.uk
  20. 20. SCRAPERWIKIWHAT IS IT? HOW TO USE IT Why would you want to use SCRAPERWIKI rather than other scraping tools or custom code?atomicant.co.uk
  21. 21. SCRAPERWIKIWHAT IS IT? HOW TO USE IT • The dataset is available to everyone • Anyone can access the data through API • If the source changed and the scraper brakes, anyone can fix the scraper • Anyone can fork the scraperatomicant.co.uk
  22. 22. IS THAT IT?CERTAINLY NOT
  23. 23. SCRAPERWIKIWHAT IS IT? HOW TO USE ITatomicant.co.uk
  24. 24. GOOGLE REFINEWHAT IS IT? HOW TO USE ITGoogle Refine is a power tool for working with messy data,cleaning it up, transforming it from one format into another,extending it with web services,...atomicant.co.uk
  25. 25. VISUALISETELL THE STORYThere is more to thatIts just not data with values in a spreadsheet or databaseData can tell the story!atomicant.co.uk
  26. 26. GOOGLE FUSION TABLESWHAT IS IT? HOW TO USE ITEasy visualisation http://tables.googlelabs.com/atomicant.co.uk
  27. 27. SCRAPING WITH DRUPALAND NOW FOR SOMETHING COMPLETELY DIFFERENTFeeds – http://drupal.org/project/feedsScrapingFeeds query path parser - project/feeds_querypath_parserFeeds xpath parser – project/feeds_xpathparserCleaning up dataFeeds tamper - http://drupal.org/project/feeds_tamperatomicant.co.uk
  28. 28. VISUALISE WITH DRUPALAND NOW FOR SOMETHING COMPLETELY DIFFERENTMapping- Location – http://drupal.org/project/location- Openlayers – http://drupal.org/project/openlayers- Gmap – http://drupal.org/project/gmapGraphs/Charts- Graphs- Graphs Charts- Open Flash Chart- Viewsatomicant.co.uk
  29. 29. GO! SCRAPE IT!CHALLENGEEU Open Data Challenge- €20,000 to win- 28 days left to enterhttp://opendatachallenge.org/atomicant.co.uk
  30. 30. TOOLSSCRAPING DATAScraperWiki – http://scraperwiki.comPHP Simple HTML DOM – http://bit.ly/phphtmldomPHPQuery - http://code.google.com/p/phpquery/Open Data Kit - http://opendatakit.org/atomicant.co.uk
  31. 31. TOOLSCLEANING DATAGoogle Refine - http://code.google.com/p/google-refine/atomicant.co.uk
  32. 32. TOOLSVISUALIZING DATAGoogle fusion tables - http://tables.googlelabs.com/The Best Tools for Visualization - http://rww.to/toolsforvisatomicant.co.uk
  33. 33. TOOLSVISUALIZING DATAOpenHeatmap http://bit.ly/openheatmapatomicant.co.uk
  34. 34. THANK YOUQ&A | LETS CONNECT QUESTIONS?@sotak - twitterhttp://sotak.co.uk - personal bloghttp://atomicant.co.uk - company websiteatomicant.co.uk

×