• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Open Government Data and MongoDB
 

Open Government Data and MongoDB

on

  • 1,533 views

Given at MongoDC on June 27, 2011.

Given at MongoDC on June 27, 2011.

Statistics

Views

Total Views
1,533
Views on SlideShare
1,251
Embed Views
282

Actions

Likes
4
Downloads
11
Comments
0

9 Embeds 282

http://www.10gen.com 266
http://www.mongodb.com 5
http://www.slideshare.net 3
http://drupal1.10gen.cc 2
http://elaiggk.mongodb.org 2
url_unknown 1
http://ww.mongodb.org 1
http://downloads.distro.mongodb.org 1
http://tray.mongodb.org 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Open Government Data and MongoDB Open Government Data and MongoDB Presentation Transcript

    • Open GovernmentData & MongoDB Luigi Montanez luigi@sunlightfoundation.com
    • Question? @LuigiMontanez
    • Open Data + Open Source = Open Government Question? @LuigiMontanez
    • MongoDB enables open data Question? @LuigiMontanez
    • Opening Up Data✴ Gather data from disparate sources ✴ Data dumps (SQL, Fixed-width columns) ✴ Web scraping ✴ Text/PDF parsing✴ Serving RESTful JSON APIs Question? @LuigiMontanez
    • JSON✴ Tree structure, not tabular✴ Still relational✴ JSON for data, XML for documents✴ Closely resembles native data structures✴ No manual parsing needed Question? @LuigiMontanez
    • Three Projects✴ Poligraft✴ Real Time Congress API✴ Open State Project Question? @LuigiMontanez
    • Three Projects✴ Poligraft✴ Real Time Congress API✴ Open State Project Question? @LuigiMontanez
    • App design drivesschema design Question? @LuigiMontanez
    • { "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com"} Text
    • { "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................",} Text
    • { "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [...] Text}
    • { "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [ { Text "name": "Barack Obama", "type": "politician", }, ... ]}
    • { "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [ { Text "name": "Barack Obama", "type": "politician", "breakdown": {"indiv": "33", "pac": "67"} "top_industries": ["Lawyers/Lobbyists","Finance/Insurance/ Real Estate","Misc. Business"] }, ... ]}
    • Natural Schemas Question? @LuigiMontanez
    • Three Projects✴ Poligraft✴ Real Time Congress API✴ Open State Project Question? @LuigiMontanez
    • Real-Time Congress API Credit: vgm8383 on Flickr
    • Android App: “Congress”
    • Politiwidgets
    • Requirements✴ Aggregate lots of data Biographical, Bills, Votes, Earmarks, Video Clips, Floor Updates, Legislative Documents, Committee Schedules, Contributions, Interest Group Ratings✴ Lightweight responses Question? @LuigiMontanez
    • {legislator: { in_office: true, title: "Rep", nickname: "", district: "9", bioguide_id: "L000551", govtrack_id: "400237", phone: "202-225-2661", website: "http://lee.house.gov/index.html", twitter_id: "", last_name: "Lee", name_suffix: "", last_updated: "2010/04/13 00:00:14 +0000", party: "D", chamber: "house", state: "CA", youtube_url: "http://www.youtube.com/RepLee", first_name: "Barbara", gender: "F", congress_office: "2444 Rayburn House Office Building", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, } ...}
    • // limit selection to a subset of fieldsdb.people.find( { first_name : john }, { last_name : 1, address : 1 } );// use dot-notation to dig into an objectdb.people.find( { state: CA }, { address.zip_code: 1 } );
    • ?sections=last_name,first_name,state,earmarks {legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, } }
    • ?sections=last_name,first_name,state,earmarks.total_amount,earmarks.total_number {legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { total_amount: 10000000, total_number: 28 } }
    • Partial responses make payloads smaller Question? @LuigiMontanez
    • Three Projects✴ Poligraft✴ Real Time Congress API✴ Open State Project Question? @LuigiMontanez
    • 50 States =50 Formats Question? @LuigiMontanez
    • Schemalessnessallows for granular control Question? @LuigiMontanez
    • Custom Fields✴ Traditional RDBMS ✴ Update the schema for new fields, run a migration, feel icky ✴ Create a custom_fields table✴ MongoDB ✴ Just store it Question? @LuigiMontanez
    • Speaking JSON natively Question? @LuigiMontanez
    • PythonSource Scraped JSON PostgreSQL Transform
    • Source Scraped JSON MongoDB
    • Three Projects✴ Poligraft✴ Real Time Congress API✴ Open State Project Question? @LuigiMontanez
    • Developer Happiness
    • Thanks!sunlightlabs.com@LuigiMontanez Question? @LuigiMontanez