Open Government Data and MongoDB

Open GovernmentData & MongoDB Luigi Montanez luigi@sunlightfoundation.com

Question? @LuigiMontanez

Open Data + Open Source = Open Government Question? @LuigiMontanez

MongoDB enables open data Question? @LuigiMontanez

Opening Up Data✴ Gather data from disparate sources ✴ Data dumps (SQL, Fixed-width columns) ✴ Web scraping ✴ Text/PDF parsing✴ Serving RESTful JSON APIs Question? @LuigiMontanez

JSON✴ Tree structure, not tabular✴ Still relational✴ JSON for data, XML for documents✴ Closely resembles native data structures✴ No manual parsing needed Question? @LuigiMontanez

Three Projects✴ Poligraft✴ Real Time Congress API✴ Open State Project Question? @LuigiMontanez

App design drivesschema design Question? @LuigiMontanez

{ "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com"} Text

{ "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................",} Text

{ "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [...] Text}

{ "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [ { Text "name": "Barack Obama", "type": "politician", }, ... ]}

{ "title": "President Obamas climate Plan B in hot water -Darren Samuelsohn - POLITICO.com", "slug": "EOsc", "source_url": "http://www.politico.com/news/stories/ 0810/40534.html", "content": ".................", "entities": [ { Text "name": "Barack Obama", "type": "politician", "breakdown": {"indiv": "33", "pac": "67"} "top_industries": ["Lawyers/Lobbyists","Finance/Insurance/ Real Estate","Misc. Business"] }, ... ]}

Natural Schemas Question? @LuigiMontanez

Real-Time Congress API Credit: vgm8383 on Flickr

Android App: “Congress”

Politiwidgets

Requirements✴ Aggregate lots of data Biographical, Bills, Votes, Earmarks, Video Clips, Floor Updates, Legislative Documents, Committee Schedules, Contributions, Interest Group Ratings✴ Lightweight responses Question? @LuigiMontanez

{legislator: { in_office: true, title: "Rep", nickname: "", district: "9", bioguide_id: "L000551", govtrack_id: "400237", phone: "202-225-2661", website: "http://lee.house.gov/index.html", twitter_id: "", last_name: "Lee", name_suffix: "", last_updated: "2010/04/13 00:00:14 +0000", party: "D", chamber: "house", state: "CA", youtube_url: "http://www.youtube.com/RepLee", first_name: "Barbara", gender: "F", congress_office: "2444 Rayburn House Office Building", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, } ...}

// limit selection to a subset of fieldsdb.people.find( { first_name : john }, { last_name : 1, address : 1 } );// use dot-notation to dig into an objectdb.people.find( { state: CA }, { address.zip_code: 1 } );

?sections=last_name,first_name,state,earmarks {legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { average_number: 20, total_amount: 10000000, average_amount: 22994535, total_number: 28, last_updated: "2010-03-18", fiscal_year: 2010, } }

?sections=last_name,first_name,state,earmarks.total_amount,earmarks.total_number {legislator: { last_name: "Lee", first_name: "Barbara", state: "CA", earmarks: { total_amount: 10000000, total_number: 28 } }

Partial responses make payloads smaller Question? @LuigiMontanez

50 States =50 Formats Question? @LuigiMontanez

Schemalessnessallows for granular control Question? @LuigiMontanez

Custom Fields✴ Traditional RDBMS ✴ Update the schema for new ﬁelds, run a migration, feel icky ✴ Create a custom_ﬁelds table✴ MongoDB ✴ Just store it Question? @LuigiMontanez

Speaking JSON natively Question? @LuigiMontanez

PythonSource Scraped JSON PostgreSQL Transform

Source Scraped JSON MongoDB

Developer Happiness

Thanks!sunlightlabs.com@LuigiMontanez Question? @LuigiMontanez

http://www.10gen.com	266
http://www.slideshare.net	3
http://drupal1.10gen.cc	2
http://elaiggk.mongodb.org	2
url_unknown	1
http://ww.mongodb.org	1
http://downloads.distro.mongodb.org	1
http://tray.mongodb.org	1

Open Government Data and MongoDB

by Luigi Montanez

Accessibility

Upload Details

Usage Rights

8 Embeds 277

Statistics