This document discusses MongoDB and its use by Sunlight Foundation for three open data projects: the National Data Catalog, Real-Time Congress API, and Open State Project. MongoDB allows storing disparate data sources and formats in a schemaless manner. This enables aggregating large amounts of raw data and serving it through lightweight RESTful APIs. The document provides examples of congressional data stored and filtered in MongoDB.

1. MongoDB
@ Sunlight
Luigi Montanez
luigi@sunlightfoundation.com

3. Question? @LuigiMontanez

4. Question? @LuigiMontanez
Open Source + Open Data
=
Open Government

5. Question? @LuigiMontanez
High Quality Raw Data
✴ First: Raw data in JSON, XML, or CSV
✴ Second: RESTful APIs in JSON or XML
✴ Third: Nothing else...

6. Question? @LuigiMontanez
MongoDB enables
open data

7. Question? @LuigiMontanez
JSON has won
(among developers)

8. Question? @LuigiMontanez
Opening Up Data
✴ Storing data from disparate sources
✴ Data dumps
✴ Web scraping
✴ Text/PDF parsing
✴ Serving RESTful JSON APIs

9. Question? @LuigiMontanez
Three Projects
✴ National Data Catalog
✴ Real-Time Congress API
✴ Open State Project

10. Question? @LuigiMontanez
Three Projects
✴ National Data Catalog
✴ Real-Time Congress API
✴ Open State Project

12. Question? @LuigiMontanez
App design
drives
schema design

14. Text
{
"title": "Worldwide M1+ Earthquakes, Past Hour"
}

16. Text
{
"title": "Worldwide M1+ Earthquakes, Past Hour",
"description": "Real-time, worldwide earthquake list for the past h
"homepage": "http://data.gov/raw/32",
"official_docs": "http://earthquake.usgs.gov/eqcenter/catalogs/",
"organization": "Department of the Interior",
"original_catalog": "data.gov",
}

17. Text
{
"title": "Worldwide M1+ Earthquakes, Past Hour",
"description": "Real-time, worldwide earthquake list for the past
"homepage": "http://data.gov/raw/32",
"official_docs": "http://earthquake.usgs.gov/eqcenter/catalogs/",
"organization_id": "4cbcc0ff2c34576ba4000001",
"catalog_id": "4cbcc0ab2d34d76b97020433",
}

18. {
"title": "Worldwide M1+ Earthquakes, Past Hour",
"description": "Real-time, worldwide earthquake list for the past h
"homepage": "http://data.gov/raw/32",
"official_docs": "http://earthquake.usgs.gov/eqcenter/catalogs/",
"organization": { "name": "Department of the Interior",
"id": "4cbcc0ff2c34576ba4000001",
"slug": "us-dept-of-interior"
},
"original_catalog": { "name": "data.gov",
"id": "4cbcc0ab2d34d76b97020433",
"slug": "datagov"
}
}

20. {
"title": "Worldwide M1+ Earthquakes, Past Hour",
"description": "Real-time, worldwide earthquake list for the past h
"homepage": "http://data.gov/raw/32",
"official_docs": "http://earthquake.usgs.gov/eqcenter/catalogs/",
"organization": {
"name": "Department of the Interior",
"id": "4cbcc0ff2c34576ba4000001",
"slug": "us-dept-of-interior"
},
"original_catalog": {
"name": "data.gov",
"id": "4cbcc0ab2d34d76b97020433",
"slug": "datagov"
},
"downloads": [ { "type": "csv", "url": "http://data.gov/download/32
"ratings" : {
"average_rating": 3.5,
"rating_count": 23
},
"comments": []
}

22. Question? @LuigiMontanez
User-centric data?
✴ Source document: contains collection of
user data
✴ User document: contains collection of
source data
✴ UserSource document
✴ Rating, Favorite, Note docs

23. Question? @LuigiMontanez
Freedom of choice

24. Question? @LuigiMontanez
Three Projects
✴ National Data Catalog
✴ Real-Time Congress API
✴ Open State Project

25. Real-Time Congress API
(Drumbone)
Credit: vgm8383 on Flickr

26. Android App: “Congress”

27. Politiwidgets

28. Question? @LuigiMontanez
Requirements
✴ Aggregate lots of data
Biographical, Bills, Votes, Earmarks,
Video Clips, Floor Updates, Legislative
Documents, Committee Schedules,
Contributions, Interest Group Ratings
✴ Lightweight responses

29. {legislator: {
in_office: true,
title: "Rep",
nickname: "",
district: "9",
bioguide_id: "L000551",
govtrack_id: "400237",
phone: "202-225-2661",
website: "http://lee.house.gov/index.html",
twitter_id: "",
last_name: "Lee",
name_suffix: "",
last_updated: "2010/04/13 00:00:14 +0000",
party: "D",
chamber: "house",
state: "CA",
youtube_url: "http://www.youtube.com/RepLee",
first_name: "Barbara",
gender: "F",
congress_office: "2444 Rayburn House Office Building",
earmarks: {
average_number: 20,
total_amount: 10000000,
average_amount: 22994535,
total_number: 28,
last_updated: "2010-03-18",
fiscal_year: 2010,
}
...
}

30. // limit selection to a subset of fields
db.people.find( { 'first_name' : 'john' },
{ 'last_name' : 1,
'address' : 1 } );
// use dot-notation to dig into an object
db.people.find( { 'state': 'CA' },
{ 'address.zip_code': 1 } );

31. {legislator: {
last_name: "Lee",
first_name: "Barbara",
state: "CA",
earmarks: {
average_number: 20,
total_amount: 10000000,
average_amount: 22994535,
total_number: 28,
last_updated: "2010-03-18",
fiscal_year: 2010,
}
}
?sections=last_name,first_name,state,earmarks

32. {legislator: {
last_name: "Lee",
first_name: "Barbara",
state: "CA",
earmarks: {
total_amount: 10000000,
total_number: 28
}
}
?sections=last_name,first_name,state,earmarks.total_amount,earmarks.total_number

33. Question? @LuigiMontanez
Partial responses
make payloads
smaller

34. Question? @LuigiMontanez
Three Projects
✴ National Data Catalog
✴ Real-Time Congress API
✴ Open State Project

36. Question? @LuigiMontanez
50 States =
50 Formats

37. Question? @LuigiMontanez
Schemalessness
allows for
losslessness

38. Source Scraped JSON
Python
Transform
PostgreSQL

39. Source Scraped JSON MongoDB

40. Question? @LuigiMontanez
Three Projects
✴ National Data Catalog
✴ Real-Time Congress API
✴ Open State Project

41. Question? @LuigiMontanez
Thanks!
sunlightlabs.com
@LuigiMontanez