Your SlideShare is downloading. ×
Open Legislation Spring 2011 Talk 1
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Open Legislation Spring 2011 Talk 1

208
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
208
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Open Legislation Spring 2011
  • 2. Open Data(Government)
  • 3. Secondary Sources are nice● OpenCongress● GovTrack.US● OpenStates● FedSpending.org● Many more
  • 4. Primary Sources are better● Data.gov● USAspending.gov● California● Oregon● Washington● Many more
  • 5. Sometimes though...Open Data is not Enough. We need Platforms.
  • 6. A Different Breed of Open● Making data accessible: ● Built-in search ● Permanent URIs ● Standardized Feeds ● Real-time Alerts● REST Architecture with Feed Publishing ● RSS/Atom => Pubsubhubbub => Alerts
  • 7. So back toOpen Legislation
  • 8. Browse, Search, and Sharehttp://open.nysenate.gov/legislation
  • 9. Its not a Service;Its an Open Platform
  • 10. 1 Year Re-cap● Open Sourced It (for real)● Improved the API (xml/json)● Decreased Load Times● Restructured the Back-end● Basic Documentation● Wrapped into a build system
  • 11. The next year● In general.. ● Data Quality and Documentation ● Usage Tracking and Statistics ● User Interface Improvements ● Further separation of the Platform and Service● Right now ● Data Quality, Data Quality, Data Quality ● And a little bit of documentation
  • 12. The Senate has Legislative Data Quality issues?
  • 13. Well, not exactly● Legislative Research Service has the data ● Big, ancient mainframe to boot● They FTP us updates every 5 minutes ● In SOBI formats (what?) ● With some XML mixed in● We parse it back into XML/JSON/SQL structure
  • 14. Reasons for Difficulty● Poorly Documented SOBI behavior● Formatted as a change log (sometimes) ● Finding sources of error can be hard● LRS is not co-operative
  • 15. Solutions● Version Control ● Write objects to JSON/XML files ● With Git, commit each new version – Commit message points to the source SOBI ● Use git to trace data errors back to SOBI files● Unit Test known corner cases● Periodically do a scrape check?
  • 16. Progress✔ Parsing has been overhauled✔ Objects are written to file✔ Bugs have been found and fixed✔ Periodic Scrapes are approved
  • 17. A short task list✗ Integrate git into the parsing system.✗ Document expected behavoir✗ Write a small test suite✗ Try to avoid having to scrape.
  • 18. HFOSS Symposium 2011● Bryan Sivak – Civic Commons● Mark Prutalis – Sahana Foundation● Many universities, Mozilla, Google● David, Moorthy, Brian, and Myself! ● 1 Hour and a few 3 x 4 posters.

×