This document discusses using citizen activism and open source tools like Scrubyt and Ruby on Rails to increase government transparency. It notes current issues with accessing government data and proposes solutions like publishing structured data feeds, enabling data visualization, and facilitating collaborative organization between citizens. Citizens could monitor government data streams, detect issues, and provide feedback to help resolve problems. The document advocates giving grants to online transparency tools and outlines a specific example of the MAPLight project, which makes the relationship between political donations and voting visible. It also describes ideas for standardizing government data scraping and creating a learning template to allow non-coders to train scrapers.

1. Citizen Activism using Scrubyt and RoR

3.  Only partially available
online
 Formatted as web page or
PDF
 Hard to search
 Can’t subscribe
 Can’t visualize
 Can’t re-use

4. Publishing Structured Data Visualization
Feeds • Makes it easy to find new
• Ability to subscribe to patterns.
interesting data
• Data streams can be ‘mashed’
in new ways.
Collaborative Crowdsourcing
Organization • Combines skills and input of
• Tagging, Voting, Sharing large numbers of people

5. • Governments publish
Governments data streams
publish data
streams
• 3rd parties create tools for
analysis and oversight
3rd
Issues are
Party
Citizens
monitor data
• Citizens collaboratively
resolved
Tools streams monitor their
government
• Citizens detect issues,
Issues are
detected
give feedback
• Issues are resolved

7.  Government has little
incentive
▪ Usually has disincentive
Why can’t the
 Don’t want a single
government do monolithic solution
everything? ▪ Want to allow evolution of best-
of-breed tools
 Tools created by citizens, for
citizens

8.  Focus:
 US Congress
 California
Legislature
 Gives grants to
online
transparency
tools
 $3.5 M Seed

9. A recent US
Congress bill
Groups for Groups
bill against bill

10. Votes
Donations

11. Publishing Structured Data Visualization
Feeds • MAPLight makes relationship
• MAPLight is a mashup of between money and votes
data streams from different visible.
sources.
Collaborative Crowdsourcing
Organization • Thousands of journalists,
• Advocacy group tags advocates, and citizens can
donating companies as browse data and flag issues.
belonging to interest groups.

14.  Accelerate online transparency
Ideas  Raise Awareness
 With public
 With government
Skills  Raise Money
 Fund External Development:
Funds  Grants
 Contests

15. Prove
Concept
Get Publicity Direct Attention and
Money and to Online
Tools For Transparency
Raise
Awareness
Show What’s
Possible

17.  2003 Directive: Must
publish travel and
hospitality expenses
on the web
 No standards for
presentation defined

18. 124 Departments
- All different

23. Standardize Stream Visualize
• Scrape data into • Publish RSS • Provide basic
standard format feeds visualization app
• Run contest

25. 1. LEARNING TEMPLATE 2. PRODUCTION SCRAPER
Input Input
• Example Page • Any Page with
• Example Text Same Format
Output: Output:
• XML
• Production Scraper
• XML

28.  Create a system
where non-coders
can train a scraper.

29. PRO CON
 Ability to use ‘learning’  Learning mode fails hard
example (sometimes)
 Doesn’t always learn
 Syntax integrates XML
builder
 Supports all hpricot Xpath
operations
Note: For compatibility reasons, this project uses an older version of scrubyt.
Issues may be fixed in newer version.

30.  Create a system
where non-coders
can train a scraper.
.... Didn’t work.

31. Still need coders w/ the following expertise:
1. XPath XML resolution
2. Regular Expressions
3. Firebug

33. 1. Open This Link
2. Paste This Text

35. ...created in the
background

36. Go To Next Level

37. Split Level: Two Types of Links
Open This Link

38. Select Element
Get the XPath

39. Split Level: Two Types of Links

41. ...created in the
background

42. Test Random Reports
Send Home

43.  Goal: Finish scraping in one day
 12/124 Completed: 112 to go
 5-20 Volunteers
 5-20 min. per department
 Downloadable app w/ setup instructions
 Integrated examples
 Benefits:
 Excuse to use scrubyt, firebug
 On-site tutorial + guidance
 Easy intro to a Rails App

44. Jennifer Bell
visiblegovernment.ca