Citizen Activism using Scrubyt and RoR

 Only partially available
online
 Formatted as web page or
PDF
 Hard to search
 Can’t subscribe
 Can’t visualize
 Can’t re-use

Publishing Structured Data Visualization
Feeds • Makes it easy to find new
• Ability to subscribe to patterns.
interesting data
• Data streams can be ‘mashed’
in new ways.
Collaborative Crowdsourcing
Organization • Combines skills and input of
• Tagging, Voting, Sharing large numbers of people

• Governments publish
Governments data streams
publish data
streams
• 3rd parties create tools for
analysis and oversight
3rd
Issues are
Party
Citizens
monitor data
• Citizens collaboratively
resolved
Tools streams monitor their
government
• Citizens detect issues,
Issues are
detected
give feedback
• Issues are resolved

 Government has little
incentive
▪ Usually has disincentive
Why can’t the
 Don’t want a single
government do monolithic solution
everything? ▪ Want to allow evolution of best-
of-breed tools
 Tools created by citizens, for
citizens

 Focus:
 US Congress
 California
Legislature
 Gives grants to
online
transparency
tools
 $3.5 M Seed

A recent US
Congress bill
Groups for Groups
bill against bill

Votes
Donations

Publishing Structured Data Visualization
Feeds • MAPLight makes relationship
• MAPLight is a mashup of between money and votes
data streams from different visible.
sources.
Collaborative Crowdsourcing
Organization • Thousands of journalists,
• Advocacy group tags advocates, and citizens can
donating companies as browse data and flag issues.
belonging to interest groups.

 Accelerate online transparency
Ideas  Raise Awareness
 With public
 With government
Skills  Raise Money
 Fund External Development:
Funds  Grants
 Contests

Prove
Concept
Get Publicity Direct Attention and
Money and to Online
Tools For Transparency
Raise
Awareness
Show What’s
Possible

 2003 Directive: Must
publish travel and
hospitality expenses
on the web
 No standards for
presentation defined

124 Departments
- All different

Standardize Stream Visualize
• Scrape data into • Publish RSS • Provide basic
standard format feeds visualization app
• Run contest

1. LEARNING TEMPLATE 2. PRODUCTION SCRAPER
Input Input
• Example Page • Any Page with
• Example Text Same Format
Output: Output:
• XML
• Production Scraper
• XML

 Create a system
where non-coders
can train a scraper.

PRO CON
 Ability to use ‘learning’  Learning mode fails hard
example (sometimes)
 Doesn’t always learn
 Syntax integrates XML
builder
 Supports all hpricot Xpath
operations
Note: For compatibility reasons, this project uses an older version of scrubyt.
Issues may be fixed in newer version.

 Create a system
where non-coders
can train a scraper.
.... Didn’t work.

Still need coders w/ the following expertise:
1. XPath XML resolution
2. Regular Expressions
3. Firebug

1. Open This Link
2. Paste This Text

...created in the
background

Go To Next Level

Split Level: Two Types of Links
Open This Link

Select Element
Get the XPath

Split Level: Two Types of Links

...created in the
background

Test Random Reports
Send Home

 Goal: Finish scraping in one day
 12/124 Completed: 112 to go
 5-20 Volunteers
 5-20 min. per department
 Downloadable app w/ setup instructions
 Integrated examples
 Benefits:
 Excuse to use scrubyt, firebug
 On-site tutorial + guidance
 Easy intro to a Rails App

Jennifer Bell
visiblegovernment.ca