Ubiquitous Angels
Open Source Bridge Portland Oregon June 2009
http://opensourcebridge.org/sessions/246
twitter.com/anselm
blog.makerlab.org
anselm hook

What I’ll cover
• A look at other similar sensor efforts
• A little bit of technicals
• Critical perspectives

If
• we could see our neighborhoods better
• There might be an opportunity to participate
• We could better help people in need
• We could connect people together with
similar needs
• We could crowd-source social problems
• We all know this.

A big use case
• #iranelection
• where are the protests?
• are there specific instances of vote fraud?
• who are the key twitterers?
• are there specific things we can do?
• are there local people who can help?

A smaller use case
• Restaurants often dump food every night
• Gleaners try to collect food
• How can we connect dumpers with
gleaners?

Other use cases
• My child is missing.
• Somebody just stole my bike.
• Can somebody give me a ride?
• Does anybody have a place where I can stay?
• Car pooling?
• I’d like somebody to share my war stories
with.

Crisis at many levels

Where is the real time
map of us?

Hard to Automate
• ‘Automated maps of feeds look like
database barf on a map’ - Chris Blow

Ushahidi Swift Initiative
• blog.ushahidi.com/index.php/tag/swift/
• a crisis listening tool
• the Mumbai Crisis for example
• electoral fraud events such as in Iran
• the upcoming quake in San Francisco
• crowd source filtering to eliminate noise
• crisis editors mark posts up with metadata
• first responders can see filtered views

Ushahidi Swift

Swift Control Panel

Code and Praxis

A software stack for a
civic lens
• search - to help restrict the firehose that is twitter and co.
• aggregation - as tuned by search
• reaping - throwing away old data as an ongoing process
• actual fetching content from twitter and co.
• geolocation of content by any means possible
• natural language parsing for semantic content and meaning
• language translation from foreign languages to english
• presentation layer for the sit-at-home-view and in-the-field views
• curatorial role in marking data up or changing metadata
• first responder views, feedback and loop closure

Roles
• Participant - somebody involved in an issue
• Reporter - as an on the ground witness
• Curator - acting as an editor to filter reports
• Matchmaker - matching reports with
potential responders
• Responder - going into the field to help
resolve a situation.

John Nunemaker’s
Twitter Gem
• http://github.com/jnunemaker/twitter
• http://apiwiki.twitter.com
• h = Twitter::HTTPAuth.new(name,pass)
• twitter = Twitter::Base.new(h)
• results = twitter.user_timeline(:id =>‘joi’)
• results.each { |twit| puts twit.location }

Social Graph is easy
• twitter.friends(:id => ‘joi’ )
• Try to rate limit queries
• Build your own model of the social graph
• Easy to do, crucial to provide subjectivity
• An alternative is to go through YQL
• Social graph issues underpin many projects

Fetching social graph

YQL for the win
• yql = “http://query.yahooapis.com/v1/public/yql?q=”
• query = “select * from party where id = ‘joi’ or id =
‘meedan’ or id = ‘bob’” ...
• schema = “use ‘http://angel.makerlab.org/yql/
twitter.user.timeline.xml’ as party;”
• url = “#{yql}#{url_escape(schema+query)}
• url = url + “format=json”;
• data = JSON.parse(open(url).read)
• data[“query”][“results”].each { |i| puts i.title }
• a higher rate limit ~may~ allow more data access

Metacarta Geolocation
• request = Net::HTTP::Get.new(‘/webservices/
GeoTagger/JSON/basic?version=1.0.0&doc=banff!’)
• request.basic_auth name,pass
• h = Net::HTTP.start(‘ondemand.metacarta.com’)
• response = h.request(request)
• data = JSON.parse(response.body)
• latitude = data[“locations”][0][“Centroid”][“Latitude”]

Yahoo Placemaker
• geolocating posts by brute force
• url = URI.parse(‘http://wherein.yahooapis.com/v1/document’)
• args = {‘documentContent’ => ‘banff’,
‘documentType’ => ‘text/plain’,
‘appid’=>mysecretappkey }
• r = Net::HTTP.post_form(url,args)
• doc = Hpricot::XML(response.body)
• latitude = (node/:latitude).innerHTML
• longitude = (node/:longitude).innerHTML

acts_as_solr for search
• picky and brittle
• doesn’t play well with others
• can geo-constrain searches more or less
• query = “volunteer lat:[42 TO 60]”
• Model.rebuild_solr_index()
• results = Model.find_by_solr(query)
• results.docs { |data| puts data.name }

engtagger for meaning
extraction
• engtagger.rubyforge.org
• phrase = “is anyone awake out there? i just got
mugged in front of my house and could really use
help calling the police”
• tagger = EngTagger.new
• tagged = tgr.add_tags(phrase)
• nouns = tagger.get_nouns(tagged)
• {“police”=>1,”help”=>1, ... }

Clustering Approaches
to clump activity
• Carrot2
• Amazon Turk
• Human ‘volunteer turk’ approach
• Contextual Network Graphs
• Latent Semantic Indexing

Carrot2
• project.carrot2.org
• Submit your requests directly via http
• Can only cluster 1000 documents at a time
• Used as a way to get even more metadata
• Forms clusters dynamically

Brief Comments on
Visualization
• Views are political
• Subjectivity is crucial
• Speed of user interface; probably gamelike
• Tactical; showing immediate context
• http://map.cartagen.org for example

Still looking for similar
efforts - any advice?
• I call this class of services ‘real time
brokerages’. But I don’t see many of them.
• Craigslist?
• Shizzow?
• IceCondor?
• Other?

Criticism
• Side effects often dominate over intended consequences of any project or endeavor.
• Abstract views may make us callous or may badly reflect ground truth.
• Ignorance is bliss. The world is filled with sob stories. Best to not dwell?
• Such services may be used solely for the most banal aspirations and goals.
• Struggle may be important - making things too easy may harm fitness and lower
diversity of skills and abilities over long term.
• What about poor people who are outside of any implied technological social
network?
• Why not just help people around you?
• Any technology should go hand in hand with day to day personal practice that is
unmediated?
• Feedback loops may be created that accelerate and disrupt society.
• Virtual and visual only for curators; not tactile; uses only one sense. A concern?

links
http://swiftapp.org/
http://crisismapping.ning.com/
http://ushahidi.com
http://ncfoodnet.com/
http://www.flickr.com/photos/unthinkingly/3507131912/
http://cartagen.net
http://blog.twittervotereport.com/
http://blog.ushahidi.com/?s=swift
http://www.qapture.net/lists/
http://www.marketingpilgrim.com/2008/12/social-media-monitoring-tools.html
http://tinker.com/users/find/?findName=&findEmail=&findLocation=banff&profType=
http://www.crunchgear.com/2007/04/02/patent-monkey-geographic-cell-messaging-with-squareloop/
http://www.dantynan.com/2009/04/02/twitter-added-to-federal-emergency-response-network/ ( April Fools Gag )
http://stephensonstrategies.com/directory-of-major-blog-posts/boy-did-i-underestimate-twitters-value-in-a-disaster/
http://www.comminit.com/en/node/290987/307
http://yvoschaap.com/twitterthoughts/?map=1
http://mashable.com/2009/03/16/twitter-visualizations/
http://theeword.blogspot.com/2009/06/best-twitter-applications-for-social.html
http://monitter.com/
http://maphawk.blogspot.com/2009/01/geographic-search-engine-for-news.html
http://muti.co.za/static/newsmap.html
http://www.planetizen.com/node/38227
http://www.localtweeps.com/
http://keithhopper.com/taxonomy/term/28
http://keithhopper.com/blog/finding-twitterstars
http://pipes.yahoo.com/pipes/pipe.run?Location=portland%2C+oregon&radius=15&_render=rss&_id=2fdf06974be88f13a63937fb4d36e499
http://delicious.com/tag/twitter+map
http://www.twitip.com/twellowhood-find-local-twitter-users/
http://www.tweepz.com/feed?q=loc%3Abanff&C=
http://www.twitterlocal.net/ ( old )
http://www.frontlinesms.com/

thanks
http://github.com/
anselm/angel
http://github.com/
anselm/SpinnyGlobe