Ubiquitous Angels; ambient sensor networks to crowd source crisis response and community awareness


Published on

A presentation for Open Source Bridge that summarizes the 6 weeks of research and development that I just wrapped up while at the Banff Centre for a residency on Arts and Sciences for Saul and Charles.



We live in dense urban landscapes that have crisis at many different scales. A crisis may be a large event such as a tsunami, an earthquake, a terrorist attack or other large scale disaster. A crisis may be something smaller such as a house fire, or a pileup on the freeway, a gas leak or a road out. A crisis may be something relatively minor, a crisis only to one person, such as being locked out of ones car, or trying to find a lost pet, or needing help sharing babysitting duties.

For the purpose of this talk I am interested in the far end of the long tail of crisis response – and what open source technology we can strap together to build tactical systems for helping tackle such crisis. I have been developing an approach using off the shelf tools that can help resolve local crisis – the smaller more personal crisis that interrupts peoples lives.

If we watch messages over a city, and provide analytic views for interested parties, then a role for “angels” to participate emerges. Interested individuals can act in a matchmaking capacity to connect complementary interests together. We can collaboratively up-score or down-score public comments aggregated from a variety of geo local sources. We can filter away noise so that responders can more easily decide what to respond to. In sum we can make it possible for people on the ground, in the area, to volunteer to help out by providing clarity.

I consider these class of services to be a kind of ubiquitous angel. They are not something that you have to even know exist until they help you.

The tools that we can leverage today to build such services are worth knowing – having many applications. In particular solr ( an enterprise search engine ) and carrot2 ( a topic clustering engine ) can play a powerful role in helping filter noise. Other related projects in particular Ushahidi and Swift have a high degree of overlap and those technologies will be discussed as well.

Beyond today there’s a role for such technology to help us deal with new kinds of crisis in the future. Over the next decades we may be facing economic and environmental concerns that are granular, evenly distributed and problematic in resolution. These kinds of tools and approaches may be useful there as well.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Ubiquitous Angels; ambient sensor networks to crowd source crisis response and community awareness

  1. 1. Ubiquitous Angels Open Source Bridge Portland Oregon June 2009 http://opensourcebridge.org/sessions/246 twitter.com/anselm blog.makerlab.org anselm hook
  2. 2. What I’ll cover • A look at other similar sensor efforts • A little bit of technicals • Critical perspectives
  3. 3. If • we could see our neighborhoods better • There might be an opportunity to participate • We could better help people in need • We could connect people together with similar needs • We could crowd-source social problems • We all know this.
  4. 4. A big use case • #iranelection • where are the protests? • are there specific instances of vote fraud? • who are the key twitterers? • are there specific things we can do? • are there local people who can help?
  5. 5. A smaller use case • Restaurants often dump food every night • Gleaners try to collect food • How can we connect dumpers with gleaners?
  6. 6. Other use cases • My child is missing. • Somebody just stole my bike. • Can somebody give me a ride? • Does anybody have a place where I can stay? • Car pooling? • I’d like somebody to share my war stories with.
  7. 7. Crisis at many levels
  8. 8. Where is the real time map of us?
  9. 9. Hard to Automate • ‘Automated maps of feeds look like database barf on a map’ - Chris Blow
  10. 10. Ushahidi Swift Initiative • blog.ushahidi.com/index.php/tag/swift/ • a crisis listening tool • the Mumbai Crisis for example • electoral fraud events such as in Iran • the upcoming quake in San Francisco • crowd source filtering to eliminate noise • crisis editors mark posts up with metadata • first responders can see filtered views
  11. 11. Ushahidi Swift
  12. 12. Swift Control Panel
  13. 13. Code and Praxis
  14. 14. A software stack for a civic lens • search - to help restrict the firehose that is twitter and co. • aggregation - as tuned by search • reaping - throwing away old data as an ongoing process • actual fetching content from twitter and co. • geolocation of content by any means possible • natural language parsing for semantic content and meaning • language translation from foreign languages to english • presentation layer for the sit-at-home-view and in-the-field views • curatorial role in marking data up or changing metadata • first responder views, feedback and loop closure
  15. 15. Roles • Participant - somebody involved in an issue • Reporter - as an on the ground witness • Curator - acting as an editor to filter reports • Matchmaker - matching reports with potential responders • Responder - going into the field to help resolve a situation.
  16. 16. John Nunemaker’s Twitter Gem • http://github.com/jnunemaker/twitter • http://apiwiki.twitter.com • h = Twitter::HTTPAuth.new(name,pass) • twitter = Twitter::Base.new(h) • results = twitter.user_timeline(:id =>‘joi’) • results.each { |twit| puts twit.location }
  17. 17. Social Graph is easy • twitter.friends(:id => ‘joi’ ) • Try to rate limit queries • Build your own model of the social graph • Easy to do, crucial to provide subjectivity • An alternative is to go through YQL • Social graph issues underpin many projects
  18. 18. Fetching social graph
  19. 19. YQL for the win • yql = “http://query.yahooapis.com/v1/public/yql?q=” • query = “select * from party where id = ‘joi’ or id = ‘meedan’ or id = ‘bob’” ... • schema = “use ‘http://angel.makerlab.org/yql/ twitter.user.timeline.xml’ as party;” • url = “#{yql}#{url_escape(schema+query)} • url = url + “format=json”; • data = JSON.parse(open(url).read) • data[“query”][“results”].each { |i| puts i.title } • a higher rate limit ~may~ allow more data access
  20. 20. Metacarta Geolocation • request = Net::HTTP::Get.new(‘/webservices/ GeoTagger/JSON/basic?version=1.0.0&doc=banff!’) • request.basic_auth name,pass • h = Net::HTTP.start(‘ondemand.metacarta.com’) • response = h.request(request) • data = JSON.parse(response.body) • latitude = data[“locations”][0][“Centroid”][“Latitude”]
  21. 21. Yahoo Placemaker • geolocating posts by brute force • url = URI.parse(‘http://wherein.yahooapis.com/v1/document’) • args = {‘documentContent’ => ‘banff’, ‘documentType’ => ‘text/plain’, ‘appid’=>mysecretappkey } • r = Net::HTTP.post_form(url,args) • doc = Hpricot::XML(response.body) • latitude = (node/:latitude).innerHTML • longitude = (node/:longitude).innerHTML
  22. 22. acts_as_solr for search • picky and brittle • doesn’t play well with others • can geo-constrain searches more or less • query = “volunteer lat:[42 TO 60]” • Model.rebuild_solr_index() • results = Model.find_by_solr(query) • results.docs { |data| puts data.name }
  23. 23. engtagger for meaning extraction • engtagger.rubyforge.org • phrase = “is anyone awake out there? i just got mugged in front of my house and could really use help calling the police” • tagger = EngTagger.new • tagged = tgr.add_tags(phrase) • nouns = tagger.get_nouns(tagged) • {“police”=>1,”help”=>1, ... }
  24. 24. Clustering Approaches to clump activity • Carrot2 • Amazon Turk • Human ‘volunteer turk’ approach • Contextual Network Graphs • Latent Semantic Indexing
  25. 25. Carrot2 • project.carrot2.org • Submit your requests directly via http • Can only cluster 1000 documents at a time • Used as a way to get even more metadata • Forms clusters dynamically
  26. 26. Brief Comments on Visualization • Views are political • Subjectivity is crucial • Speed of user interface; probably gamelike • Tactical; showing immediate context • http://map.cartagen.org for example
  27. 27. Still looking for similar efforts - any advice? • I call this class of services ‘real time brokerages’. But I don’t see many of them. • Craigslist? • Shizzow? • IceCondor? • Other?
  28. 28. Criticism • Side effects often dominate over intended consequences of any project or endeavor. • Abstract views may make us callous or may badly reflect ground truth. • Ignorance is bliss. The world is filled with sob stories. Best to not dwell? • Such services may be used solely for the most banal aspirations and goals. • Struggle may be important - making things too easy may harm fitness and lower diversity of skills and abilities over long term. • What about poor people who are outside of any implied technological social network? • Why not just help people around you? • Any technology should go hand in hand with day to day personal practice that is unmediated? • Feedback loops may be created that accelerate and disrupt society. • Virtual and visual only for curators; not tactile; uses only one sense. A concern?
  29. 29. links http://swiftapp.org/ http://crisismapping.ning.com/ http://ushahidi.com http://ncfoodnet.com/ http://www.flickr.com/photos/unthinkingly/3507131912/ http://cartagen.net http://blog.twittervotereport.com/ http://blog.ushahidi.com/?s=swift http://www.qapture.net/lists/ http://www.marketingpilgrim.com/2008/12/social-media-monitoring-tools.html http://tinker.com/users/find/?findName=&findEmail=&findLocation=banff&profType= http://www.crunchgear.com/2007/04/02/patent-monkey-geographic-cell-messaging-with-squareloop/ http://www.dantynan.com/2009/04/02/twitter-added-to-federal-emergency-response-network/ ( April Fools Gag ) http://stephensonstrategies.com/directory-of-major-blog-posts/boy-did-i-underestimate-twitters-value-in-a-disaster/ http://www.comminit.com/en/node/290987/307 http://yvoschaap.com/twitterthoughts/?map=1 http://mashable.com/2009/03/16/twitter-visualizations/ http://theeword.blogspot.com/2009/06/best-twitter-applications-for-social.html http://monitter.com/ http://maphawk.blogspot.com/2009/01/geographic-search-engine-for-news.html http://muti.co.za/static/newsmap.html http://www.planetizen.com/node/38227 http://www.localtweeps.com/ http://keithhopper.com/taxonomy/term/28 http://keithhopper.com/blog/finding-twitterstars http://pipes.yahoo.com/pipes/pipe.run?Location=portland%2C+oregon&radius=15&_render=rss&_id=2fdf06974be88f13a63937fb4d36e499 http://delicious.com/tag/twitter+map http://www.twitip.com/twellowhood-find-local-twitter-users/ http://www.tweepz.com/feed?q=loc%3Abanff&C= http://www.twitterlocal.net/ ( old ) http://www.frontlinesms.com/
  30. 30. thanks http://github.com/ anselm/angel http://github.com/ anselm/SpinnyGlobe