Vistology STIDS 2013 Situation Awareness from Social Media
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


Vistology STIDS 2013 Situation Awareness from Social Media



This paper describes VIStology's HADRian system for semantically integrating disparate information sources into a common operational picture (COP) for humanitarian assistance/disaster relief (HADR) ...

This paper describes VIStology's HADRian system for semantically integrating disparate information sources into a common operational picture (COP) for humanitarian assistance/disaster relief (HADR) operations. Here the system is applied to the task of determining where unexploded or additional bombs were being reported via Twitter in the hours immediately after the Boston Marathon bombing in April, 2013. We provide an evaluation of the results and discuss future directions.



Total Views
Views on SlideShare
Embed Views



1 Embed 2 2



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Distinguish from DISIS

Vistology STIDS 2013 Situation Awareness from Social Media Presentation Transcript

  • 2. About VIStology, Inc.  Past and present R&D contracts  ONR, Army RDECOM , AFOSR, AFRL, DARPA, MDA  Professional Collaborations  Northeastern University, W3C, Lockheed Martin, OMG, Referentia Systems, BBN, Raytheon, Vulcan, Inc.  Products & Services  BaseVISor: highly efficient inference engine  ConsVISor: consistency checker of ontologies  PolVIsor: policy-compliant information exchange  HADRian: next-generation smart COP for HA/DR ops  Ontology Engineering and Systems Design  Areas of Expertise  Level 2+ Information Fusion, Situation Awareness, Formal Reasoning Systems, Artificial Intelligence, Ontology Engineering, Object-Oriented Design 2
  • 3. VIStology HADRian  HADRian is a next-generation COP for HA/DR ops  HADRian enables an Incident Commander to  Find  Filter  Geocode and  Display the information that he or she needs to make the best decisions about the situation on the basis of semantic annotation of information repositories.  Repositories may contain text reports, photos, videos, tracks (KML), chemical plumes, 3D building models (SketchUp), etc. 4
  • 4. HADrian: Concept of Operations: Find, Filter, Map and Display Info Re:Disasters  COP Operator annotates repositories, using ontology  Info from new repositories can be potentially integrated by annotating metadata  COP operator formulates High Level Query to describes info needs for current operation  System infers repositories that may contain relevant info by reasoning over metadata that the repository has been annotated with.   Information remains in place until need (not ETL); Users upload data wherever they want; ingested as needed  System issues appropriate low level query to repositories  Repositories contain disparate data in disparate formats  System filters out irrelevant data  System aggregates and displays data in Google Earth COP  Users interact with data in COP  COP Operator can send information from COP to responder phone 5
  • 5. Dataflow for HADRian Situation Awareness from Social Media Twitter Status Updates Civilians/Media/ Others Ontologybased annotation of metadata Identifies relevant repositories based on metadata and high level queries. Here, only one repository is relevant Processes content of relevant tweets HADRian Formulates high level queries based on Commander needs: Here, “Where are unexploded or additional bombs reported after Boston Marathon?” Uses COP to make decisions. COP Operator Identifies information needs. Conveys to COP Operator Field Operators 1. Identifies tweets that report an additional or unexploded bomb. 2. Identifies where the bomb is reported 3. Geolocates the reported location 4. Anonymizes the reporter‟s identity 5. Presents the information in HADRian COP Commander 6
  • 6. Current State of the Art: Fixed Feeds, On/Off VIStology, Inc
  • 7. High Level Query “Show me Locations mentioned in Tweets about Boston Marathon and Unexploded Bombs”
  • 8. Repository Annotation
  • 9. Filter Processing Chain The bulk of 0.5 million tweets were collected on April 15, 2013 in 3 hours after the Boston Marathon bombing and stored in a giant CSV file. After this file is determined to be relevant: 1) Using metadata description of schema and rules, the 0.5m tweets were filtered to only those that contained mentions of unexploded or undetonated bombs, and converted into an OWL representation. This file is identified as being relevant to the HLQ “find locations mentioned in tweets about Boston Marathon and unexploded bombs at Boston Marathon on April 15, 2013” 2) The OWL-formatted tweets were loaded into BaseVISor along with rules. These rules extract information about locations mentioned in the tweets - we call them "location phrases", for instance: "jfk", "jfk library", "kennedy library", "#boston", etc.
  • 10. Location Phrases Locations referenced in tweet; NOT location of user, although that can help disambiguate INFO Initializing BaseVISor.. INFO There are 26679 asserted facts in the knowledge base. INFO Initialization complete. Running inference... INFO There is a total of 44414 facts in the knowledgebase after running the inference. INFO Done. INFO INFO INFO INFO INFO INFO INFO INFO INFO INFO 'mandarin hotel' -> 3 'bostonmarathon' -> 798 'bpd commissioner ed davis' -> 1 'back bay' -> 1 … 'jfk library' -> 311 'jfk' -> 73 'jfklibrary' -> 2 'jkf library' -> 1 … 'copley place' -> 8 … „st. ignatius catholic church‟ -> 47
  • 11. Mapping Location Phrases to Placemarks For each extracted location phrase a rule with a special procedural attachment is fired. This attachment, takes the location phrase and tries to locate the place on the map using the following algorithm:  1. (Places API, Exact) Lookup Google Maps Places API ( and see if it returns exact match for the phrase. If it does, use its Lat/Lon to assign the tweet to that location, otherwise, move to next step.  2. (Geocoding API) Lookup Google Geocoding API ( and see if it returns exact match for the phrase, if it does, use its Lat/Lon to assign the tweet to that location, otherwise move to the next step.  3. (Places API, First on the list) Pick the first result in the Google Maps Places API result (from 3a), if there were no results, ignore the location.
  • 12. 3 tweets mention Mandarin Hotel, 2 Copley Place, 1 Back Bay Station,…
  • 13. Information in Placemark  We indicate the source of      the location inside the placemark. (Here, Places API first result) Corresponding location phrases Photo Number of tweets (1158) Place type (here, “library, museum, establishment”) Sample Tweets (not shown)
  • 14. Placemark Size and Color The number next to the placemark's name indicates the number of tweets that used on of the location phrases mapping to this location. The higher the number, the more tweets were talking about the same spot. We emphasize this fact by rendering polygons underneath the placemarks the higher and darker the color, the more frequently mentioned was the location.
  • 15. 47 Tweets mention St Ignatius Church, at Boston College
  • 16. Relevant Tweet Retrieval (Finding) Evaluation  Corpus: ~500K tweets  Identified 7,748 tweets that were about additional or unexploded bombs with a precision of 94.5%, based on a random sample of 200 tweets identified as such.  That is, only 1.5% of the original corpus was identified as referring to additional bombs, using our pattern matching.  Based on a random sample of 236 tweets from the original corpus, our recall (identification of tweets that discussed additional bombs) was determined to be 50%.    That is, there were many more ways to refer to additional bombs than our rules considered. Thus, our F1 measure for accurately identifying tweets about additional bombs was 65%. Nevertheless, because of the volume of tweets, this did not affect the results appreciably.
  • 17. Location Phrase (Filtering) Evaluation  Location phrases were identified purely by means of generic pattern       matching. We did not use any list of known places. Nor did we include any scenario-specific patterns. The precision with which we identified location phrases was 95%. That is, in 95% of the cases, when we identified a phrase as a location phrase, it actually did refer to a location in that context. Mistakes included temporal references and references to online sites. Our recall was only 51.3% if we counted uses of #BostonMarathon that were locative. (We mishandled hashtags with camel case.) If we ignore this hashtag, then our recall was 79.2%.  That is, of all the locations mentioned in tweets about additional bombs at the Boston Marathon, we identified 79.2% percent of the locations that were mentioned.  Using the more lenient standard, our F1 measure for identifying location phrases in the text was 86.3%. 18
  • 18. GeoCoding Evaluation  Our precision in associating tweets with known places via the Google APIs was 97.2%.  Our precision in assigning unique location phrases to known places via Google APIs was 50%.  That is, there were many location phrases that were repeated several times that we assigned correctly to a known place, but half of the unique phrase names that we extracted were not assigned correctly.  Ten location phrases that were extracted corresponded to no known locations identified via the Google APIs.  These included location phrases such as “#jfklibrary” and “BPD Commissioner Ed Davis”. The former is a phrase we would like to geolocate, but lowercase hashtags which concatenate several words are challenging. The latter is the sort of phrase that we expect would be rejected as non-geolocatable. 19
  • 19. Top 20 Locations by Frequency Known Place JFK Library 1158 Boston 629 Boston Marathon 325 St Ignatius Catholic Church 47 PD 29 Boylston 8 CNN 5 Copley Sq 4 Huntington Ave 4 Iraq 3 Mandarin Hotel 3 Dorchester 3 Marathon 3 US Intelligence 3 Copley Place 2 Boston PD 2 BBC 2 Cambridge 2 John 2 St James Street #Boston 20 #Tweets 2
  • 20. Comparing Locations Mentioned in Media Blogs  For three of these sites – Mass. Location [Source]: (# of Tweets Identified with That Location Boylston Street [Globe, CNN]: 8 Commonwealth Ave near Centre Street, Newton [Globe]: 0 Commonwealth Ave (Boston) [Globe]: 0 Copley Square [NYT]: 4 Harvard MBTA station [Globe]: 0 JFK Library [CNN, Globe, NYT]: 1158 Mass. General Hospital [Globe, NYT]: 0 (glass footbridge over) Huntington Ave near Copley place [Globe]: 4 Tufts New England Medical Center [NYT]: 0 Washington Square, Brookline [NYT]: 0 21 General Hospital, Tufts Medical Center and Washington Square, Brookline -- reports of unexploded bombs or suspicious packages occurred after the end of the tweet collection period, at 7:06 PM.  Otherwise, the recall of our system was good, missing only the report of unexploded bombs at the Harvard MBTA station.  Media failed to report other locations prominent to us (e.g. St Ignatius Catholic Church)  In our corpus, but missed, due to capitalization.
  • 21. Qualitative Evaluation  On average, tweets reflecting same locations as media blogs were produced 11 minutes prior to their being reported on the sites mentioned.  Thus, the tweet processing was more timely and more comprehensive (included more locations) than simply relying on a handful of news sites alone for situational awareness 22
  • 22. Conclusion  We described a system for integrating disparate information sources into a      COP for Humanitarian Assistance/Disaster Relief operations by means of semantic annotations and queries, using a common ontology. We applied our technology to a repository of tweets collected in the immediate aftermath of the Boston Marathon bombings in April, 2013, and demonstrated that a ranked set of places could be incorporated into the COP, showing the prominence of each site by tweet volume that was reported as being the site of an additional unexploded bomb or bombs. We evaluated the results formally and compared the results with the situational awareness that could be gleaned only from mainstream media blogs being updated at the same time. On average, the automatic processing would have had access to locations from tweets eleven minutes before these sites were mentioned on the mainstream media blogs. Additionally, sites that were prominent on Twitter (e.g. St Ignatius Church at Boston College or the Mandarin Oriental Hotel in Boston) were not mentioned on the news blog sites at all. We believe that these results show that this approach is a promising one for deriving situational awareness from social media going forward. 23