Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Subword and spatiotemporal models for identifying actionable information in Haitian Kreyol

436 views

Published on

Crisis-affected populations are often able to maintain digital communications but in a sudden-onset crisis any aid organizations will have the least free resources to process such communications. Information that aid agencies can actually act on, ‘actionable’ information, will be sparse so there is great potential to (semi)automatically identify actionable communications. However, there are hurdles as the languages spoken will often be underresourced, have orthographic variation, and the precise definition of ‘actionable’ will be response-specific and evolving.
We present a novel system that addresses this, drawing on 40,000 emergency text messages sent in Haiti following the January 12, 2010 earthquake, predominantly in Haitian Kreyol. We show that keyword/ngram-based models using streaming MaxEnt achieve up to F=0.21 accuracy. Further, we find current state-of-the-art subword models increase this substantially to F=0.33 accuracy, while modeling the spatial, temporal, topic and source contexts of the messages can increase this to a very accurate F=0.86 over direct text messages and F=0.90-0.97 over social media, making it a viable strategy for message prioritization.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Subword and spatiotemporal models for identifying actionable information in Haitian Kreyol

  1. 1. Subword and spatiotemporal models for identifying actionable information in Haitian Kreyol Robert Munro Stanford University CoNLL 2011 Munro, Robert. "Subword and spatiotemporal models for identifying actionable information in Haitian Kreyol." Proceedings of the Fifteenth Conference on Computational Natural Language Learning. Association for Computational Linguistics, 2011. http://www.robertmunro.com/research/munro11kreyol.pdf
  2. 2. January 12, 2010
  3. 3. Messages start streaming in
  4. 4. Messages start streaming in
  5. 5. 80,000 messages
  6. 6. 80,000 messages
  7. 7. 80,000 messages
  8. 8. 80,000 messages
  9. 9. 80,000 messages
  10. 10. 80,000 messages
  11. 11. Crowdsourced (Mission 4636)
  12. 12. Feedback  US Marines ◦ “Saving lives every day.”  FEMA: ◦ “The most comprehensive and up-to-date map available to the humanitarian community.”  World Food Program ◦ “We delivered food to an informal camp of 2500 people that you identified for us.”
  13. 13. Sudden-onset language processing
  14. 14. Sudden-onset language processing
  15. 15. Sudden-onset language processing
  16. 16. Sudden-onset language processing
  17. 17. Prioritization  Only 2% of messages were ‘actionable’ ◦ An identifiable location ◦ Medical, S+R, water, clustered food requests, security, unaccompanied children.  How can we prioritize the actionable items in the original Haitian Kreyol?  Can we leverage the models for
  18. 18. Evaluation data  Mission 4636. 40,811 text-messages sent to a free number, ‘4636’, in Haiti. (predominantly in Haitian Kreyol, with translations, UN-defined categories, and geolocation)  Radio Station. 7,528 text-messages sent to a Haitian radio station.  Twitter. 63,195 Haiti-related tweets.
  19. 19. Variation is the norm mesi mèsi mèci meci merci Kreyol French Abbrv. Full Form Pattern Meaning s’on se yon sVn is a av`en av`eknou VvVn with us relem rele mwen relem call me wap ouap uVp you are map mwen ap map I will be zanmimzanmi mwen zanmim my friend lavel lave li lavel to wash (it)
  20. 20. Evaluation data
  21. 21. Combining spatio-temporal models
  22. 22. Streaming architecture  Build from initial items time Model
  23. 23. Streaming architecture  Predict (and evaluate) on incoming items ◦ (penalty for training) time Model
  24. 24. Streaming architecture  Repeat / retrain time Model
  25. 25. Streaming architecture  Repeat / retrain time Model
  26. 26. Streaming architecture  Repeat / retrain time Model
  27. 27. Streaming architecture  Repeat / retrain time Model
  28. 28. Streaming architecture  Repeat / retrain time Model
  29. 29. Features  G : Words and ngrams  W : Subword patterns  P : Source of the message  T : Time received  C : Categories (c0,...,47)  L : Location (longitude and latitude)  L : Has-location (a location is written in the message)
  30. 30. Features  Subword models ◦ Full-forms and normalizations (Munro and Manning 2010) Abbrv. Full Form Pattern Meaning s’on se yon sVn is a av`en av`eknou VvVn with us relem rele mwen relem call me wap ouap uVp you are map mwen ap map I will be zanmimzanmi mwen zanmim my friend lavel lave li lavel to wash (it)
  31. 31. Time / Space / source  Timestamp (in place of discounting)  Phone-number of sender  Spatial tile-membership:
  32. 32. Additional streaming models  Message contains an identifiable location time Model
  33. 33. Additional streaming models  Message contains an identifiable location  Prediction for 47 categories time Model time Model time Model time Model …
  34. 34. Final streaming model  Prediction for ‘is actionable’ time Model
  35. 35. Final streaming model  Prediction for ‘is actionable’ time Model time Model time Model time Model … Combines features with predictions from Category and Has-Location models
  36. 36. Evaluation  100 training epochs ◦ Calculated on predictions over epochs 2- 100  Comparison of two-tier architecture with Oracle ‘has-location’ and ‘Categories’  Identification of actionable messages in Radio Station and Twitter messages  (full results in paper)
  37. 37. Feature-based improvements Subwords and Source (G, T,W, P) Temporal feature (G, T) Words/Ngrams (G) 0.326 0.252 0.207
  38. 38. Combined models All Features/Models Spatial clusters(L) Words/Ngrams (G) 0.855 0.756 0.207
  39. 39. Outperforming the oracle Location (two-teir prediction) Location (oracle) Words/Ngrams (G) 0.310 0.274 0.207
  40. 40. Negative results from filtering Oracle true-negfiltering All Features/Models Words/Ngrams (G) 0.428 0.855 0.207
  41. 41. Social media Twitter Radio station All Features/Models 0.969 0.904 0.855
  42. 42. Conclusions (usability)  Subword and spatio-temporal models can give a 10-fold increase in prioritization  Adding multi-tiered streaming models can give a 50-fold increase in prioritization  Cross-domain adaptation is possible for need(le)-in-haystack information extraction from social media
  43. 43. Questions?
  44. 44. Appendix: abstract Crisis-affected populations are often able to maintain digital communications but in a sudden-onset crisis any aid organizations will have the least free resources to process such communications. Information that aid agencies can actually act on, ‘actionable’ information, will be sparse so there is great potential to (semi)automatically identify actionable communications. However, there are hurdles as the languages spoken will often be underresourced, have orthographic variation, and the precise definition of ‘actionable’ will be response-specific and evolving. We present a novel system that addresses this, drawing on 40,000 emergency text messages sent in Haiti following the January 12, 2010 earthquake, predominantly in Haitian Kreyol. We show that keyword/ngram-based models using streaming MaxEnt achieve up to F=0.21 accuracy. Further, we find current state-of-the-art subword models increase this substantially to F=0.33 accuracy, while modeling the spatial, temporal, topic and source contexts of the messages can increase this to a very accurate F=0.86 over direct text messages and F=0.90-0.97 over social media, making it a viable strategy for message prioritization.

×