0
SWIFT WEB SERVICES
   Verifying and Filtering the Crowd




             An Ushahidi Initiative

        by Neville Newey ...
SWIFT IS THE FILTER
SWIFTRIVER IS FOR...
Improving information findability
Surfacing content you didn't know you were looking for
Understanding...
SwiftRiver Web Services
•   SiLCC - NLP for SMS and Twitter
•   SULSa - Location Services
•   SiCDS - Duplication Filterin...
RIVER ID
Distributed Trust and Reputation




           Web Services
REVERBERATIONS
  Measuring Content Influence




           Web Services
SILCC
SwiftRiver Language Computation Core




              Web Services
WHAT IS SILCC?
•Swift Language Computation Component
•One of the SwiftRiver Web Services
•Open Web API
•Semantic Tagging o...
Swiftriver     SiLCC  Dataflow  
       
                          SiSLS  
                                               ...
OUR GOALS
•Simple Tagging of short snippets of text
•Rapid tagging for high volume environments
•Simple API, easy to use
•...
WHAT IT’S NOT
•Does not do deep analysis of text
•Only identifies words within original text
HOW DOES IT WORK?
•Step 1: Lexical Analysis
•Step 2: Parsing into constituent parts
•Step 3: Part of Speech tagging
•Step ...
STEP 1: LEXICAL ANALYSIS
•For news headlines, email subjects this is trivial, just
 split on spaces.

•For Twitter this is...
TWEET ANALYSIS
•Tweets are surprisingly complex
•Only 140 characters but many features
•Emergent features from community (...
TWEET ANALYSIS
 The typical Tweet: “RT @directrelief: RT
 @PIH: PBS @NewsHour addresses mental health
 needs in the afterm...
TWEET ANALYSIS 2
•Two or more hashtags within a tweet (e.g.
 #health and #earthquake)
•Continuation dots “...” indicates t...
TWEET ANALYSIS 3
We want to break up the tweet into the following
parts:
{

  'text': ['PBS addresses mental health needs ...
TWEET ANALYSIS 4
 Why do we want to break up the tweet into parts
 (parsing)?

•Because we want to further process the
 gr...
TWEET ANALYSIS 5
•We now take the “text” portion of the tweet and
 perform part of speech tagging on it
•After part of spe...
HEADLINE AND EMAIL
    SUBJECT ANALYSIS
•This is much simpler to do
•Its a subset of the steps in Tweet Analysis
•There is...
FEATURE EXTRACTION
• For the active learning algorithm we need to extract features to use in classification
• These feature...
FEATURE EXTRACTION
• Capitalization of individual words: Either first caps, or all caps, this is an
    important indicator...
TRAINING
• Requires user reviewed examples
• Lexical analysis, parsing and feature extraction on the examples
• Multinomia...
ACTIVE LEARNING
•The API also provides a method for users to send
 back corrected text
•The corrected text is saved and th...
DEVELOPER FRIENDLY
•Two levels of API, the web API and the internal
 Python API
•Either one may be used but most users wil...
PYTHON CLASSES
Most of the classes that make up the library are
divided into three types:

   1) Tokenizers
   2) Parsers
...
PYTHON API
•A tagger calls a parser
•A parser calls a tokenizer
•Output of the tokenizer goes into the parser
•Output of t...
CLASSES
• BasicTokenizer – This is used for splitting basic (non-tweet) text into individual
    words
•   TweetTokenizer ...
DEPENDANCIES
•Part of speech tagging is currently performed by the
 Python NLTK
•The Web API uses the Pylons web framework
CURRENT STATUS
•Tag method of API is ready for use, individual
 deployments can choose between using the
 BasicTagger or t...
CURRENT LIMITATIONS
•Only English text is supported at the moment
•Tags are always one of the words in the supplied
 text ...
FUTURE WORK
•Multilingual, use non-english part of speech taggers
•UTF8 compatible
•Experiment with different learning alg...
SWIFT RIVER
       jon@ushahidi.com
   http://swift.ushahidi.com
http://github.com/appfrica/silcc




         An Ushahidi...
Swift Web Services Overiview
Swift Web Services Overiview
Swift Web Services Overiview
Swift Web Services Overiview
Swift Web Services Overiview
Upcoming SlideShare
Loading in...5
×

Swift Web Services Overiview

2,211

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,211
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
32
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Swift Web Services Overiview"

  1. 1. SWIFT WEB SERVICES Verifying and Filtering the Crowd An Ushahidi Initiative by Neville Newey and Jon Gosier
  2. 2. SWIFT IS THE FILTER
  3. 3. SWIFTRIVER IS FOR... Improving information findability Surfacing content you didn't know you were looking for Understanding media from other parts of the world (translation) Making urgent data more discoverable (structured, published and accessible) Verifying eyewitness accounts Using location as context Expanding the grassroots reporting network Preserving information (archiving)
  4. 4. SwiftRiver Web Services • SiLCC - NLP for SMS and Twitter • SULSa - Location Services • SiCDS - Duplication Filtering • River ID - Distributed Reputation • Reverberations - Measures influence of online content
  5. 5. RIVER ID Distributed Trust and Reputation Web Services
  6. 6. REVERBERATIONS Measuring Content Influence Web Services
  7. 7. SILCC SwiftRiver Language Computation Core Web Services
  8. 8. WHAT IS SILCC? •Swift Language Computation Component •One of the SwiftRiver Web Services •Open Web API •Semantic Tagging of Short Text •Multilingual •Multiple sources (twitter, email, SMS, blogs etc) •Active Learning capability •Open Source •Easy to Deploy, Modify and Run
  9. 9. Swiftriver    SiLCC  Dataflow     SiSLS   Content   Items   coming   from   the   SiSLS   have     where   Swiftriver  Source     SiSLS   integrations   is   enabled     global   trust   values   Library  Service   added  to  the  object  model.     SiLCC   Swiftriver  Language   An  API  key  is  sent  along  with  the  text  to  ensure  that   the  SiLCC  is  not  open  to  any  malicious  usage.     Computational  Core     The  text  of  the   content  is  sent  to  the   SiLCC.   There  is  still  a  bit  of  ambiguity  around  what  the  NLP   should  extract  from  the  text  but  at  its  most  simple,   Using  NLP,  the  SiLCC   all  the  nouns  would  be  a  good  start.   extracts  Nouns  and   other  keywords  from   the  text.   The  SiLCC  send  back   The   lists   of   tags   sent   back   from   the   SiLCC   can   be   a  list  of  tags  that  are   added  to  the  Content  Item  along  with  any  that  were   added  to  the   extracted  from  the  source  data  by  the  parser.   Content  Item   SLISa   Although   the   NLP   tags   have   now   been   applied,   the   SLISa   is   now   responsible   for   applying   instance   Swiftriver  Language   specific  tagging  corrections.   Improvement  Service    
  10. 10. OUR GOALS •Simple Tagging of short snippets of text •Rapid tagging for high volume environments •Simple API, easy to use •Learns from user feedback •Routing of messages to upstream services •Semantic Classification •Sorts rapid streams into buckets •Clusters like messages •Visual effects •Cross-referencing
  11. 11. WHAT IT’S NOT •Does not do deep analysis of text •Only identifies words within original text
  12. 12. HOW DOES IT WORK? •Step 1: Lexical Analysis •Step 2: Parsing into constituent parts •Step 3: Part of Speech tagging •Step 4: Feature extraction •Step 5: Compute using feature weights •Lets examine each one in turn...
  13. 13. STEP 1: LEXICAL ANALYSIS •For news headlines, email subjects this is trivial, just split on spaces. •For Twitter this is more complex...
  14. 14. TWEET ANALYSIS •Tweets are surprisingly complex •Only 140 characters but many features •Emergent features from community (e.g. hashtags) •Lets take a look at a typical tweet...
  15. 15. TWEET ANALYSIS The typical Tweet: “RT @directrelief: RT @PIH: PBS @NewsHour addresses mental health needs in the aftermath of the #Haiti earthquake #health #earthquake... http://bit.ly/bNhyK6” •RT indicates a “re-tweet” •@name indicates who the original tweeter was •Multiple embedded retweets •Hashtags (e.g. #Haiti) can play two roles, as a tag and as part of the sentence
  16. 16. TWEET ANALYSIS 2 •Two or more hashtags within a tweet (e.g. #health and #earthquake) •Continuation dots “...” indicates that there was more text that didn’t fit into the 140 limit somewhere in it’s history •Urls many tweets contain one or more urls As we can see this simple tweet contains no less than 7 different features and that’s not all!
  17. 17. TWEET ANALYSIS 3 We want to break up the tweet into the following parts: { 'text': ['PBS addresses mental health needs in the aftermath of the Haiti earthquake'], 'hashtags': ['#Haiti', '#health', '#earthquake'], 'names': ['@directrelief', '@PIH', '@NewsHour'], 'urls': ['http://bit.ly/bNhyK6'], }
  18. 18. TWEET ANALYSIS 4 Why do we want to break up the tweet into parts (parsing)? •Because we want to further process the grammatically correct english text •Part of speech tagging would otherwise be corrupted by words it cannot recognize (e.g. urls, hashtags, @names etc.) •We want to save the hashtags for later use •Many of the features are irrelevant to the task of identifying tags (e.g. dots, punctuation, @name, RT)
  19. 19. TWEET ANALYSIS 5 •We now take the “text” portion of the tweet and perform part of speech tagging on it •After part of speech tagging, we perform feature extraction •Features are now passed through the keyword classifier which returns a list of keywords / tags •Finally we combine these tags with the hashtags we saved earlier to give the complete tag set
  20. 20. HEADLINE AND EMAIL SUBJECT ANALYSIS •This is much simpler to do •Its a subset of the steps in Tweet Analysis •There is no parsing since there are no hashtags, @names etc.
  21. 21. FEATURE EXTRACTION • For the active learning algorithm we need to extract features to use in classification • These features should be subject/domain independent • We therefore never use the actual words as features • This would for example give artificially high weights to words such as “earthquake” • We don't want these artificial weights as we can’t foresee future disasters and we want to be as generic with classification as possible • The use of training sets does allow for domain customization if where necessary
  22. 22. FEATURE EXTRACTION • Capitalization of individual words: Either first caps, or all caps, this is an important indicator of proper nouns or other important words that make good tag candidates • Position in text: Tags seem to have a greater preponderance near the beginning of text • Part of Speech: Nouns and proper nouns are particularly important but so are some adjectives and adverbs • Capitalization of entire text: sometimes the whole text is capitalized and this should reduce overall weighting of other features • Length of the text: In shorter texts the words are more likely to be tags • The parts of speech of previous and next words (effectively this means we are using trigrams; or a window of 3)
  23. 23. TRAINING • Requires user reviewed examples • Lexical analysis, parsing and feature extraction on the examples • Multinomial naïve Bayes algorithm • NB: The granularity we are classifying is at the word level • For each word in the text, we classify it as either a keyword or not • This has pleasant side effect of providing several training examples from each user reviewed text • Even with less than 50 reviewed texts the results are comparable to the simple approach of using nouns only
  24. 24. ACTIVE LEARNING •The API also provides a method for users to send back corrected text •The corrected text is saved and then used in the next iteration of training •User may optionally specify a corpus for the example to go into •Training can be performed using any combination of corpora
  25. 25. DEVELOPER FRIENDLY •Two levels of API, the web API and the internal Python API •Either one may be used but most users will use the web API •Design is highly modular and maintainable •For very rapid backend processing the native Python API can be used
  26. 26. PYTHON CLASSES Most of the classes that make up the library are divided into three types: 1) Tokenizers 2) Parsers 3) Taggers All three types have consistent API's and are interchangeable.
  27. 27. PYTHON API •A tagger calls a parser •A parser calls a tokenizer •Output of the tokenizer goes into the parser •Output of the parser goes into the tagger •Output of the tagger goes into the user!
  28. 28. CLASSES • BasicTokenizer – This is used for splitting basic (non-tweet) text into individual words • TweetTokenizer – This is used to tokenize a tweet, it may also be used to tokenize plain text since plain text is a subset of tweets • TweetParser – Calls the TweetTokenizer and the parses the output (see previous example) • TweetTagger – Calls the TweetTokenizer and then tags the output of the text part and adds the hashtags • BasicTagger – Calls the BasicTokenizer and then tags the text, should only be used for non-tweet text, uses simple Part of Speech to identify tags • BayesTagger – Same as BasicTagger but uses weights from the naïve Bayes training algorithm
  29. 29. DEPENDANCIES •Part of speech tagging is currently performed by the Python NLTK •The Web API uses the Pylons web framework
  30. 30. CURRENT STATUS •Tag method of API is ready for use, individual deployments can choose between using the BasicTagger or the BayesTagger •Tell method (for user feedback) will be ready by the time you read this! •Training is possible on corpora of tagged data in .csv format (see examples in distribution)
  31. 31. CURRENT LIMITATIONS •Only English text is supported at the moment •Tags are always one of the words in the supplied text ie they can never be a word not in the supplied text •Very few training examples exist at the moment
  32. 32. FUTURE WORK •Multilingual, use non-english part of speech taggers •UTF8 compatible •Experiment with different learning algorithms (e.g. neural networks) •Perform external text analysis (e.g. if there is a url, analyze the text in the url as well as in the tweet) •Allow users to specify required density of tags
  33. 33. SWIFT RIVER jon@ushahidi.com http://swift.ushahidi.com http://github.com/appfrica/silcc An Ushahidi Initiative by Neville Newey and Jon Gosier
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×