Saurav Kumar developed two tools as part of an internship at LinkedIn to help identify spam text. The first tool classifies spam by tabulating scores from different classifiers. The second tool computes the contribution of each word towards the overall spam score, allowing users to see which specific words are causing a text to be classified as spam. It works by modifying words and reclassifying to measure score changes. The tools provide insights into classifier performance and help find spam text more efficiently.

1. presentation
Internship Report
Saurav Kumar, Software Engineering Intern
July 15, 2015
LinkedIn Bangalore

2. the problem

3. problem
Emails like these...
2

4. problem
Emails like these...
3

5. problem
There was no existing solution to find which specific part
of the text is causing a text to be classified as spam.
My task was to build a tool to solve this problem.
4

6. tool 1: spam classification tool

7. spam classification tool
∙ Given a content source, title and body of content, this
tool tabulates the scores of each classifier
6

8. spam classification tool
∙ Given a content source, title and body of content, this
tool tabulates the scores of each classifier
∙ Query request is sent to BAM, and the response
summary is presented in a table
6

9. spam classification tool
7

10. tool 2: spam token classification tool

11. spam token classification tool
∙ Given a content source, title and body of content, this
tool computes the contribution of each word (token)
towards the overall score
9

12. spam token classification tool
10

13. spam token classification tool
∙ Given a content source, title and body of content, this
tool computes the contribution of each word (token)
towards the overall score
∙ The UI allows you to set the number of tokens you
want to examine
11

14. method

15. method
∙ Suppose, we have a content with tokens t1, t2, ..., tn
13

16. method
∙ Suppose, we have a content with tokens t1, t2, ..., tn
∙ We need to find the effect of each token on the total
score
13

17. method
∙ Suppose, we have a content with tokens t1, t2, ..., tn
∙ We need to find the effect of each token on the total
score
∙ Let the score of this content be S0
13

18. method
∙ Suppose, we have a content with tokens t1, t2, ..., tn
∙ We need to find the effect of each token on the total
score
∙ Let the score of this content be S0
∙ Modify the ith
token to make it a non-word, and
obtain the score Si
13

19. method
∙ Suppose, we have a content with tokens t1, t2, ..., tn
∙ We need to find the effect of each token on the total
score
∙ Let the score of this content be S0
∙ Modify the ith
token to make it a non-word, and
obtain the score Si
∙ The difference wi = (S0 − Si) signifies the effect of ith
token on the score
13

20. method
Coloring
∙ Collect the score for each token and for each
classifier
14

21. method
Coloring
∙ Collect the score for each token and for each
classifier
∙ Normalize the scores for each classifier
14

22. method
Coloring
∙ Collect the score for each token and for each
classifier
∙ Normalize the scores for each classifier
∙ Color the top k1% good words with green and top k2%
bad words with red, with intensity proportional to
their scores.
14

23. demonstration

24. spam token classification tool
Escalation a few weeks ago
16

25. benefits

26. spam token classification tool
Benefits
∙ Saves time and effort in finding specific spam text
18

27. spam token classification tool
Benefits
∙ Saves time and effort in finding specific spam text
∙ More insights with token-wise scores
18

28. spam token classification tool
Benefits
∙ Saves time and effort in finding specific spam text
∙ More insights with token-wise scores
∙ Can be used to test performance of a classifier
18

29. spam token classification tool
Benefits
∙ Saves time and effort in finding specific spam text
∙ More insights with token-wise scores
∙ Can be used to test performance of a classifier
∙ Permalink of result is created, so links can be shared
18

30. spam token classification tool
Benefits
∙ Saves time and effort in finding specific spam text
∙ More insights with token-wise scores
∙ Can be used to test performance of a classifier
∙ Permalink of result is created, so links can be shared
∙ Content URN not required, so any text can be tested
18

31. spam token classification tool
Benefits
∙ Saves time and effort in finding specific spam text
∙ More insights with token-wise scores
∙ Can be used to test performance of a classifier
∙ Permalink of result is created, so links can be shared
∙ Content URN not required, so any text can be tested
∙ Method used is independent of classifier’s model
18

32. assumptions and limitations

33. spam token classification tool
Assumptions
∙ Scoring from classifier should be incremental, and
not 0-1
20

34. spam token classification tool
Assumptions
∙ Scoring from classifier should be incremental, and
not 0-1
∙ Same classifiers should run for all the requests: New
end-point in BAM ensures this
20

35. spam token classification tool
Limitations
∙ For such classifiers where total score is either 0 or 1,
this tool cannot extract any meaningful information
21

36. spam token classification tool
Limitations
∙ For such classifiers where total score is either 0 or 1,
this tool cannot extract any meaningful information
∙ For a large content, significant amount of time is
required
21

37. spam token classification tool
Figure: Measure of response time vs number of words
22

38. technologies used

39. technologies used
∙ Play Framework
24

40. technologies used
∙ Play Framework
∙ D2 (Dynamic Discovery) for making RestLi calls
24

41. technologies used
∙ Play Framework
∙ D2 (Dynamic Discovery) for making RestLi calls
∙ ParSeq for making parallel requests
24

42. technologies used
∙ Play Framework
∙ D2 (Dynamic Discovery) for making RestLi calls
∙ ParSeq for making parallel requests
∙ Stork for email
24

43. technologies used
∙ Play Framework
∙ D2 (Dynamic Discovery) for making RestLi calls
∙ ParSeq for making parallel requests
∙ Stork for email
∙ Couchbase to store responses
24

44. challenges

45. challenges
∙ Dealing with R2 (Request/Response) timeout
26

46. challenges
∙ Dealing with R2 (Request/Response) timeout
∙ Running an offline job after client may have closed
connection
26

47. Questions?
27

48. Thank You
Credits: Beamer(mtheme), ShareLaTeX
28