Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Development  Emails  Content  Analyzer:  
Intention  Mining  in  Developer  Discussions	
  Andrea	
Di  Sorbo	
  Sebastiano...
Outline  
	
Context:  	
Wri5en  	
Development  Discussions	
	
Case  Study:  	
Development  Mailing  List	
of  2  Open  Sou...
Open  Source  (OS)  and    
Industrial  Projects  	
3
Open  Source  (OS)  and    
Industrial  Projects	
4
Open  Source  (OS)  and    
Industrial  Projects	
5
Open  Source  (OS)  and    
Industrial  Projects	
6
Development    
Communication  Means	
Recommender  systems:	
-­‐‑  Bug  Triaging  [1]	
-­‐‑  Suggest  Mentors  [2]	
-­‐‑  ...
Development    
Communication  Means	
8
Development    
Communication  Means	
[1]  Bacchelli  et  al.  “Content  classification  of  development  emails”.	
[2]  Ce...
Different  Kinds  of  Data  
	
Structured	
Semi-­‐‑Structured	
Unstructured	
10
A  Considerable  Effort  for  
Developers	
Many  messages  	
Developers  get  lost  in  unnecessary  details  
missing  pot...
Previous  Work  
	
12
Hana  et  al.	
“…Lazy”  RTC  occurs  when  
a  core  developer  post  a  
change  to  a  mailing  li...
Previous  Work  
	
Approaches  for:  	
-­‐‑  Generating  summaries  	
        of  emails.  	
            à  Lam  et  al. ...
Different  Purposes  
	
Feature  requests	
Bug  disclosures	
Project  Management	
14
DECA  
(Development  Email  Content  Analyzer)	
An  approach  to  Classify  Paragraphs  	
According  to  Intentions	
hSp:/...
Why  use  NLP  for  Classifying  
Paragraphs  According  to  
Intentions?	
16
Example	
i.  We  could  use  a  leaky  bucket  algorithm  to  limit  
the  bandwidth	
ii.  The  leaky  bucket  algorithm  ...
i.  We  could  use  a  leaky  bucket  algorithm  to  limit  
the  bandwidth	
ii.  The  leaky  bucket  algorithm  fails  in...
i.  We  could  use  a  leaky  bucket  algorithm  to  limit  
the  bandwidth	
ii.  The  leaky  bucket  algorithm  fails  in...
i.  We  could  use  a  leaky  bucket  algorithm  to  limit  
the  bandwidth	
ii.  The  leaky  bucket  algorithm  fails  in...
i.  We  could  use  a  leaky  bucket  algorithm  to  limit  
the  bandwidth	
ii.  The  leaky  bucket  algorithm  fails  in...
Perspective  
	
22
Goal:  Understanding  to  what  extent  NL  parsing  could  be  
used     in  recognizing  informative  text  fragments  i...
Research  Questions  
	
RQ1:   Can   an   NLP   approach   (i.e.   DECA)   be  
effective   in   classifying   writers’   i...
Qt	
Ubuntu	
Context	
25
STEPS:	
1)    Taxonomy  Definition	
  	
  2)    Classification  Based  on  DECA  (NLP  Analyzer)	
26
Taxonomy Definition
27
Sampling  	
	
We  selected  100	
Of  the                            Project      	
28
Clustering	
	
Clusters	
Implementation	
Technical  Infrastructure	
Project  Status	
Social  Interations	
Usage	
Discarded	...
Clustering	
	
Guzzi  et.  al  –  ICSE2012	
30
The  final  taxonomy	
	
31
Differences  with  Guzzi  et.  al.	
	
32
Examples	
	
33
Natural Language
Parsing
DECA  
(Development  Email  Content  Analyzer)	
34
Recurrent  Linguistic  PaSerns	
	
35
Why  NL  parsing?  
	
Well  defined  predicate-­‐‑argument  structures	
use	
we	
 could	
 algorithm	
a	
 leaky	
 bucket	
li...
NL  parsing	
Natural  Language  Templates	
use	
[someone]	
 could	
 [something]	
                          nsubj          ...
Natural  Language  Templates	
use	
[someone]	
 could	
 [something]	
                          nsubj                    aux...
Natural  Language  Templates	
use	
[someone]	
 could	
 [something]	
                          nsubj                    aux...
NLP  Heuristics	
	
40
NLP  Parser	
	
raw  text	
 NLP  parser	
 NLP  heuristics	
41
42
43
RQ1:  	
Is  DECA  effective  in  	
classifying  writers’  intentions  in  
development  emails?	
	
44
Experiment  I	
	
training	
test	
102 87
100
45
Experiment  I	
	
training	
test	
102 87
100
Experiment  II	
False  
Negative	
 46
Experiment  II	
	
training	
100 169
test	
100
Experiment  III	
False  
Negative	
 47
Experiment  III	
	
training	
100 231
test	
100
48
49
50
51
52
53
54
RQ2:  	
Is  the  proposed  approach  more  
effective  than  existing  ML  in  classifying  
development  emails  content?	...
ML  for  Email  Classification	
	
An  Approach  Based  on  ML  for  Email  Content  Classification	
            à  Antoniol...
ML  for  Email  Classification	
	
An  Approach  Based  on  ML  for  Email  Content  Classification	
1)Text  Features	
	
57
ML  for  Email  Classification	
	
An  Approach  Based  on  ML  for  Email  Content  Classification	
1)Text  Features	
	
2)  ...
ML  for  Email  Classification	
	
An  Approach  Based  on  ML  for  Email  Content  Classification	
1)Text  Features	
	
2)  ...
ML  for  Email  Classification	
	
An  Approach  Based  on  ML  for  Email  Content  Classification	
1)Text  Features	
	
2)  ...
61
62
63
64
65
66
67
68
69
Summary	
	
•  RQ2:   DECA outperforms traditional ML techniques in
terms of recall, precision and F-Measure when
classifyi...
Summary	
	
•  RQ2:   DECA outperforms traditional ML techniques in
terms of recall, precision and F-Measure when
classifyi...
72
Code  e-­‐‑documentation	
	
àPanichella  et.  al.  –  ICPC  2012  
Extract  methods’  descriptions  from  
developers  di...
Code  re-­‐‑documentation	
	
 “…  several  are  the  discourse  
paIerns  that  characterize  false  
negative  method  de...
Code  re-­‐‑documentation	
	
 “…  several  are  the  discourse  
paIerns  that  characterize  false  
negative  method  de...
Code  re-­‐‑documentation	
	
 “…  several  are  the  discourse  
paIerns  that  characterize  false  
negative  method  de...
Code  re-­‐‑documentation	
	
 “…  several  are  the  discourse  
paIerns  that  characterize  false  
negative  method  de...
Code  re-­‐‑documentation	
	
 “…  several  are  the  discourse  
paIerns  that  characterize  false  
negative  method  de...
Code  re-­‐‑documentation	
	
 “…  several  are  the  discourse  
paIerns  that  characterize  false  
negative  method  de...
Code  re-­‐‑documentation	
	
delete	
  
80
Conclusion	
	
81
Conclusion	
	
82
Conclusion	
	
83
Conclusion	
	
84
Conclusion	
	
85
Conclusion	
	
86
Future  work	
	
1)DECA  as  preprocessing  
support  to  discard  irrelevant  
sentences  in  summarization  
approaches	
...
Future  work	
	
1)DECA  as  preprocessing  
support  to  discard  irrelevant  
sentences  in  summarization  
approaches	
...
Upcoming SlideShare
Loading in …5
×

Development Emails Content Analyzer: Intention Mining in Developer Discussions

967 views

Published on

Written development communication (e.g. mailing lists, issue trackers) constitutes a precious source of information to build recommenders for software engineers, for example aimed at suggesting experts, or at redocumenting existing source code. In this paper we propose a novel, semi-supervised approach
named DECA (Development Emails Content Analyzer) that uses Natural Language Parsing to classify the content of development emails according to their purpose (e.g. feature request, opinion asking, problem discovery, solution proposal, information giving etc), identifying email elements that can be used for specific tasks.
A study based on data from Qt and Ubuntu, highlights a high precision (90%) and recall (70%) of DECA in classifying email content, outperforming traditional machine learning strategies. Moreover, we successfully used DECA for re-documenting sourcecode of Eclipse and Lucene, improving the recall, while keeping high precision, of a previous approach based on ad-hoc heuristics.

  • Be the first to comment

  • Be the first to like this

Development Emails Content Analyzer: Intention Mining in Developer Discussions

  1. 1. Development  Emails  Content  Analyzer:   Intention  Mining  in  Developer  Discussions  Andrea Di  Sorbo  Sebastiano Panichella  Corrado Visaggio  Massimiliano Di  Penta  Gerardo Canfora  Harald Gall
  2. 2. Outline   Context:   Wri5en   Development  Discussions Case  Study:   Development  Mailing  List of  2  Open  Source  Projects Results: Automatic  Classification  of  Relevant Contents  in  Developers’  Communication 2
  3. 3. Open  Source  (OS)  and     Industrial  Projects   3
  4. 4. Open  Source  (OS)  and     Industrial  Projects 4
  5. 5. Open  Source  (OS)  and     Industrial  Projects 5
  6. 6. Open  Source  (OS)  and     Industrial  Projects 6
  7. 7. Development     Communication  Means Recommender  systems: -­‐‑  Bug  Triaging  [1] -­‐‑  Suggest  Mentors  [2] -­‐‑  Code  re-­‐‑documentation  [3] -­‐‑  Etc. [1]  Anvik  et  al.  “Who  should  fix  this  bug?”. [2]  Canfora  et  al.  “Who  is  going  to  mentor  newcomers  in  open  source  projects?”   [3]  Panichella  et  al.  “Mining  source  code  descriptions  from  developer  communications” 7
  8. 8. Development     Communication  Means 8
  9. 9. Development     Communication  Means [1]  Bacchelli  et  al.  “Content  classification  of  development  emails”. [2]  Cerulo  et  al.  “A  Hidden  Markov  Model  to  detect  coded  information  islands  in  free  text.”   9
  10. 10. Different  Kinds  of  Data   Structured Semi-­‐‑Structured Unstructured 10
  11. 11. A  Considerable  Effort  for   Developers Many  messages   Developers  get  lost  in  unnecessary  details   missing  potential  useful  information… 11
  12. 12. Previous  Work   12 Hana  et  al. “…Lazy”  RTC  occurs  when   a  core  developer  post  a   change  to  a  mailing  lists   and  nobody  responds,  it  assumed  that  other   developers  reviewed  the   code…”
  13. 13. Previous  Work   Approaches  for:   -­‐‑  Generating  summaries          of  emails.              à  Lam  et  al.  ,              à  Rambow  et  al. -­‐‑  Generating  summaries            of  bug  reports.            à    Rastkar  et  al. 13
  14. 14. Different  Purposes   Feature  requests Bug  disclosures Project  Management 14
  15. 15. DECA   (Development  Email  Content  Analyzer) An  approach  to  Classify  Paragraphs   According  to  Intentions hSp://www.ifi.uzh.ch/seal/people/panichella/tools/DECA.html 15
  16. 16. Why  use  NLP  for  Classifying   Paragraphs  According  to   Intentions? 16
  17. 17. Example i.  We  could  use  a  leaky  bucket  algorithm  to  limit   the  bandwidth ii.  The  leaky  bucket  algorithm  fails  in  limiting  the   bandwidth   17
  18. 18. i.  We  could  use  a  leaky  bucket  algorithm  to  limit   the  bandwidth ii.  The  leaky  bucket  algorithm  fails  in  limiting  the   bandwidth        An  high  percentage  of  words  in  common Example 18
  19. 19. i.  We  could  use  a  leaky  bucket  algorithm  to  limit   the  bandwidth ii.  The  leaky  bucket  algorithm  fails  in  limiting  the   bandwidth   Discuss  about  the  same  topics Example 19
  20. 20. i.  We  could  use  a  leaky  bucket  algorithm  to  limit   the  bandwidth ii.  The  leaky  bucket  algorithm  fails  in  limiting  the   bandwidth   Have  different  intentions Example 20
  21. 21. i.  We  could  use  a  leaky  bucket  algorithm  to  limit   the  bandwidth ii.  The  leaky  bucket  algorithm  fails  in  limiting  the   bandwidth   Have  different  intentions Example “Techniques  based  on  lexicon  analysis,  such  as  VSM  [1],  LSI  [2],  or  LDA  [3]  would   not  be  sufficient  to  classify  paragraphs  according  to  intentions”. . [1]  Baeza-­‐‑Yates  et  al.  “Modern  Information  Retrieval”. [2]  de  Marneffe  et  al.,  “The  Stanford  typed  dependencies  representation”. [3]  Blei  et  al.,  “Latent  dirichlet  allocation”. 21
  22. 22. Perspective   22
  23. 23. Goal:  Understanding  to  what  extent  NL  parsing  could  be   used    in  recognizing  informative  text  fragments  in  emails   from  a  software  maintenance  and  evolution  perspective Quality   focus:   Detection   of   text   paragraphs   in   development   discussions     containing   helpful   information   for  developers.   Perspective:   Guide   developers   in   maintaining   and   evolving  their  products.   Case  Study   23
  24. 24. Research  Questions   RQ1:   Can   an   NLP   approach   (i.e.   DECA)   be   effective   in   classifying   writers’   intentions   in   development  emails? RQ2:  Is  DECA  more  effective  than  existing   Machine   Learning   techniques   in   classifying  development  emails  content? 24
  25. 25. Qt Ubuntu Context 25
  26. 26. STEPS: 1)    Taxonomy  Definition    2)    Classification  Based  on  DECA  (NLP  Analyzer) 26
  27. 27. Taxonomy Definition 27
  28. 28. Sampling   We  selected  100 Of  the                            Project       28
  29. 29. Clustering Clusters Implementation Technical  Infrastructure Project  Status Social  Interations Usage Discarded Guzzi  et.  al  –  MSR2013 29
  30. 30. Clustering Guzzi  et.  al  –  ICSE2012 30
  31. 31. The  final  taxonomy 31
  32. 32. Differences  with  Guzzi  et.  al. 32
  33. 33. Examples 33
  34. 34. Natural Language Parsing DECA   (Development  Email  Content  Analyzer) 34
  35. 35. Recurrent  Linguistic  PaSerns 35
  36. 36. Why  NL  parsing?   Well  defined  predicate-­‐‑argument  structures use we could algorithm a leaky bucket limit to bandwidth the            nsubj                    aux                    dobj                        xcomp            det                  amod                    nn                      aux                        dobj             det fails algorithm the leaky bucket in limiting bandwidth the                                    nsubj                                                          prep                det                amod          nn                    pcomp                  dobj                  det   36
  37. 37. NL  parsing Natural  Language  Templates use [someone] could [something]                          nsubj                    aux                    dobj fails [somehing] nsubj 37
  38. 38. Natural  Language  Templates use [someone] could [something]                          nsubj                    aux                    dobj fails [somehing] nsubj NL  parsing 38
  39. 39. Natural  Language  Templates use [someone] could [something]                          nsubj                    aux                    dobj fails [somehing] nsubj NL  parsing 39
  40. 40. NLP  Heuristics 40
  41. 41. NLP  Parser raw  text NLP  parser NLP  heuristics 41
  42. 42. 42
  43. 43. 43
  44. 44. RQ1:   Is  DECA  effective  in   classifying  writers’  intentions  in   development  emails? 44
  45. 45. Experiment  I training test 102 87 100 45
  46. 46. Experiment  I training test 102 87 100 Experiment  II False   Negative 46
  47. 47. Experiment  II training 100 169 test 100 Experiment  III False   Negative 47
  48. 48. Experiment  III training 100 231 test 100 48
  49. 49. 49
  50. 50. 50
  51. 51. 51
  52. 52. 52
  53. 53. 53
  54. 54. 54
  55. 55. RQ2:   Is  the  proposed  approach  more   effective  than  existing  ML  in  classifying   development  emails  content? 55
  56. 56. ML  for  Email  Classification An  Approach  Based  on  ML  for  Email  Content  Classification            à  Antoniol  et.  al.,  CASCON  2008                à  Zhou  et  al.  ,  ICSME  2014 56
  57. 57. ML  for  Email  Classification An  Approach  Based  on  ML  for  Email  Content  Classification 1)Text  Features 57
  58. 58. ML  for  Email  Classification An  Approach  Based  on  ML  for  Email  Content  Classification 1)Text  Features 2)  Split  training   and  test  sets 58
  59. 59. ML  for  Email  Classification An  Approach  Based  on  ML  for  Email  Content  Classification 1)Text  Features 2)  Split  training   and  test  sets 3)  Oracle   building 59
  60. 60. ML  for  Email  Classification An  Approach  Based  on  ML  for  Email  Content  Classification 1)Text  Features 2)  Split  training   and  test  sets 3)  Oracle   building 4)  Classification training prediction            à  Antoniol  et.  al.,  CASCON  2008                à  Zhou  et  al.  ,  ICSME  2014 60
  61. 61. 61
  62. 62. 62
  63. 63. 63
  64. 64. 64
  65. 65. 65
  66. 66. 66
  67. 67. 67
  68. 68. 68
  69. 69. 69
  70. 70. Summary •  RQ2:   DECA outperforms traditional ML techniques in terms of recall, precision and F-Measure when classifying e-mail content. •  RQ1:   the automatic classification performed by DECA achieves very good results in terms of both precision, recall and F-measure (over all the experiments). 70
  71. 71. Summary •  RQ2:   DECA outperforms traditional ML techniques in terms of recall, precision and F-Measure when classifying e-mail content. ”…it took the MSR community more than 10 years to figure out that machine learning is not the best method for analyzing human-written text. Thank you for helping move the field forward…”  [One of the ASE Reviewers] •  RQ1:   the automatic classification performed by DECA achieves very good results in terms of both precision, recall and F-measure (over all the experiments). 71
  72. 72. 72
  73. 73. Code  e-­‐‑documentation àPanichella  et.  al.  –  ICPC  2012   Extract  methods’  descriptions  from   developers  discussions à  Vector  Space  Models à  ad  hoc  heuristics “…  several  are  the  discourse   paIerns  that  characterize  false   negative  method  descriptions…  “ 73
  74. 74. Code  re-­‐‑documentation “…  several  are  the  discourse   paIerns  that  characterize  false   negative  method  descriptions…  “ 74
  75. 75. Code  re-­‐‑documentation “…  several  are  the  discourse   paIerns  that  characterize  false   negative  method  descriptions…  “ 75
  76. 76. Code  re-­‐‑documentation “…  several  are  the  discourse   paIerns  that  characterize  false   negative  method  descriptions…  “ 76
  77. 77. Code  re-­‐‑documentation “…  several  are  the  discourse   paIerns  that  characterize  false   negative  method  descriptions…  “ 77
  78. 78. Code  re-­‐‑documentation “…  several  are  the  discourse   paIerns  that  characterize  false   negative  method  descriptions…  “ 78
  79. 79. Code  re-­‐‑documentation “…  several  are  the  discourse   paIerns  that  characterize  false   negative  method  descriptions…  “ 79
  80. 80. Code  re-­‐‑documentation delete   80
  81. 81. Conclusion 81
  82. 82. Conclusion 82
  83. 83. Conclusion 83
  84. 84. Conclusion 84
  85. 85. Conclusion 85
  86. 86. Conclusion 86
  87. 87. Future  work 1)DECA  as  preprocessing   support  to  discard  irrelevant   sentences  in  summarization   approaches 87
  88. 88. Future  work 1)DECA  as  preprocessing   support  to  discard  irrelevant   sentences  in  summarization   approaches 2)DECA  in  combination  with   topic  models  for  mining   contents  with  the  same  intentions   and  the  same  topics   88

×