Too Long; Didn’t Watch! Extracting Relevant Fragments from Software Development Video Tutorials @ ICSE 2016, Austin, TX, USA

1. Luca Ponzanelli
Andrea Mocci
Michele Lanza
Sonia Haiduc
Mir Hasan
Gabriele Bavota
Barbara Russo
Max Di Penta Rocco Oliveto
Florida State
University
University of
Molise
University of
Sannio
Free University of
Bozen-Bolzano
Università della
Svizzera italiana
Too Long;
Didn't Watch!
Extracting Relevant Fragments from
Software Development Video Tutorials

4. L. MacLeod, M. A. Storey and A. Bergen
Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube
In Proceedings of ICPC 2015 (23rd International Conference on Program Comprehension)
YouTube
and Video Tutorials

5. L. MacLeod, M. A. Storey and A. Bergen
Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube
In Proceedings of ICPC 2015 (23rd International Conference on Program Comprehension)
Sharing Customization Knowledge
YouTube
and Video Tutorials

6. L. MacLeod, M. A. Storey and A. Bergen
Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube
In Proceedings of ICPC 2015 (23rd International Conference on Program Comprehension)
Sharing Customization Knowledge
Sharing Development Exercise
YouTube
and Video Tutorials

7. L. MacLeod, M. A. Storey and A. Bergen
Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube
In Proceedings of ICPC 2015 (23rd International Conference on Program Comprehension)
Sharing Customization Knowledge
Sharing Development Exercise
Sharing Implementation Approaches
YouTube
and Video Tutorials

8. L. MacLeod, M. A. Storey and A. Bergen
Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube
In Proceedings of ICPC 2015 (23rd International Conference on Program Comprehension)
Sharing Customization Knowledge
Sharing Development Exercise
Sharing Implementation Approaches
Demonstrating the
Application of Design Patterns
YouTube
and Video Tutorials

9. L. MacLeod, M. A. Storey and A. Bergen
Code, Camera, Action: How Software Developers Document and Share Program Knowledge Using YouTube
In Proceedings of ICPC 2015 (23rd International Conference on Program Comprehension)
Sharing Customization Knowledge
Sharing Development Exercise
Sharing Implementation Approaches
Demonstrating the
Application of Design Patterns
Explaining Data Structures
YouTube
and Video Tutorials

10. YouTube
and Video Tutorials
Information within a video
is not indexed

11. YouTube
and Video Tutorials
Information within a video
is not indexed
Video contents
are not searchable

14. TOOL DEMO SESSION
Friday May 20, 2:20 PM
CodeTube: Extracting Relevant Fragments from
Software Development Video Tutorials
L. Ponzanelli, G. Bavota, A. Mocci, M. Di Penta, R. Oliveto, B. Russo, S. Haiduc, M. Lanza

16. F1 F2 F3 F4

17. F1 F2 F3 F4

18. Frames
Extraction
English Terms
Extraction
Code Frames
Identification
Video Fragments
Identification
Video Fragments
Indexing

19. Frames
Extraction
English Terms
Extraction

20. Frames
Extraction

21. diff
Frames
Extraction

22. diff
Frames
Extraction

23. diff
discarded if diff < 10%
Frames
Extraction

24. Frames
Extraction

25. Frames
Extraction

26. Frames
Extraction
English Terms
Extraction

27. Frames
Extraction
English Terms
Extraction
Code Frames
Identificatio
n

28. English Terms
Extraction

29. Frame
English Terms
Extraction

30. https://github.com/tesseract-ocr
Tesseract OCRFrame
English Terms
Extraction

31. https://github.com/tesseract-ocr
Tesseract OCR
English Terms
Extraction

32. https://github.com/tesseract-ocr
Tesseract OCR
Text
From OCR
English Terms
Extraction

33. Text
From OCR
English Terms
Extraction
// Bulld the "set clmer" lhcenh
Intent timerlntent : new
Intent[AlarmClock.ACTION_SET_TIMER)
.putExtxa(AlarmClock.EXTRA7MESSAGE, message)
.putEXtra(AlarmClocklEXTRAiLENGTH, seconds)
lputExtra(AlarmC10cklEXTRA_SKIP_UI, true);

34. Text
From OCR
English Terms
Extraction
// Bulld the "set clmer" lhcenh
Intent timerlntent : new
Intent[AlarmClock.ACTION_SET_TIMER)
.putExtxa(AlarmClock.EXTRA7MESSAGE, message)
.putEXtra(AlarmClocklEXTRAiLENGTH, seconds)
lputExtra(AlarmC10cklEXTRA_SKIP_UI, true);

35. English Dictionary
Splitting & Stemming
Text
From OCR
English Terms
Extraction

36. C
English Dictionary
Splitting & Stemming
English Terms
Extraction

37. English Dictionary
Splitting & Stemming
Cleaned Text
From OCR
English Terms
Extraction

38. English Dictionary
Splitting & Stemming
Cleaned Text
From OCR
English Terms
Extraction

39. Frames
Extraction
English Terms
Extraction
Code Frames
Identificatio
n

40. English Terms
Extraction
Code Frames
Identification
Video Fragments
Identification

41. Code Frames
Identification

42. Code Frames
Identification

43. http://boofcv.org/
Code Frames
Identification

44. http://boofcv.org/
Code Frames
Identification

45. Code Frames
Identification

46. Code Frames
Identification

47. Code Frames
Identification

48. Code Frames
Identification

49. Text
From OCR Island Parser
StORMeD
http://stormed.inf.usi.ch
Code Frames
Identification

50. Island Parser
StORMeD
http://stormed.inf.usi.ch
Code Frames
Identification

51. Island Parser
StORMeD
http://stormed.inf.usi.ch
H-AST
Model
Code Frames
Identification

52. H-AST
Model
A code frame must contain
a complete or incomplete
Java construct
Code Frames
Identification

53. H-AST
Model
A code frame must contain
a complete or incomplete
Java construct
Code Frames
Identification

54. English Terms
Extraction
Code Frames
Identification
Video Fragments
Identification

55. Code Frames
Identification
Video Fragments
Identification
Video Fragments
Indexing

56. Video Fragments
Identification

57. Video Fragments
Identification

58. If they share code
they belong to the same fragment
Video Fragments
Identification

59. If they share code
they belong to the same fragment
Video Fragments
Identification

60. Ø
Video Fragments
Identification

61. Longest Common Substring (LCS)
Video Fragments
Identification

62. Longest Common Substring (LCS)
8 bit grayscale
Video Fragments
Identification

63. Longest Common Substring (LCS)
8 bit grayscale
Video Fragments
Identification

64. Longest Common Substring (LCS)
hfn5V77As1xgHY9nrqsGlcddsIzju5
XAIGtY2MMgaP3XK3jRX4xgHY9nrqsG
Video Fragments
Identification

65. Longest Common Substring (LCS)
hfn5V77As1xgHY9nrqsGlcddsIzju5
XAIGtY2MMgaP3XK3jRX4xgHY9nrqsG
If LCS > 5% both frame belong
to the same sequence
Video Fragments
Identification

66. Code Sequence
Video Fragments
Identification

67. Code Sequence
Video Fragments
Identification

68. Code Sequence
Video Fragments
Identification

69. Code Sequence
Video Fragments
Identification

70. Transcript
Code Sequence
Video Fragments
Identification

71. Transcript
Code Sequence
Fragment
Video Fragments
Identification

72. Transcript
Code Sequence
Fragment
Video Fragments
Identification

73. Transcript
Code Sequence
Fragment
if textual similarity > 15%
merge two consecutive fragments
Video Fragments
Identification

74. Transcript
Code Sequence
if textual similarity > 15%
merge two consecutive fragments
Fragment
Video Fragments
Identification

75. Transcript
Code Sequence
if fragment length < 50s
merge with next fragment
Fragment
Video Fragments
Identification

76. Fragment
Transcript
Code Sequence
if fragment length < 50s
merge with next fragment
Video Fragments
Identification

77. Code Frames
Identification
Video Fragments
Identification
Video Fragments
Indexing

78. Video Fragments
Identification
Video Fragments
Indexing

79. Video Fragments
Indexing
F1 F2 F3 F4

80. F1 F2
F3 F4
Fragments
Index
StackOverflow
Index
Video Fragments
Indexing

81. Fragments
Index
StackOverflow
Index
Query
Query from
Fragment
F
F
Fragment
+
Discussions
CodeTube
Video Fragments
Indexing

82. Frames
Extraction
English Terms
Extraction
Code Frames
Identification
Video Fragments
Identification
Video Fragments
Indexing

83. CodeTube
http://codetube.inf.usi.ch

84. CodeTube
http://codetube.inf.usi.ch
4,747 Videos

85. CodeTube
http://codetube.inf.usi.ch
4,747 Videos
38,783 Fragments

86. CodeTube
http://codetube.inf.usi.ch
4,747 Videos
38,783 Fragments

87. Approach
Evaluation
Study IIStudy I
Online Survey
Interview with
Practitioners

88. Approach
Evaluation

89. 40 participants
70.6% Professional Developers
17.6% Master Students
11.8% PhD Students
Approach
Evaluation

90. RQ2
To what extent are the extracted video tutorial
fragments cohesive and self-contained?
Study I
Online Survey

91. Study I
Online Survey
RQ2
To what extent are the extracted video tutorial
fragments cohesive and self-contained?

92. Study I
Online Survey
RQ2
To what extent are the extracted video tutorial
fragments cohesive and self-contained?

93. Study I
Online Survey
RQ2
To what extent are the extracted video tutorial
fragments cohesive and self-contained?

94. Cohesion
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
Study I
Online Survey
RQ2
To what extent are the extracted video tutorial
fragments cohesive and self-contained?

95. Cohesion
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
71%
Study I
Online Survey
RQ2
To what extent are the extracted video tutorial
fragments cohesive and self-contained?

96. Cohesion
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
Self-Containment
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
71%
Study I
Online Survey
RQ2
To what extent are the extracted video tutorial
fragments cohesive and self-contained?

97. Cohesion
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
Self-Containment
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
71%
47%31%
Study I
Online Survey
RQ2
To what extent are the extracted video tutorial
fragments cohesive and self-contained?

98. Approach
Evaluation
Study IIStudy I
Online Survey
Interview with
Practitioners

99. Study II
Practitioners
Semi-Structured Interviews
3 Project Managers
RQ5
Would CodeTube be useful for practitioners?

100. “I have just used CodeTube this morning. I was
looking for something related to Android
WebSocket. I found all I needed. Awesome!”
Luciano Cutone, Project Manager @ IdeaSoftware
Study II
Practitioners
RQ5
Would CodeTube be useful for practitioners?

101. “I think that the tool is nice. You are trying to solve an
important and challenging problem, that is merging
accurately different sources of information.”
Giuseppe Socci, Project Manager @ Genialapp
Study II
Practitioners
RQ5
Would CodeTube be useful for practitioners?

102. “I am not 100% sure that extracting shorter
fragments makes you more productive. It depends
on the scenario where the video tutorial is used“
Nicola Novello, Project Manager @ Next
Study II
Practitioners
RQ5
Would CodeTube be useful for practitioners?

103. “I strongly believe that the tool has
an enormous potential”
Nicola Novello, Project Manager @ Next
Study II
Practitioners
RQ5
Would CodeTube be useful for practitioners?

104. Open
Challenges
✓

105. Open
Challenges
✓
Videos get canceled

106. Open
Challenges
✓
Videos get canceled
Video clones

107. Open
Challenges
✓
Videos get canceled
Video clones
More scalable approach

108. Open
Challenges
✓
Videos get canceled
Video clones
More scalable approach
More robust approach

109. Too Long;
Didn't Watch!

110. Luca Ponzanelli
Andrea Mocci
Michele Lanza
Sonia Haiduc
Mir Hasan
Gabriele Bavota
Barbara Russo
Max Di Penta Rocco Oliveto
Florida State
University
University of
Molise
University of
Sannio
Free University of
Bozen-Bolzano
Università della
Svizzera italiana
CodeTube
Extracting Relevant Fragments from
Software Development Video Tutorials
http://codetube.inf.usi.ch

112. Parameters
Tuning
Manual
Tagging
10 Videos
Tagged
Fragments

113. Parameters
Tuning
Tagged
Fragments
CodeTube
Fragments
α β γ
CodeTube
Configuration
MoJoFM
Comparison

114. Parameters
Tuning
LCS percentageα
β
γ
Textual Similarity
Fragment Length

115. Parameters
Tuning
Parameter Min Max Delta Selected
LCS percentage (α) 5% 50% 5% 5%
Textual Similarity (β) 10% 80% 5% 15%
Fragment Length (γ) 10s 120s 10s 50s

116. Approach
Evaluation

117. 40 participants
70.6% Professional Developers
17.6% Master Students
11.8% PhD Students
Approach
Evaluation

118. 40 participants
70.6% Professional Developers
17.6% Master Students
11.8% PhD Students
38.5% less than 1 year
23.5% more than 3 years
20.5% between 2 and 3 years
Android Experience
17.6% between 1 and 2 years
Approach
Evaluation

119. Video Tutorial Usage
38.2% - several times a week
35.3% - few times a month
23.5% - less than once a month
3.0% - daily
Approach
Evaluation

120. RQ1
What are the perceived benefits and obstacles
of using video tutorials?
Study I
Online Survey

121. “video tutorials describe a complete
process step-by-step”
+
RQ1
What are the perceived benefits and obstacles
of using video tutorials?
Study I
Online Survey

122. “browsing is not easy, unless the
video has an index to navigate through
the concepts/sections in the video“
Study I
Online Survey
RQ1
What are the perceived benefits and obstacles
of using video tutorials?
“video tutorials describe a complete
process step-by-step”
+

123. RQ3
To what extent are the Stack Overflow discussions identified by
CodeTube relevant and complementary to the linked video fragments?
Study I
Online Survey

124. RQ3
To what extent are the Stack Overflow discussions identified by
CodeTube relevant and complementary to the linked video fragments?
Relevance
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
Study I
Online Survey

125. RQ3
To what extent are the Stack Overflow discussions identified by
CodeTube relevant and complementary to the linked video fragments?
Relevance
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
38%32%
Study I
Online Survey

126. RQ3
To what extent are the Stack Overflow discussions identified by
CodeTube relevant and complementary to the linked video fragments?
Relevance
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
Complementariness
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
38%32%
Study I
Online Survey

127. RQ3
To what extent are the Stack Overflow discussions identified by
CodeTube relevant and complementary to the linked video fragments?
Relevance
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
Complementariness
0%
25%
50%
75%
100%
1 1.5 2 2.5 3
38%32%
82%
Study I
Online Survey

128. RQ4
To what extent is CodeTube able to return results
relevant to a textual query?
CodeTube
Results
YouTube
Results
Query (Q)
NDCGCT(Q,3) = 0.67
NDCGYT(Q,3) = 0.63
Normalized Cumulative Discounted Gain (NDCG)
Study I
Online Survey

129. RQ4
To what extent is CodeTube able to return results
relevant to a textual query?
NDCGCT(Q,3) = 0.67
NDCGYT(Q,3) = 0.63
No statistically
significant difference
Normalized Cumulative Discounted Gain (NDCG)
Study I
Online Survey