Eric Anderson, Sihan Li, and Tao Xie. A Preliminary Field Study of Game Programming on Mobile Devices. Presented in Workshop on Programming for Mobile and Touch (PROMOTO 2013), Indianapolis, IN, October 2013.

1. A Preliminary Field Study of
Game Programming on Mobile Devices
Eric Anderson
Sihan Li Tao Xie
North Carolina State University
University of Illinois
at Urbana-Champaign
http://arxiv.org/abs/1310.3308
1

2. Mining App Stores: Mobile App Markets
Apple App Store
Google Play
Microsoft Windows (Phone)
Apps
Microsoft Research TouchDevelop

3. Mining App Stores:
Program Analysis + Text Analytics
o Previous approaches look at
permissions  code (runtime behaviors)
o What does the users expect?
App
Description
App
Code
Pandita et al. WHYPER: Towards
Automating Risk Assessment of
Mobile Applications. USENIX Security
2013
http://web.engr.illinois.edu/~taoxie/p
ublications/usenixsec13-whyper.pdf
App
Permissions
3

4. Mining App Stores:
Program Analysis + Text Analytics
o Previous approaches look at
permissions  code (runtime behaviors)
o What does the users expect?
App
Description
App
Code
Pandita et al. WHYPER: Towards
Automating Risk Assessment of
Mobile Applications. USENIX Security
2013
http://web.engr.illinois.edu/~taoxie/p
ublications/usenixsec13-whyper.pdf
App
Permissions
3

5. Mining App Stores:
Program Analysis + Text Analytics
o Previous approaches look at
permissions  code (runtime behaviors)
o What does the users expect?
App
Description
App
Code
Pandita et al. WHYPER: Towards
Automating Risk Assessment of
Mobile Applications. USENIX Security
2013
http://web.engr.illinois.edu/~taoxie/p
ublications/usenixsec13-whyper.pdf
App
Permissions
3

6. Mining App Stores:
Program Analysis + Text Analytics
o Previous approaches look at
permissions  code (runtime behaviors)
o What does the users expect?
App
Description
App
Code
Pandita et al. WHYPER: Towards
Automating Risk Assessment of
Mobile Applications. USENIX Security
2013
http://web.engr.illinois.edu/~taoxie/p
ublications/usenixsec13-whyper.pdf
App
Permissions
3

7. Mining App Stores:
Program Analysis + Text Analytics
o Previous approaches look at
permissions  code (runtime behaviors)
o What does the users expect?
App
Description
App
Code
Pandita et al. WHYPER: Towards
Perhaps it's my ignorance, but would
Automating Risk Assessment of
anything related to storage, system
Mobile Applications. USENIX Security
tools, your location, camera,
2013
development tools, phone calls, and
http://web.engr.illinois.edu/~taoxie/p
ublications/usenixsec13-whyper.pdf
network communication be
ridiculously more accessible to the
App
smartphone capabilities than what a
Permissions
3
flashlight app would require?

8. Mining App Stores:
Open Source Mobile App Repo
https://f-droid.org/
http://research.microsoft.com/en-us/projects/touchdevelop/

9. Rapid Increase of TouchDevelop Popularity
vs. 17322 Feb this year
5

10. Mining App Stores:
Educational/Learning Analytics
TouchDevelop
Team@MSR
TouchDevelop Users
SE/HCI/Education …
Researchers
6

11. Mining App Stores:
Educational/Learning Analytics
TouchDevelop
Team@MSR
TouchDevelop Users
SE/HCI/Education …
Researchers
6

12. Mining App Stores:
Educational/Learning Analytics
TouchDevelop
Team@MSR
TouchDevelop Users
Data-Driven
Decision Making
(not just gut feeling)
SE/HCI/Education …
Researchers
6

13. Mining App Stores:
Educational/Learning Analytics
TouchDevelop
Team@MSR
TouchDevelop Users
Data-Driven
Decision Making
(not just gut feeling)
SE/HCI/Education …
Researchers
6

14. Our Previous Study on
TouchDevelop Scripts and Users
• Characteristic and longitudinal study
– 17,322 scripts and 4,275 users (July 2011-Feb 2013)
• Scripts
– Structural features
– Code reuse
– Change of features over time
• Users
– Expertise
– Publishing and learning activities over time
Sihan Li, Tao Xie, and Nikolai Tillmann. A Comprehensive Field Study of End-User Programming
on Mobile Devices. VL/HCC 2013.
7
http://web.engr.illinois.edu/~taoxie/publications/vlhcc13-study.pdf

15. Previous Characteristic Study
What do TouchDevelop scripts look like?
What is the code-reuse ratio of TouchDevelop
scripts?
What kinds of TouchDevelop users are there?
How are TouchDevelop scripts changing
over time?
What is the users’ progress of developing
TouchDevelop scripts?
8

16. What do TouchDevelop scripts look like?
Structural Features: Findings
EMC: # external method call sites
9

17. What do TouchDevelop scripts look like?
Structural Features: Findings
Small in size
73% less than 100 LOC, avg. 133 LOC
EMC: # external method call sites
9

18. What do TouchDevelop scripts look like?
Structural Features: Findings
Small in size
73% less than 100 LOC, avg. 133 LOC
Heavily use of external code
106 EMC out of 114 MC per script
Every two lines contain an EMC
EMC: # external method call sites
9

19. What do TouchDevelop scripts look like?
Structural Features: Findings
Small in size
73% less than 100 LOC, avg. 133 LOC
Heavily use of external code
106 EMC out of 114 MC per script
Every two lines contain an EMC
Implications: Heavily rely on external methods
to achieve functionalities. Enriching the built-in
APIs and library scripts.
EMC: # external method call sites
9

20. Types of Code Reuse
•
•
•
•
Inherit code from a parent script
Use the existing library scripts
Use TouchDevelop built-in APIs
Copy and paste
10

21. Types of Code Reuse
•
•
•
•
Inherit code from a parent script
Use the existing library scripts
Use TouchDevelop built-in APIs
Copy and paste
10

22. What is the code-reuse ratio of TouchDevelop scripts?
Code Reuse: Findings
11

23. What is the code-reuse ratio of TouchDevelop scripts?
Code Reuse: Findings
High code-reuse ratio
58% scripts have a parent
11

24. What is the code-reuse ratio of TouchDevelop scripts?
Code Reuse: Findings
High code-reuse ratio
58% scripts have a parent
Reuse their own code
74% parent-child pairs are written
by the same user
11

25. What is the code-reuse ratio of TouchDevelop scripts?
Code Reuse: Findings
High code-reuse ratio
58% scripts have a parent
Reuse their own code
74% parent-child pairs are written
by the same user
Small modifications
Quite a lot less than 10 LOC
11

26. What is the code-reuse ratio of TouchDevelop scripts?
Code Reuse: Findings
High code-reuse ratio
58% scripts have a parent
Reuse their own code
74% parent-child pairs are written
by the same user
Small modifications
Quite a lot less than 10 LOC
Implications: Users tend to reuse code. Lack of code
reuse across different users. Improve tool support for
code reuse.
11

27. Expertise of Users: Metrics
• Activity of script-publishing
– # scripts published
• Quality of scripts and comments
– # positive reviews received
• Grasp of the TouchDevelop language
– # TouchDevelop language features used (e.g.,
programming concepts, built-in APIs)
12

28. Clusters of Users
74.2%
21.5%
4.2%
13

29. Clusters of Users
74.2%
21.5%
4.2%
13

30. What kinds of TouchDevelop users are there?
Expertise of Users: Findings
14

31. What kinds of TouchDevelop users are there?
Expertise of Users: Findings
Most users are novices
74% novices, 22% ordinary users,
4% experts
Big differences between users’ expertise
14

32. What kinds of TouchDevelop users are there?
Expertise of Users: Findings
Most users are novices
74% novices, 22% ordinary users,
4% experts
Big differences between users’ expertise
Implications: Give novices high priority when
designing tools and services. Provide customized
services (e.g., UI) to different kinds of users.
14

33. How are TouchDevelop scripts changing over time?
Changes of Scripts: Findings
15

34. How are TouchDevelop scripts changing over time?
Changes of Scripts: Findings
Important increasing trends
# scripts
# library
, # large scripts ,
, code-reuse ratio
.
15

35. How are TouchDevelop scripts changing over time?
Changes of Scripts: Findings
Important increasing trends
# scripts
# library
, # large scripts ,
, code-reuse ratio
.
Implications: Enriching TouchDevelop language and
Improving the management of libraries to better
accommodate these trends.
15

36. What is users’ progress of developing TouchDevelop scripts?
Evolution of Users: Findings
16

37. What is users’ progress of developing TouchDevelop scripts?
Evolution of Users: Findings
Three patterns of publishing
scripts
22.1% users active initially, less active later
9.6% users not active initially, more active later
68.3% users publish one or two scripts, and then leave
16

38. What is users’ progress of developing TouchDevelop scripts?
Evolution of Users: Findings
Three patterns of publishing
scripts
22.1% users active initially, less active later
9.6% users not active initially, more active later
68.3% users publish one or two scripts, and then leave
Implications: Useful to know why users leave or
become less active and why they become more active.
Design better strategies to retain users.
16

39. Evolution of Users: Findings
17

40. Evolution of Users: Findings
Three patterns of learning
language features
16.3% users learn most initially, and only a few later
12.4% users learn some initially, and quite a lot at
certain point
71.3% users learn a few initially, and stop learning
17

41. Evolution of Users: Findings
Three patterns of learning
language features
16.3% users learn most initially, and only a few later
12.4% users learn some initially, and quite a lot at
certain point
71.3% users learn a few initially, and stop learning
Implications: Users have different learning behaviors.
Provide an adaptive tutoring system based on the
history of users’ language-feature usage.
17

42. Overview of This Study
• Motivation
– As mobile gaming grows, it is important to analyze
game applications and how it compares to other
applications.
• Characteristic study
– 31,699 scripts, including 539 game scripts
• Goal
– To discover how TouchDevelop games compare to
non-games
– To discover what makes some games more popular
18

43. Characteristic study on Games
• RQ1: What do TouchDevelop game scripts
look like when compared to non-game
scripts?
• RQ2: How do games compare to non-games in
terms of popularity?
• RQ3: What features do more popular games
have that less popular games lack?
19

44. Research Question 1
• RQ1: What do TouchDevelop game scripts
look like when compared to non-game
scripts?
20

45. Structural Comparison
Mean
#LOC
Median
#LOC
Game
369
272
Non-Game
210
50
• Finding: Game scripts are longer than nongame scripts.
• Implication: Expanding IDE support could
enhance the creation of large scripts.
21

46. Structural Comparison cont.
Mean
# action
Game
Non-Game
Mean
# event
11.3
3.2
6.1
1.6
• Finding: Game scripts typically have more
actions and events than non-game scripts
• Implication: IDE expansion could be highly
beneficial.
22

47. Structural Comparison cont.
Mean
# MC
Mean
MC density
Game
347
0.94
Non-Game
165
0.79
• Finding: Game scripts invoke more methods than
non-game scripts.
• Implication: Games include more features than
non-games. Emphasis on games in marketing
could be beneficial.
23

48. Research Question 2
• RQ2: How do games compare to non-games in
terms of popularity?
24

49. Comparison of Popularity
Mean
# run
Game
Non-Game
Mean
# install
Mean
# likes
216
35
1.80
23
3
0.32
• Finding: Game scripts are more popular than nongame scripts
• Implication: TouchDevelop should market games
as important types of scripts.
25

50. Research Question 3
• RQ3: What features do more popular games
have that less popular games lack?
26

51. Game Popularity Comparison
Games
Mean
# LOC
Median
# LOC
Popular
462
384
Unpopular
363
255
• Metric: Top 25% of frequently run games as
popular
• Finding: Popular games are longer than
unpopular games
• Implication: Expansion of the IDE could encourage
larger scripts
27

52. Game Popularity Comparison cont.
Games
Mean
# action
Mean
# event
Popular
14.1
3.6
Unpopular
11.1
3.2
• Finding: Popular and unpopular game scripts
have similar numbers of actions and events
• Implication: Large numbers of actions and
events may not contribute to popularity
28

53. Comparison of Actions and Events
16
# action and
events
14
12
10
8
6
4
2
0
Mean Actions
Non-Games
Mean Events
Games
Popular Games
Unpopular Games
29

54. Comparison of Mean and Median
Length
500
# LOC
450
400
350
300
250
200
150
100
50
0
Mean Length
Non-Games
Median Length
Games
Popular Games
Unpopular Games
30

55. Conclusion
• Previous study [VL/HCC 13] reports a number
of valuable findings
• This study focuses on
game vs. non-games
popular vs. unpopular games
– Games are typically longer and more feature-rich than
non-games
– Popular games are typically longer but not necessarily
more feature rich than unpopular games
31

56. Thank you!
ACKNOWLEDGMENTS. This work is supported in part by NSF grants CCF-0845272, CCF0915400, CNS-0958235, CNS-1160603, CCF-1349666, CNS-1318419, a Microsoft Research award, and
NSF of China No. 61228203. We thank Nikolai Tillmann for his help and feedback in conducting the
study described in this paper.
32

57. Conclusion
• Previous study [VL/HCC 13] reports a number
of valuable findings
• This study focuses on
game vs. non-games
popular vs. unpopular games
– Games are typically longer and more feature-rich than
non-games
– Popular games are typically longer but not necessarily
more feature rich than unpopular games
33