A talk given at Google Test Automation Conference (GTAC 2013) https://developers.google.com/google-test-automation-conference/

1. Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)
Microsoft Research Connections (Judith Bishop)
Testing for
Educational Gaming and
Educational Gaming
for Testing
Tao Xie
North Carolina State University
contact: taoxie@gmail.com

2. Testing  Educational Gaming
Dynamic Symbolic Execution (Pex)
Parameterized Unit Testing
Pex for Fun (Pex4Fun)
Training
Support
http://research.microsoft.com/pex/ http://pex4fun.com

3. Agenda
Background
Pex, Dynamic Symbolic Execution,
Parameterized Unit Testing
Writing Code in a Browser
Coding Duels
Educational Platform
Conclusion

4. PEX, DYNAMIC SYMBOLIC EXECUTION,
PARAMETERIZED UNIT TESTING
Background

5. http://research.microsoft.com/pex/

6. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

7. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
Input
null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

8. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
T
a==null
Input
null
Execute&Monitor
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

9. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
T
a==null
Constraints to solve
a!=null
Input
null
Execute&Monitor
Choose next path
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

10. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
T
a==null
Constraints to solve
a!=null
Input
null
{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

11. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
F
TF
a==null
Constraints to solve
a!=null
Input
null
{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

12. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
F
TF
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
Input
null
{}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

13. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
Input
null
{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

14. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
Input
null
{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

15. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

16. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

17. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

18. void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
a.Length>0
a[0]==123…
TF
T
F
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==123456890
Input
null
{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
Done: There is no path left.
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork

19. Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}

20. Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}
for any list,
for any item,

21. Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}
… adding 1 item
increases Count by 1

22. Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
Assume.IsTrue(list != null);
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}

23. Parameterized Unit Test =
Unit Test with Parameters
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
Assume.IsTrue(list != null);
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}

24. Parameterized Unit Test =
Unit Test with Parameters
Separation of concerns
Specification of behavior
Data to achieve coverage
Vision: Parameterized Unit Testing
void AddAndCountTest(List list, int item) {
Assume.IsTrue(list != null);
var count = list.Count;
list.Add(item);
Assert.AreEqual(count + 1, list.Count);
}

25. Parameterized Unit Testing in Practice
Getting Popular
Parameterized Unit Tests (PUTs) commonly
supported by various test frameworks
• .NET: Supported by .NET test frameworks
• Java: Supported by JUnit 4.X
Generating test inputs for PUTs supported by tools
• .NET: Supported by Microsoft Research Pex
• Java: Supported by Agitar AgitarOne
(http://www.agitar.com/)

26. http://pex4fun.com/CoverMe

27. Limitations
16K characters of code (single file / editor window)*
Single-threaded only
No environment interactions*
No non-determinism*
Try to avoid floating-point computations
Max. 20s total exploration time*
* Limitation only applies to pex4fun.com, but not standalone Pex tool

28. Dynamic Symbolic Execution
Summary
“Ask Pex” sends code to cloud
Code is compiled and analyzed in cloud
Dynamic Symbolic Execution automatically finds
relevant interesting test inputs that achieve high
code coverage
Results are shown in browser

29. WRITING CODE
IN A BROWSER
Code Auto-Completion

30. Auto-Completion

31. Writing Code in Browser
Summary
Code is compiled in cloud
Code is executed in cloud
Auto-completion via cloud/Javascript
in browser

32. CODING DUELS
Fun and Engaging Serious Game – Win by Writing Code

33. Coding Duels
1,168,577 clicked 'Ask Pex!'

34. Coding Duels
Pex computes “semantic diff” in cloud
code written in browser vs.
secret reference implementation
You win when Pex finds no differences

35. Behind the Scene of Pex for Fun
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
18

36. Behind the Scene of Pex for Fun
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
18

37. Behind the Scene of Pex for Fun
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
class Test {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
18

38. Coding Duels
Fun and Engaging
Iterative gameplay
Adaptive
Personalized
No cheating
Clear winning criterion

39. Example User Feedback
“It really got me *excited*. The part that got me most is
about spreading interest in teaching CS: I do think that it’s
REALLY great for teaching | learning!”
“I used to love the first person shooters and the
satisfaction of blowing away a whole team of
Noobies playing Rainbow Six, but this is far more
fun.”
“I’m afraid I’ll have to constrain myself to spend just an hour
or so a day on this really exciting stuff, as I’m really stuffed
with work.”
Released since 2010
X

40. Coding Duel Competition
@ICSE 2011
http://pexforfun.com/icse2011

41. Coding Duels
Summary
Coding Duel: write function that matches
specification given by another implementation
Semantic equivalence checked in cloud
Student’s tasks:
Infer likely intended behavior
Express behavior as code
http://pex4fun.com/CreatingAndPublishingCodingDuels
http://pex4fun.com/TipsAndTricksForGreatDuels

42. SOCIAL EXPERIENCE
Connecting Students – Competitive yet Self-Paced Environment

43. Social Experience

44. Social Experience
Summary
Community
High score lists, leaderboard
Live feed
http://pex4fun.com/Community.aspx
http://pex4fun.com/Livefeed.aspx

45. PEX4FUN, AN EDUCATIONAL PLATFORM
Pages – Courses – Automated Assessments

46. Teaching and Learning

47. Coding Duels for Automatic Grading
@Grad Software Engineering Course
http://pexforfun.com/gradsofteng

48. Coding Duels for Training Testing
public static string Puzzle(int[] elems, int capacity, int elem) {
if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1)))
return "Assumption Violation!";
Stack s= new Stack(capacity);
for (int i = 0; i < elems.Length; i++)
s.Push(elems[i]);
int origSize = s.GetNumOfElements();
//Please fill in below test scenario on the s stack
//The lines below include assertions to assert the program behavior
PexAssert.IsTrue(s.GetNumOfElements() == origSize + 1);
PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty());
PexAssert.IsTrue(s.IsMember(elem));
return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “
+ s.IsMember(elem).ToString() + "; " + s.IsEmpty();
}
Set up a stack with some elements
Cache values used in assertions

49. Usage Scenarios of Pex4Fun
• Massive Open Online Courses (MOOC): Challenges
– Grading, addressed by Pex4Fun
– Cheating [Open Challenge]
• Course assignments (students/professionals)
– E.g., intro programming, software engineering
• Student/professional competitions
– E.g., coding-duel competition at ICSE 2011
• Assessment of testing/programming/problem
solving skills for job applicants
– Not just final results of problem solving but also process!

50. Teaching and Learning
Summary
How to become a teacher:
Sign in, choose nickname
Send us email with nickname
Teachers can
Create courses
Reuse existing or author existing pages
Get automated assessments of students
See students code
http://pex4fun.com/TeachingWithPexForFun

51. Outlook
Analyzing code submissions
How to guide students?
Detect plagiarism
Further ranking
Support for more languages

52. More Reading
Nikolai Tillmann, Jonathan De Halleux, Tao Xie,
Sumit Gulwani and Judith Bishop
Teaching and Learning Programming and
Software Engineering via Interactive Gaming
In Proceedings of the 35th International
Conference on Software Engineering (ICSE 2013),
Software Engineering Education (SEE), San
Francisco, CA, May 2013.
http://people.engr.ncsu.edu/txie/publications/ic
se13see-pex4fun.pdf

53. Conclusion
Pex
Test generation for .NET
based on dynamic symbolic execution
Enables Parameterized Unit Testing
Pex4Fun on the web
.NET: C#, Visual Basic, F#
Execution, code analysis with Pex
Coding Duels
Great for teaching, learning
http://pex4fun.com
http://research.microsoft.com/pex

54. Testing  Educational Gaming
Training
Support
http://research.microsoft.com/pex/ http://pex4fun.com
Q & A
Thank you!
contact: taoxie@gmail.com
Acknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603,
a Microsoft Research SEIF Award, and a Microsoft Research Award.