Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)Microsoft Research Connections (Judith Bisho...
Testing  Educational GamingDynamic Symbolic Execution (Pex)Parameterized Unit TestingPex for Fun (Pex4Fun)TrainingSuppor...
AgendaBackgroundPex, Dynamic Symbolic Execution,Parameterized Unit TestingWriting Code in a BrowserCoding DuelsEducational...
PEX, DYNAMIC SYMBOLIC EXECUTION,PARAMETERIZED UNIT TESTINGBackground
http://research.microsoft.com/pex/
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Dynamic Sy...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}InputnullD...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullIn...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullCo...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullCo...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0...
void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0...
Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert....
Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert....
Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert....
Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list....
Parameterized Unit Test =Unit Test with ParametersVision: Parameterized Unit Testingvoid AddAndCountTest(List list, int it...
Parameterized Unit Test =Unit Test with ParametersSeparation of concernsSpecification of behaviorData to achieve coverageV...
Parameterized Unit Testing in PracticeGetting PopularParameterized Unit Tests (PUTs) commonlysupported by various test fra...
http://pex4fun.com/CoverMe
Limitations16K characters of code (single file / editor window)*Single-threaded onlyNo environment interactions*No non-det...
Dynamic Symbolic ExecutionSummary“Ask Pex” sends code to cloudCode is compiled and analyzed in cloudDynamic Symbolic Execu...
WRITING CODEIN A BROWSERCode Auto-Completion
Auto-Completion
Writing Code in BrowserSummaryCode is compiled in cloudCode is executed in cloudAuto-completion via cloud/Javascriptin bro...
CODING DUELSFun and Engaging Serious Game – Win by Writing Code
Coding Duels1,168,577 clicked Ask Pex!
Coding DuelsPex computes “semantic diff” in cloudcode written in browser vs.secret reference implementationYou win when Pe...
Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;re...
Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;re...
Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;re...
Coding DuelsFun and EngagingIterative gameplayAdaptivePersonalizedNo cheatingClear winning criterion
Example User Feedback“It really got me *excited*. The part that got me most isabout spreading interest in teaching CS: I d...
Coding Duel Competition@ICSE 2011http://pexforfun.com/icse2011
Coding DuelsSummaryCoding Duel: write function that matchesspecification given by another implementationSemantic equivalen...
SOCIAL EXPERIENCEConnecting Students – Competitive yet Self-Paced Environment
Social Experience
Social ExperienceSummaryCommunityHigh score lists, leaderboardLive feedhttp://pex4fun.com/Community.aspxhttp://pex4fun.com...
PEX4FUN, AN EDUCATIONAL PLATFORMPages – Courses – Automated Assessments
Teaching and Learning
Coding Duels for Automatic Grading@Grad Software Engineering Coursehttp://pexforfun.com/gradsofteng
Coding Duels for Training Testingpublic static string Puzzle(int[] elems, int capacity, int elem) {if ((maxsize <= 0) || (...
Usage Scenarios of Pex4Fun• Massive Open Online Courses (MOOC): Challenges– Grading, addressed by Pex4Fun– Cheating [Open ...
Teaching and LearningSummaryHow to become a teacher:Sign in, choose nicknameSend us email with nicknameTeachers canCreate ...
OutlookAnalyzing code submissionsHow to guide students?Detect plagiarismFurther rankingSupport for more languages
More ReadingNikolai Tillmann, Jonathan De Halleux, Tao Xie,Sumit Gulwani and Judith BishopTeaching and Learning Programmin...
ConclusionPexTest generation for .NETbased on dynamic symbolic executionEnables Parameterized Unit TestingPex4Fun on the w...
Testing  Educational GamingTrainingSupporthttp://research.microsoft.com/pex/ http://pex4fun.comQ & AThank you!contact: t...
Upcoming SlideShare
Loading in …5
×

Testing for Educational Gaming and Educational Gaming for Testing

379
-1

Published on

A talk given at Google Test Automation Conference (GTAC 2013) https://developers.google.com/google-test-automation-conference/

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
379
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Testing for Educational Gaming and Educational Gaming for Testing

  1. 1. Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)Microsoft Research Connections (Judith Bishop)Testing forEducational Gaming andEducational Gamingfor TestingTao XieNorth Carolina State Universitycontact: taoxie@gmail.com
  2. 2. Testing  Educational GamingDynamic Symbolic Execution (Pex)Parameterized Unit TestingPex for Fun (Pex4Fun)TrainingSupporthttp://research.microsoft.com/pex/ http://pex4fun.com
  3. 3. AgendaBackgroundPex, Dynamic Symbolic Execution,Parameterized Unit TestingWriting Code in a BrowserCoding DuelsEducational PlatformConclusion
  4. 4. PEX, DYNAMIC SYMBOLIC EXECUTION,PARAMETERIZED UNIT TESTINGBackground
  5. 5. http://research.microsoft.com/pex/
  6. 6. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  7. 7. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}InputnullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  8. 8. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullInputnullExecute&MonitorObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  9. 9. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnullExecute&MonitorChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  10. 10. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  11. 11. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0FTFa==nullConstraints to solvea!=nullInputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  12. 12. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0FTFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  13. 13. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  14. 14. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  15. 15. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  16. 16. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  17. 17. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullTConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  18. 18. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullTConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890Done: There is no path left.Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  19. 19. Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  20. 20. Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}for any list,for any item,
  21. 21. Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}… adding 1 itemincreases Count by 1
  22. 22. Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  23. 23. Parameterized Unit Test =Unit Test with ParametersVision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  24. 24. Parameterized Unit Test =Unit Test with ParametersSeparation of concernsSpecification of behaviorData to achieve coverageVision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  25. 25. Parameterized Unit Testing in PracticeGetting PopularParameterized Unit Tests (PUTs) commonlysupported by various test frameworks• .NET: Supported by .NET test frameworks• Java: Supported by JUnit 4.XGenerating test inputs for PUTs supported by tools• .NET: Supported by Microsoft Research Pex• Java: Supported by Agitar AgitarOne(http://www.agitar.com/)
  26. 26. http://pex4fun.com/CoverMe
  27. 27. Limitations16K characters of code (single file / editor window)*Single-threaded onlyNo environment interactions*No non-determinism*Try to avoid floating-point computationsMax. 20s total exploration time** Limitation only applies to pex4fun.com, but not standalone Pex tool
  28. 28. Dynamic Symbolic ExecutionSummary“Ask Pex” sends code to cloudCode is compiled and analyzed in cloudDynamic Symbolic Execution automatically findsrelevant interesting test inputs that achieve highcode coverageResults are shown in browser
  29. 29. WRITING CODEIN A BROWSERCode Auto-Completion
  30. 30. Auto-Completion
  31. 31. Writing Code in BrowserSummaryCode is compiled in cloudCode is executed in cloudAuto-completion via cloud/Javascriptin browser
  32. 32. CODING DUELSFun and Engaging Serious Game – Win by Writing Code
  33. 33. Coding Duels1,168,577 clicked Ask Pex!
  34. 34. Coding DuelsPex computes “semantic diff” in cloudcode written in browser vs.secret reference implementationYou win when Pex finds no differences
  35. 35. Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl18
  36. 36. Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl18
  37. 37. Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl18
  38. 38. Coding DuelsFun and EngagingIterative gameplayAdaptivePersonalizedNo cheatingClear winning criterion
  39. 39. Example User Feedback“It really got me *excited*. The part that got me most isabout spreading interest in teaching CS: I do think that it’sREALLY great for teaching | learning!”“I used to love the first person shooters and thesatisfaction of blowing away a whole team ofNoobies playing Rainbow Six, but this is far morefun.”“I’m afraid I’ll have to constrain myself to spend just an houror so a day on this really exciting stuff, as I’m really stuffedwith work.”Released since 2010X
  40. 40. Coding Duel Competition@ICSE 2011http://pexforfun.com/icse2011
  41. 41. Coding DuelsSummaryCoding Duel: write function that matchesspecification given by another implementationSemantic equivalence checked in cloudStudent’s tasks:Infer likely intended behaviorExpress behavior as codehttp://pex4fun.com/CreatingAndPublishingCodingDuelshttp://pex4fun.com/TipsAndTricksForGreatDuels
  42. 42. SOCIAL EXPERIENCEConnecting Students – Competitive yet Self-Paced Environment
  43. 43. Social Experience
  44. 44. Social ExperienceSummaryCommunityHigh score lists, leaderboardLive feedhttp://pex4fun.com/Community.aspxhttp://pex4fun.com/Livefeed.aspx
  45. 45. PEX4FUN, AN EDUCATIONAL PLATFORMPages – Courses – Automated Assessments
  46. 46. Teaching and Learning
  47. 47. Coding Duels for Automatic Grading@Grad Software Engineering Coursehttp://pexforfun.com/gradsofteng
  48. 48. Coding Duels for Training Testingpublic static string Puzzle(int[] elems, int capacity, int elem) {if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1)))return "Assumption Violation!";Stack s= new Stack(capacity);for (int i = 0; i < elems.Length; i++)s.Push(elems[i]);int origSize = s.GetNumOfElements();//Please fill in below test scenario on the s stack//The lines below include assertions to assert the program behaviorPexAssert.IsTrue(s.GetNumOfElements() == origSize + 1);PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty());PexAssert.IsTrue(s.IsMember(elem));return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “+ s.IsMember(elem).ToString() + "; " + s.IsEmpty();}Set up a stack with some elementsCache values used in assertions
  49. 49. Usage Scenarios of Pex4Fun• Massive Open Online Courses (MOOC): Challenges– Grading, addressed by Pex4Fun– Cheating [Open Challenge]• Course assignments (students/professionals)– E.g., intro programming, software engineering• Student/professional competitions– E.g., coding-duel competition at ICSE 2011• Assessment of testing/programming/problemsolving skills for job applicants– Not just final results of problem solving but also process!
  50. 50. Teaching and LearningSummaryHow to become a teacher:Sign in, choose nicknameSend us email with nicknameTeachers canCreate coursesReuse existing or author existing pagesGet automated assessments of studentsSee students codehttp://pex4fun.com/TeachingWithPexForFun
  51. 51. OutlookAnalyzing code submissionsHow to guide students?Detect plagiarismFurther rankingSupport for more languages
  52. 52. More ReadingNikolai Tillmann, Jonathan De Halleux, Tao Xie,Sumit Gulwani and Judith BishopTeaching and Learning Programming andSoftware Engineering via Interactive GamingIn Proceedings of the 35th InternationalConference on Software Engineering (ICSE 2013),Software Engineering Education (SEE), SanFrancisco, CA, May 2013.http://people.engr.ncsu.edu/txie/publications/icse13see-pex4fun.pdf
  53. 53. ConclusionPexTest generation for .NETbased on dynamic symbolic executionEnables Parameterized Unit TestingPex4Fun on the web.NET: C#, Visual Basic, F#Execution, code analysis with PexCoding DuelsGreat for teaching, learninghttp://pex4fun.comhttp://research.microsoft.com/pex
  54. 54. Testing  Educational GamingTrainingSupporthttp://research.microsoft.com/pex/ http://pex4fun.comQ & AThank you!contact: taoxie@gmail.comAcknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603,a Microsoft Research SEIF Award, and a Microsoft Research Award.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×