Testing for Educational Gaming and Educational Gaming for Testing
Upcoming SlideShare
Loading in...5
×
 

Testing for Educational Gaming and Educational Gaming for Testing

on

  • 440 views

A talk given at Google Test Automation Conference (GTAC 2013) https://developers.google.com/google-test-automation-conference/

A talk given at Google Test Automation Conference (GTAC 2013) https://developers.google.com/google-test-automation-conference/

Statistics

Views

Total Views
440
Views on SlideShare
440
Embed Views
0

Actions

Likes
0
Downloads
8
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Testing for Educational Gaming and Educational Gaming for Testing Testing for Educational Gaming and Educational Gaming for Testing Presentation Transcript

  • Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)Microsoft Research Connections (Judith Bishop)Testing forEducational Gaming andEducational Gamingfor TestingTao XieNorth Carolina State Universitycontact: taoxie@gmail.com
  • Testing  Educational GamingDynamic Symbolic Execution (Pex)Parameterized Unit TestingPex for Fun (Pex4Fun)TrainingSupporthttp://research.microsoft.com/pex/ http://pex4fun.com
  • AgendaBackgroundPex, Dynamic Symbolic Execution,Parameterized Unit TestingWriting Code in a BrowserCoding DuelsEducational PlatformConclusion
  • PEX, DYNAMIC SYMBOLIC EXECUTION,PARAMETERIZED UNIT TESTINGBackground
  • http://research.microsoft.com/pex/
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}InputnullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullInputnullExecute&MonitorObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnullExecute&MonitorChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0FTFa==nullConstraints to solvea!=nullInputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0FTFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullTConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullTConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890Done: There is no path left.Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  • Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}for any list,for any item,
  • Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}… adding 1 itemincreases Count by 1
  • Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  • Parameterized Unit Test =Unit Test with ParametersVision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  • Parameterized Unit Test =Unit Test with ParametersSeparation of concernsSpecification of behaviorData to achieve coverageVision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  • Parameterized Unit Testing in PracticeGetting PopularParameterized Unit Tests (PUTs) commonlysupported by various test frameworks• .NET: Supported by .NET test frameworks• Java: Supported by JUnit 4.XGenerating test inputs for PUTs supported by tools• .NET: Supported by Microsoft Research Pex• Java: Supported by Agitar AgitarOne(http://www.agitar.com/)
  • http://pex4fun.com/CoverMe
  • Limitations16K characters of code (single file / editor window)*Single-threaded onlyNo environment interactions*No non-determinism*Try to avoid floating-point computationsMax. 20s total exploration time** Limitation only applies to pex4fun.com, but not standalone Pex tool
  • Dynamic Symbolic ExecutionSummary“Ask Pex” sends code to cloudCode is compiled and analyzed in cloudDynamic Symbolic Execution automatically findsrelevant interesting test inputs that achieve highcode coverageResults are shown in browser
  • WRITING CODEIN A BROWSERCode Auto-Completion
  • Auto-Completion
  • Writing Code in BrowserSummaryCode is compiled in cloudCode is executed in cloudAuto-completion via cloud/Javascriptin browser
  • CODING DUELSFun and Engaging Serious Game – Win by Writing Code
  • Coding Duels1,168,577 clicked Ask Pex!
  • Coding DuelsPex computes “semantic diff” in cloudcode written in browser vs.secret reference implementationYou win when Pex finds no differences
  • Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl18
  • Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl18
  • Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl18
  • Coding DuelsFun and EngagingIterative gameplayAdaptivePersonalizedNo cheatingClear winning criterion
  • Example User Feedback“It really got me *excited*. The part that got me most isabout spreading interest in teaching CS: I do think that it’sREALLY great for teaching | learning!”“I used to love the first person shooters and thesatisfaction of blowing away a whole team ofNoobies playing Rainbow Six, but this is far morefun.”“I’m afraid I’ll have to constrain myself to spend just an houror so a day on this really exciting stuff, as I’m really stuffedwith work.”Released since 2010X
  • Coding Duel Competition@ICSE 2011http://pexforfun.com/icse2011
  • Coding DuelsSummaryCoding Duel: write function that matchesspecification given by another implementationSemantic equivalence checked in cloudStudent’s tasks:Infer likely intended behaviorExpress behavior as codehttp://pex4fun.com/CreatingAndPublishingCodingDuelshttp://pex4fun.com/TipsAndTricksForGreatDuels
  • SOCIAL EXPERIENCEConnecting Students – Competitive yet Self-Paced Environment
  • Social Experience
  • Social ExperienceSummaryCommunityHigh score lists, leaderboardLive feedhttp://pex4fun.com/Community.aspxhttp://pex4fun.com/Livefeed.aspx
  • PEX4FUN, AN EDUCATIONAL PLATFORMPages – Courses – Automated Assessments
  • Teaching and Learning
  • Coding Duels for Automatic Grading@Grad Software Engineering Coursehttp://pexforfun.com/gradsofteng
  • Coding Duels for Training Testingpublic static string Puzzle(int[] elems, int capacity, int elem) {if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1)))return "Assumption Violation!";Stack s= new Stack(capacity);for (int i = 0; i < elems.Length; i++)s.Push(elems[i]);int origSize = s.GetNumOfElements();//Please fill in below test scenario on the s stack//The lines below include assertions to assert the program behaviorPexAssert.IsTrue(s.GetNumOfElements() == origSize + 1);PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty());PexAssert.IsTrue(s.IsMember(elem));return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “+ s.IsMember(elem).ToString() + "; " + s.IsEmpty();}Set up a stack with some elementsCache values used in assertions
  • Usage Scenarios of Pex4Fun• Massive Open Online Courses (MOOC): Challenges– Grading, addressed by Pex4Fun– Cheating [Open Challenge]• Course assignments (students/professionals)– E.g., intro programming, software engineering• Student/professional competitions– E.g., coding-duel competition at ICSE 2011• Assessment of testing/programming/problemsolving skills for job applicants– Not just final results of problem solving but also process!
  • Teaching and LearningSummaryHow to become a teacher:Sign in, choose nicknameSend us email with nicknameTeachers canCreate coursesReuse existing or author existing pagesGet automated assessments of studentsSee students codehttp://pex4fun.com/TeachingWithPexForFun
  • OutlookAnalyzing code submissionsHow to guide students?Detect plagiarismFurther rankingSupport for more languages
  • More ReadingNikolai Tillmann, Jonathan De Halleux, Tao Xie,Sumit Gulwani and Judith BishopTeaching and Learning Programming andSoftware Engineering via Interactive GamingIn Proceedings of the 35th InternationalConference on Software Engineering (ICSE 2013),Software Engineering Education (SEE), SanFrancisco, CA, May 2013.http://people.engr.ncsu.edu/txie/publications/icse13see-pex4fun.pdf
  • ConclusionPexTest generation for .NETbased on dynamic symbolic executionEnables Parameterized Unit TestingPex4Fun on the web.NET: C#, Visual Basic, F#Execution, code analysis with PexCoding DuelsGreat for teaching, learninghttp://pex4fun.comhttp://research.microsoft.com/pex
  • Testing  Educational GamingTrainingSupporthttp://research.microsoft.com/pex/ http://pex4fun.comQ & AThank you!contact: taoxie@gmail.comAcknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603,a Microsoft Research SEIF Award, and a Microsoft Research Award.