Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Transferring Software Testing and Analytics Tools to Practice

1,031 views

Published on

Keynote Talk in the Workshop on Testing: Academia-Industry Collaboration, Practice and Research Techniques (TAIC PART 2016) http://www2016.taicpart.org/

Published in: Software
  • Be the first to comment

Transferring Software Testing and Analytics Tools to Practice

  1. 1. Tao Xie University of Illinois at Urbana-Champaign Part of the research work described in this talk was done in collaboration with the Pex team and SoftwareAnalytics Group @Microsoft Research, students @IllinoisASE, and other collaborators
  2. 2.  Research impact: inspiring/impactful ideas/directions/subareas… for researchers  Example: model checking  Practice impact: Practice adoption of tools/systems/technologies… for practitioners  Some examples discussed in this talk  Societal impact: inspiring/impactful ideas/thinking/awareness… for general public  Example: computational thinking, privacy, medical- device security, MOOCs, …
  3. 3.  Publishing research results  technologies there adopted by companies, e.g., ICSE 00 Daikon paper by Ernst et al.  Agitar Agitator https://homes.cs.washington.edu/~mernst/pubs/invariants-relevance-icse2000.pdf ASE 04 Rostra paper by Xie et al.  Parasoft Jtest improvement http://taoxie.cs.illinois.edu/publications/ase04.pdf PLDI/FSE 05 DART/CUTE papers by Sen et al.  MSR SAGE, Pex http://srl.cs.berkeley.edu/~ksen/papers/dart.pdf http://srl.cs.berkeley.edu/~ksen/papers/C159-sen.pdf …
  4. 4.  Commercializing research results in startup  tools/products used by companies, e.g., Reactis® …
  5. 5.  Transferring research results to product groups  tools/products used inside company or outside, e.g., SAGE Flash Fill … CloudBuild Tools for Software Engineers Fakes
  6. 6.  Release open source infrastructures or libraries to engage academic/industrial communities to use and contribute, e.g., ▪ MPI/PETSc by Bill Gropp et al. ▪ Charm++ by Laxmikant (Sanjay) Kale et al. ▪ LLVM byVikram Adve, Chris Lattner, et al. “The openness of the LLVM technology and the quality of its architecture and engineering design are key factors in understanding the success it has had both in academia and industry.” KLEE? JPF?FindBugs?Shipshape? Soot? WALA? …
  7. 7.  50 years of automated debugging research  N papers  only 5 evaluated with actual programmers “ ” [Parnin&Orso ISSTA’11] http://dl.acm.org/citation.cfm?id=2001445
  8. 8.  Human  Expensive, incomplete, …  Brute Force  Pairwise, predefined data, etc…  Tool Automation!!
  9. 9. Running Symbolic PathFinder ... … ============================================ ========== results no errors detected ============================================ ========== statistics elapsed time: 0:00:02 states: new=4, visited=0, backtracked=4, end=2 search: maxDepth=3, constraints=0 choice generators: thread=1, data=2 heap: gc=3, new=271, free=22 instructions: 2875 max memory: 81MB loaded code: classes=71, methods=884 … 9
  10. 10.  Pex (released on May 2008)  Shipped with Visual Studio 15 as IntelliTest  30,388 download# (20 months, Feb 08-Oct 09)  22,466 download# (10 months, Apr 13-Jan 14): Code Digger  Active user community: 1,436 forum posts during ~3 years (Oct 08- Nov 11)  Moles (released on Sept 2009)  Shipped with Visual Studio 12 as Fakes  “Provide Microsoft Fakes w/ allVisual Studio editions” got 1,457 community votes
  11. 11. https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/6773265-make-intellitest-available-to-visual-studio-profes https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/2216195-include-pex-and-moles-with-all-visual-studio-editi
  12. 12. 1,753,594 clicked 'Ask Pex!' http://pex4fun.com/
  13. 13. https://www.codehunt.com/
  14. 14. Code Hunt can identify top coders http://programming2015.cstnet.cn/
  15. 15. Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Player Implementation class Player { public static int Puzzle(int x) { return x; } } classTest { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } } behavior Secret Impl == Player Impl 15
  16. 16. “It really got me *excited*.The part that got me most is about spreading interest in teaching CS: I do think that it’s REALLY great for teaching | learning!” “I used to love the first person shooters and the satisfaction of blowing away a whole team of Noobies playing Rainbow Six, but this is far more fun.” “I’m afraid I’ll have to constrain myself to spend just an hour or so a day on this really exciting stuff, as I’m really stuffed with work.” X
  17. 17.  NOT Random:  Cheap, Fast  “It passed a thousand tests” feeling  …  But Dynamic Symbolic Execution: e.g., Pex, CUTE,EXE  White box  Constraint Solving
  18. 18. Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T TF T F F Execute&MonitorSolve Choose next path Done: There is no path left. Negated condition
  19. 19. There are decision procedures for individual path conditions, but…  Number of potential paths grows exponentially with number of branches  Reachable code not known initially  Without guidance, same loop might be unfolded forever Fitnex search strategy [Xie et al. DSN 09] http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
  20. 20. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(0, {0}) Path condition: !(x == 90) ↓ New path condition: (x == 90) ↓ New test input: TestLoop(90, {0})
  21. 21. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(90, {0}) Path condition: (x == 90) && !(y[0] ==15) && !(x == 110) ↓ New path condition: (x == 90) && (y[0] ==15) ↓ New test input: TestLoop(90, {15})
  22. 22. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(90, {15}) Path condition: (x == 90) && (y[0] ==15) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] ==15) && (x+1 == 110) ↓ New test input: No solution!?
  23. 23. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(90, {15}) Path condition: (x == 90) && (y[0] ==15) && (0 < y.Length) && !(1 < y.Length) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] ==15) && (0 < y.Length) && (1 < y.Length)  Expand array size
  24. 24. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(90, {15}) We can have infinite paths! Manual analysis  need at least 20 loop iterations to cover the target branch Exploring all paths up to 20 loop iterations is infeasible: 220 paths
  25. 25. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Key observations: with respect to the coverage target  not all paths are equally promising for branch-node flipping  not all branch nodes are equally promising to flip • Our solution: – Prefer to flip branch nodes on the most promising paths – Prefer to flip the most promising branch nodes on paths – Fitness function to measure “promising” extents TestLoop(90, {15, 0}) TestLoop(90, {15, 15}) [Xie et al. DSN 2009] http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
  26. 26.  FF computes fitness value (distance between the current state and the goal state)  Search tries to minimize fitness value [Tracey et al. 98, Liu at al. 05, …]
  27. 27. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Fitness function: |110 – x |
  28. 28. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } (90, {0}) 20 (90, {15}) 19 (90, {15, 0}) 19 (90, {15, 15}) 18 (90, {15, 15, 0}) 18 (90, {15, 15, 15}) 17 (90, {15, 15, 15, 0}) 17 (90, {15, 15, 15, 15}) 16 (90, {15, 15, 15, 15, 0}) 16 (90, {15, 15, 15, 15, 15}) 15 … FitnessValue(x, y) Fitness function: |110 – x | Give preference to flip paths with better fitness values We still need to address which branch node to flip on paths …
  29. 29. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } (90, {0}) 20 (90, {15})  flip b4 19 (90, {15, 0})  flip b2 19 (90, {15, 15})  flip b4 18 (90, {15, 15, 0})  flip b2 18 (90, {15, 15, 15})  flip b4 17 (90, {15, 15, 15, 0})  flip b2 17 (90, {15, 15, 15, 15})  flip b4 16 (90, {15, 15, 15, 15, 0})  flip b2 16 (90, {15, 15, 15, 15, 15})  flip b4 15 … FitnessValue(x, y) Fitness function: |110 – x | Branch b1: i < y.Length Branch b2: i >= y.Length Branch b3: y[i] == 15 Branch b4: y[i] != 15 •Flipping Branch b4 (b3) gives us average 1 (-1) fitness gain (loss) •Flipping branch b2 (b1) gives us average 0 fitness gain (loss)
  30. 30.  For a flipped node leading to Fnew, find out the old fitness value Fold before flipping • Assign Fitness Gain (Fold – Fnew) for the branch of the flipped node • Assign Fitness Gain (Fnew – Fold ) for the other branch of the branch of the flipped node  Compute the average fitness gain for each branch over time
  31. 31.  Each branch node candidate for being flipped is prioritized based on its composite fitness value: • (Fitness value of node – Fitness gain of its branch)  Select first the one with the best composite fitness value
  32. 32.  Pex (released on May 2008)  Shipped with Visual Studio 15 as IntelliTest  30,388 download# (20 months, Feb 08-Oct 09)  22,466 download# (10 months, Apr 13-Jan 14): Code Digger  Active user community: 1,436 forum posts during ~3 years (Oct 08- Nov 11)  Moles (released on Sept 2009)  Shipped with Visual Studio 12 as Fakes  “Provide Microsoft Fakes w/ allVisual Studio editions” got 1,457 community votes How to make such successful case????
  33. 33. voidTestAdd(ArrayList a, object o) { Assume.IsTrue(a!=null); int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o); } Parameterized UnitTests Supported by Pex Moles/Fakes Code Digger Pex4Fun/Code Hunt  Surrounding (Moles/Fakes)  Simplifying (Code Digger)  Retargeting (Pex4Fun/Code Hunt)
  34. 34.  Developer/manager: “Who is using your tool?”  Pex team: “Do you want to be the first?”  Developer/manager: “I love your tool but no.” Tool Adoption by (Mass)Target Users Tool Shipping withVisual Studio Macro Perspective Micro Perspective
  35. 35.  Developer: “Code digger generates a lot of “0” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string.”  Pex team: “In C#, a 0 in a string does not mean zero- termination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “0”.”  Developer: “Your tool generated “0””  Pex team: “What did you expect?”  Developer: “Marc.”
  36. 36.  Developer: “Your tool generated a test called Foo001. I don’t like it.”  Pex team: “What did you expect?”  Developer:“Foo_Should_Fail_When_Bar_Is_Negative.”
  37. 37. Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’11]) ExceptionTreeView Exploration TreeView Exploration ResultsView http://taoxie.cs.illinois.edu/publications/icse11-covana.pdf
  38. 38. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Key observations: with respect to the coverage target  not all paths are equally promising for branch-node flipping  not all branch nodes are equally promising to flip • Our solution: – Prefer to flip branch nodes on the most promising paths – Prefer to flip the most promising branch nodes on paths – Fitness function to measure “promising” extents Fitnex by Xie et al. [DSN’09] To avoid local optimal or biases, the fitness-guided strategy is integrated with Pex’s fairness search strategies http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
  39. 39.  “Simply one mouse click and then everything would work just perfectly”  Often need environment isolation w/ Moles/Fakes or factory methods, …  “One mouse click, a test generation tool would detect all or most kinds of faults in the code under test”  Developer: “Your tool only finds null references.”  Pex team: “Did you write any assertions?”  Developer: “Assertion???”  “I do not need test generation; I already practice unit testing (and/orTDD).Test generation does not fit into the TDD process”
  40. 40. Gathered feedback from target tool users  Directly, e.g., via  MSDN Pex forum, tech support, outreach to MS engineers and .NET user groups  Indirectly, e.g., via  interactions with MSVisual Studio team (a tool vendor to its huge user base)  Motivations of Moles  Refactoring testability issue faced resistance in practice  Observation at Agile 2008: high attention on mock objects and tool supports
  41. 41.  Win-win collaboration model  Win (Ind Lab): longer-term research innovation, man power, research impacts, …  Win (Univ): powerful infrastructure, relevant/important problems in practice, both research and industry impacts, …  Industry-located Collaborations  Faculty visits, e.g., Fitnex, Pex4Fun  Student internships, e.g., FloPSy, DyGen, state cov  Academia-locatedCollaborations
  42. 42. Academia-located Collaborations  Immediate indirect impacts, e.g.,  Reggae [ASE’09s]  Rex  MSeqGen [FSE’09]  DyGen  Guided Cov [ICSM’10]  state coverage  Long-term indirect impacts, e.g.,  DySy by Csallner et al. [ICSE’08]  Seeker [OOPSLA’11]  Covana [ICSE’11]
  43. 43.  Pex  practice impacts  Moles/Fakes, Code Digger, Pex4Fun/Code Hunt  Lessons in transferring tools  Started as (Evolved) Dream  Chicken and Egg  Human Factors  Best vs.Worst Cases  Tool Users’ Stereotypical Mindset or Habits  Practitioners’Voice  Collaboration w/ Academia
  44. 44.  Start a startup  but desirable to have right people (e.g., former students) to start  Release free tools/libraries to aim for adoption  but a lot of efforts to be invested on “non-researchy” stuffs  Collaborate with industrial research labs  but many research lab projects may look like univ. projects  Collaborate with industrial product groups  but many probs faced by product groups may not be “researchy”
  45. 45.  NikolaiTillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience Papers. http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf  Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013, Experience Paper. http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf  Dongmei Zhang, Shi Han,Yingnong Dang, Jian-Guang Lou, Haidong Zhang, andTao Xie. Software Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics, 2013. http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf  Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu,Yingjun Qiu, and Tao Xie. XIAO:Tuning Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC 2012. http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf
  46. 46. Questions ? https://sites.google.com/site/asergrp/ http://research.microsoft.com/pex http://research.microsoft.com/sa/ taoxie@illinois.edu
  47. 47.  Start a startup  but desirable to have right people (e.g., former students) to start  Release free tools/libraries to aim for adoption  but a lot of efforts to be invested on “non-researchy” stuffs  Collaborate with industrial research labs  but many research lab projects may look like univ. projects  Collaborate with industrial product groups  but many probs faced by product groups may not be “researchy”

×