Synergy of Human and Artificial Intelligence in Software Engineering

  • 3,609 views
Uploaded on

Keynote Talk by Tao Xie at International NSF sponsored Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE 2013) http://promisedata.org/raise/2013/

Keynote Talk by Tao Xie at International NSF sponsored Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE 2013) http://promisedata.org/raise/2013/

More in: Technology , Spiritual
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,609
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
36
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • http://techyyouth.com/google-driverless-car/http://www.lackuna.com/2012/03/15/microsoft-develops-translation-tool-that-mimics-users-own-voice/
  • “Even highly automated systems, such as electric power networks, need human beings... one can draw the paradoxical conclusion that automated systems still are man-machine systems, for which both technical and human factors are important.’http://spectrum.ieee.org/computing/software/automated-to-deathhttp://spectrum.ieee.org/riskfactor/computing/it/are-we-automating-ourselves-into-a-corner
  • “Even highly automated systems, such as electric power networks, need human beings... one can draw the paradoxical conclusion that automated systems still are man-machine systems, for which both technical and human factors are important.’http://spectrum.ieee.org/computing/software/automated-to-deathhttp://spectrum.ieee.org/riskfactor/computing/it/are-we-automating-ourselves-into-a-corner
  • To study what problems
  • To study what problems
  • To study what problems
  • http://www.google.com.hk/url?sa=t&source=web&cd=1&ved=0CBwQFjAA&url=https%3A%2F%2Fwww.iam.unibe.ch%2Fscg%2Fsvn_repos%2FLectures%2FArchive%2FOORPT-W05%2F11ArchitectureRecovery.ppt&ei=PnLPTf7CBoGGuQPu0cCTCg&usg=AFQjCNF7yXlmr5r5rUd8NEf2YYltA8-erA
  • To study what problems
  • To study what problems
  • To study what problems
  • To study what problems
  • To study what problems
  • To study what problems
  • http://developers.de/blogs/damir_dobric/archive/2009/04/13/using-of-factory-in-pex.aspx
  • http://geekswithblogs.net/thomasweller/archive/2010/04/28/mocking-the-unmockable-using-microsoft-moles-with-gallio.aspx
  • http://www.mysvc.it/myapps/constraints/
  • http://www.mysvc.it/myapps/constraints/

Transcript

  • 1. Tao XieNorth Carolina State UniversityRaleigh, NC, USARAISE 2013
  • 2. IBMs Deep Blue defeated chess championGarry Kasparov in 1997IBMWatson defeated top human Jeopardy!players in 2011
  • 3. Google’s driverless carMicrosofts instant voice translation toolIBMWatson as Jeopardy! player
  • 4. "Completely AutomatedPublic Turing test to tellComputers and HumansApart"
  • 5. Movie: Minority ReportCNN NewsiPad
  • 6.
  • 7.  Machine is better at task set A Mechanical, tedious, repetitive tasks, … Ex. solving constraints along a long path Human is better at task set B Intelligence, human intent, abstraction, domainknowledge, … Ex. local reasoning after a loop, recognizing namingsemantics= A U B 8
  • 8. Malaysia Airlines Flight 124@2005Lisanne Bainbridge, "Ironies of Automation”,Automatica 1983 .Ironies of Automation“Even highly automated systems, such aselectric power networks, need human beings...one can draw the paradoxical conclusion thatautomated systems still are man-machinesystems, for which both technical and humanfactors are important.”“As the plane passed 39 000 feet, the stall and overspeedwarning indicators came on simultaneously—somethingthat’s supposed to be impossible, and a situation the crewis not trained to handle.” IEEE Spectrum 2009
  • 9. Malaysia Airlines Flight 124@2005Lisanne Bainbridge, "Ironies of Automation”,Automatica 1983 .Ironies of Automation“The increased interest in human factors amongengineers reflects the irony that the moreadvanced a control system is, so the morecrucial may be the contribution of the humanoperator.”
  • 10.  Don’t forget human intelligence Using your tools as end-to-end solutions Helping your tools Don’t forget cooperations of human and toolintelligence; human and human intelligence Human can help your tools too Human and human could work together to help yourtools, e.g., crowdsourcing11
  • 11.  Don’t forget human intelligence Using your tools as end-to-end solutions Helping your tools Don’t forget cooperations of human and toolintelligence; human and human intelligence Human can help your tools too Human and human could work together to help yourtools, e.g., crowdsourcing12
  • 12. 14“During the past 21 years, over 75 papers and 9Ph.D. theses have been published on pointeranalysis.Given the tones of work on this topicone may wonder, “Havent we solved thisproblem yet?With input from manyresearchers in the field, this paper describesissues related to pointer analysis and remainingopen problems.”Michael Hind. Pointer analysis: havent we solved this problem yet?. In Proc.ACMSIGPLAN-SIGSOFT Workshop on Program Analysis for SoftwareTools andEngineering (PASTE 2001)
  • 13. 15Section 4.3 Designing an Analysis for a Client’s Needs“Barbara Ryder expands on this topic: “…We can all writean unbounded number of papers that compare differentpointer analysis approximations in the abstract.However, this does not accomplish the key goal, which isto design and engineer pointer analyses that are usefulfor solving real software problems for realisticprograms.”
  • 14. 17Zhenmin Li, Shan Lu, Suvda Myagmar, andYuanyuan Zhou. CP-Miner: a tool forfinding copy-paste and related bugs inoperating system code. In Proc. OSDI2004.MSRAXIAOYingnong Dang, Dongmei Zhang, Song Ge,Chengyun Chu,Yingjun Qiu, andTao Xie.XIAO:Tuning Code Clones at Hands ofEngineers in Practice. In Proc. ACSAC2012,MSR 2011 Keynote byYY Zhou: ConnectingTechnology withReal-world Problems – From Copy-paste Detection toDetecting Known BugsHuman Intelligence toDetermine What areSerious Bugs
  • 15. 18Available inVisual Studio 2012Searching similar snippets forfixing bug onceFinding refactoringopportunityYingnong Dang, Dongmei Zhang, SongGe,YingjunQiu, andTao Xie. XIAO:Tuning Code Clones at Hands of Engineers inPractice. In Proc. AnnualComputer Security ApplicationsConference (ACSAC 2012)XIAO Code Clone Search service integrated into workflow ofMicrosoft Security Response Center (MSRC)MicrosoftTechnet Blog about XIAO:We wanted to be sure to address the vulnerable code wherever it appearedacross the Microsoft code base.To that end, we have been working withMicrosoft Research to develop a “Cloned Code Detection” system that we canrun for every MSRC case to find any instance of the vulnerablecode in any shipping product.This system is the one that found several ofthe copies of CVE-2011-3402 that we are now addressing with MS12-034.
  • 16. 19XIAO enables code clone analysis withHigh scalability, High compatibilityHigh tunability: what you tune is what you getHigh explorability:1. Clone navigation based on source tree hierarchy2. Pivoting of folder level statistics3. Folder level statistics4. Clone function list in selected folder5. Clone function filters6. Sorting by bug or refactoring potential7. Tagging1 2 3 4 5 671. Block correspondence2. Block types3. Block navigation4. Copying5. Bug filing6. Tagging1234165How to navigate through the largenumber of detected clones? How to quickly review a pair of clones?
  • 17.  50 years of automated debugging research N papers  only 5 evaluated with actual programmers“”Chris Parnin and Alessandro Orso. Are automated debugging techniques actually helpingprogrammers?. In Proc. ISSTA 2011
  • 18.  Academia Tend to leave human out of loop (involving human makesevaluations difficult to conduct or write) Tend not to spend effort on improving tool usability▪ tool usability would be valued more in HCI than in SE▪ too much to include both the approach/tool itself and usability/its evaluationin a single paper Real-world Often has human in the loop (familiar IDE integration, socialeffect, lack of expertise/willingness to write specs,…) Examples Agitar [ISSTA 2006] vs. Daikon [TSE 2001] Test generation in Pex based on constraint solving
  • 19.  Goal: to identify the future directions in researchin formal methods and its transition to industrialpractice. The workshop will bring together researchers andidentify primary challenges in the field, bothfoundational, infrastructural, and in transitioningideas from research labs to developer tools.http://goto.ucsd.edu/~rjhala/NSFWorkshop/
  • 20.  “Lack of education amongst practitioners” “Education of students in logic and design forverification” “Expertise required to create and use a verificationtool. E.g., both Astre for Airbus and SDV forWindows drivers were closely shepherded byverification experts.” “Tools require lots of up-front effort (e.g., to writespecifications)” “User effort required to guide verification tools,such as assertions or specifications”
  • 21.  “Not integrated with standard development flows(testing)” “Too many false positives and no ranking of errors” “General usability of tools, in terms of false alarmsand error messages.The Coverity CACM paperpointed out that they had developed features thatthey do not deploy because they baffle users.Many tools choose unsoundness over soundness toavoid false alarms.”
  • 22.  “The necessity of detailed specifications andcomplex interaction with tools, which is very costlyand discouraging for industrial, who lack high-levelspecialists.” “Feedback to users. It’s difficult to explain to userswhy automated verification tools are failing.Counterexamples to properties can be very difficultfor users to understand, especially when they areabstract, or based on incomplete environmentmodels or constraints.”
  • 23. 2010 Dagstuhl Seminar 10111Practical Software Testing: Tool Automation and Human Factorshttp://www.dagstuhl.de/programm/kalender/semhp/?semnr=1011
  • 24. 2010 Dagstuhl Seminar 10111Practical Software Testing: Tool Automation and Human FactorsHuman Factorshttp://www.dagstuhl.de/programm/kalender/semhp/?semnr=1011
  • 25. Andy Ko and Brad Myers. Debugging Reinvented: Asking and Answering Why and Why NotQuestions about Program Behavior. In Proc. ICSE 2008
  • 26.  Don’t forget human intelligence Using your tools as end-to-end solutions Helping your tools Don’t forget cooperations of human and toolintelligence; human and human intelligence Human can help your tools too Human and human could work together to help yourtools, e.g., crowdsourcing29
  • 27.  Motivation Architecture recovery is challenging (abstraction gap) Human typically has high-level view in mind Repeat Human: define/update high-level model of interest Tool: extract a source model Human: define/update declarative mapping betweenhigh-level model and source model Tool: compute a software reflexion model Human: interpret the software reflexion modelUntil happyGail C. Murphy, David Notkin. Reengineering with Reflection Models: A Case Study. IEEE Computer 1997
  • 28. Running Symbolic PathFinder ...…====================================================== resultsno errors detected====================================================== statisticselapsed time: 0:00:02states: new=4, visited=0,backtracked=4, end=2search: maxDepth=3,constraints=0choice generators: thread=1, data=2heap: gc=3, new=271, free=22instructions: 2875max memory: 81MBloaded code: classes=71, methods=884…31
  • 29.  object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.32 Ex: Dynamic Symbolic Execution (DSE) /ConcolicTesting[Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] Instrument code to explore feasible paths Challenge: path explosion When desirablereceiver or argumentobjects are notgenerated
  • 30. 33 A graph example fromQuickGraph library Includes two classesGraphDFSAlgorithm GraphAddVertexAddEdge: requiresboth vertices to bein graph00: class Graph { …03: public void AddVertex (Vertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (Vertex v1, Vertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (Vertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } 33[OOPSLA 11]
  • 31. 34 Test target: Cover truebranch (B4) of Line 24 Desired objectstate: graph shouldinclude at least oneedge Target sequence:Graph ag = new Graph();Vertex v1 = new Vertex(0);Vertex v2 = new Vertex(1);ag.AddVertex(v1);ag.AddVertex(v2);ag.AddEdge(v1, v2);DFSAlgorithm algo = newDFSAlgorithm(ag);algo.Compute(v1);3400: class Graph { …03: public void AddVertex (Vertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (Vertex v1, Vertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (Vertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } }[OOPSLA 11]
  • 32.  object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.35 Ex: Dynamic Symbolic Execution (DSE) /ConcolicTesting[Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] Instrument code to explore feasible paths Challenge: path explosionTypically DSE instruments or exploresonly methods @ project under test;Third-party API external methods(network, I/O, ..):•too many paths•uninstrumentable
  • 33. 36
  • 34. Total block coverage achieved is 50%, lowest coverage 16%.37 Ex: Dynamic Symbolic Execution (DSE) /ConcolicTesting[Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08] Instrument code to explore feasible paths Challenge: path explosionXusheng Xiao,Tao Xie, NikolaiTillmann, and Jonathan de Halleux. Precise Identification of Problems forStructural Test Generation. In Proc. ICSE 2011
  • 35. 2010 Dagstuhl Seminar 10111Practical SoftwareTesting: Tool Automation and Human Factors
  • 36.  Tackling object-creation problems Seeker [OOSPLA 11] , MSeqGen [ESEC/FSE 09]Covana [ICSE 2011], OCAT [ISSTA 10]Evacon [ASE 08], Symclat [ASE 06] Still not good enough (at least for now)!▪ Seeker (52%) > Pex/DSE (41%) > Randoop/random (26%) Tackling external-method call problems DBAppTesting [ESEC/FSE 11], [ASE 11] CloudAppTesting [IEEE Soft 12] Deal with only common environment APIs@NCSUASE
  • 37. 40 Test target: Cover truebranch (B4) of Line 24 Desired objectstate: graph shouldinclude at least oneedge Target sequence:Graph ag = new Graph();Vertex v1 = new Vertex(0);Vertex v2 = new Vertex(1);ag.AddVertex(v1);ag.AddVertex(v2);ag.AddEdge(v1, v2);DFSAlgorithm algo = newDFSAlgorithm(ag);algo.Compute(v1);4000: class Graph { …03: public void AddVertex (Vertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (Vertex v1, Vertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (Vertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } }
  • 38. Tackle object-creation problems with Factory Methods41
  • 39. Tackle external-method call problems with Mock Methods orMethod InstrumentationMocking System.IO.File.ReadAllText42
  • 40.  Human-Assisted Computing Driver: tool Helper: human Ex. Covana [Xiao et al. ICSE 2011] Human-Centric Computing Driver: human  Helper: tool Ex. Coding duels @Pex for FunInterfaces are important. Contents are important too!43
  • 41.  Motivation Tools are often not powerful enough Human is good at some aspects that tools are not What difficulties does the tool face? How to communicate info to the user to get help? How does the user help the tool based on the info?44Iterations to form Feedback Loop
  • 42.  Human-Assisted Computing Driver: tool Helper: human Ex. Covana [Xiao et al. ICSE 2011] Human-Centric Computing Driver: human  Helper: tool Ex. Coding duels @Pex for FunInterfaces are important. Contents are important too!45
  • 43. 1,230,309 clicked Ask Pex!www.pexforfun.com46NikolaiTillmann, Jonathan De Halleux,Tao Xie, Sumit Gulwani and Judith Bishop.Teaching and Learning Programming and Software Engineering via InteractiveGaming. In Proc. ICSE 2013 SEE.
  • 44. Secret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}classTest {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl47
  • 45.  Coding duels at http://www.pexforfun.com/ Brain exercising/learning while having fun Fun: iterative, adaptive/personalized, w/ win criterion Abstraction/generalization, debugging, problem solvingBrain exercising
  • 46. http://pexforfun.com/gradsoftengObserved Benefits• Automatic Grading• Real-time Feedback (for Both Students andTeachers)• Fun Learning Experiences
  • 47. “It really got me *excited*.The part that got me most isabout spreading interest in teaching CS: I do think that it’sREALLY great for teaching | learning!”“I used to love the first person shooters and thesatisfaction of blowing away a whole team ofNoobies playing Rainbow Six, but this is far morefun.”“I’m afraid I’ll have to constrain myself to spend just an houror so a day on this really exciting stuff, as I’m really stuffedwith work.”X
  • 48. 52Internetclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1); } } Everyone cancontribute Coding duels Duel solutions
  • 49. InternetPuzzle Games Made fromDifficult Constraints or Object-Creation ProblemsSupported by MSR SEIFAwardNing Chen and Sunghun Kim. Puzzle-based Automatic Testing: bringing humans into the loop bysolving puzzles. In Proc. ASE 2012
  • 50. http://www.cs.washington.edu/verigames/
  • 51. 55Pattern MatchingBug updateProblematicPattern RepositoryBug DatabaseTrace analysisBugfilingStackMine [Han et al. ICSE 12]Trace StorageTrace collectionInternetShi Han,Yingnong Dang, Song Ge, Dongmei Zhang, and Tao Xie. Performance Debugging in the Largevia Mining Millions of StackTraces. In Proc. ICSE 2012
  • 52. “We believe that the MSRA tool is highly valuable andmuch more efficient for mass trace (100+ traces) analysis.For 1000 traces, we believe the tool saves us 4-6 weeks oftime to create new signatures, which is quite a significantproductivity boost.”- from Development Manager inWindowsHighly effective new issue discovery onWindows mini-hangContinuous impact on futureWindows versionsShi Han,Yingnong Dang, Song Ge, Dongmei Zhang, and Tao Xie. Performance Debugging in the Largevia Mining Millions of StackTraces. In Proc. ICSE 2012
  • 53.  Don’t forget human intelligence Using your tools as end-to-end solutions Helping your tools Don’t forget cooperations of human and toolintelligence; human and human intelligence Human can help your tools too Human and human could work together to help yourtools, e.g., crowdsourcing57
  • 54.  Human-Assisted Computing Human-Centric Computing Human-Human Cooperation
  • 55.  Wonderful current/former students@NCSU ASE Collaborators, especially those from MicrosoftResearch Redmond/Asia, Peking University Colleagues who gave feedback and inspired meNSF grants CCF-0845272, CCF-0915400, CNS-0958235, ARO grant W911NF-08-1-0443, anNSA Science of Security, Lablet grant, a NIST grant, a 2011 Microsoft Research SEIFAward
  • 56. Questions ?https://sites.google.com/site/asergrp/