Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automating Software Development Using Artificial Intelligence (AI)

482 views

Published on

In recent years, traditional software development activities have been enhanced through the use of Artificial Intelligence (AI) techniques including genetic algorithms, machine learning and deep learning. The use cases for AI in software development have ranged from developer recommendations to complete automation of software developer activities. To demonstrate the breadth of application, I will present several recent examples of how AI can be leveraged to automate software development. First, I will present an approach to predicting future code changes in GitHub projects using historical data and machine learning. Next, I will present our framework for repairing multi-threaded software bugs using genetic algorithms. I will conclude with a broad discussion of the impact AI is having on software development.

Published in: Software

Automating Software Development Using Artificial Intelligence (AI)

  1. 1. Funding provided by: http://www.sqrlab.ca Faculty of Science (Computer Science) University of Ontario Institute of Technology Oshawa, Ontario, Canada Automating Software Development Using AI Jeremy Bradbury, PhD jeremy.bradbury@uoit.ca
  2. 2. Software Quality Research Lab © 2018 J.S. Bradbury Gabrielle Peres Dias, Michael Miljanovic, Luisa Rojas Garcia Kevil Jalbert Mark Green David Kelk Joseph Heron 2
  3. 3. Defining AI • What is artificial intelligence? • An evolving concept that means different things to different people • What are examples of AI? • Metaheuristic search techniques (e.g., genetic algorithms) • Machine learning (e.g., support vector machines) • Deep learning (and neural networks) © 2018 J.S. Bradbury 3
  4. 4. “ Metaheuristic Definition …a metaheuristic can be seen as a general algorithmic framework which can be applied to different optimization problems with relatively few modifications to make them adapted to a specific problem. [MHN17] [MHN17] Metaheuristic Network. Website: http://www.metaheuristics.net/ (Last accessed: Oct. 17, 2017). © 2018 J.S. Bradbury 4
  5. 5. Example metaheuristic techniques include hill climbing, particle swarm optimizations, genetic algorithms (GAs)… © 2018 J.S. Bradbury 5
  6. 6. Metaheuristic Strategies SOLUTION CONSTRUCTION SOLUTION MODIFICATION SOLUTION RECOMBINATION © 2018 J.S. Bradbury [ZBB10] Gunther Zapfel, Roland Braune, Michael Bogl. “Metaheurustic Search Concepts: A Tutorial with Applications to Production and Logistics.” 2010. 6 • Ant Colony Optimization • Greedy Randomized Adaptive Search Procedure (GRASP) • Hill Climbing • Tabu Search • Simulated Annealing • Genetic Algorithms
  7. 7. Local vs. Global Search Local Search • Scope = local • Strategy = making iterative local changes • Solution = local optima © 2018 J.S. Bradbury 7 Global Search • Scope = global • Strategy = making iterative changes over the entire solution space • Solution = global optima "Location_of_Cape_Verde_in_the _globe.svg" derivative work by Luan and original by Eddo is licensed under CC BY-SA 3.0
  8. 8. The Machine Learning (ML) Process • Machine Learning (ML) techniques can generally be applied to tasks (problems) as follows[Fla12]: [Fla12] Peter Flach. “Machine Learning: The Art and Science of Algorithms that Make Sense of Data.” 2012. © 2018 J.S. Bradbury 8 Training Data (features) ML Algorithm ModelData Output (grouping, grading)
  9. 9. [Par17] David L. Parnas. “The Real Risks of Artificial Intelligence,” Communications of the ACM 60(10), pages 27-31, 2017. Photo from https://alchetron.com/David-Parnas and is licensed under CC BY-SA 3.0 © 2018 J.S. Bradbury 9 “ Learning is not magic, it is the use of data collected during use to improve future performance. – David Parnas
  10. 10. Machine Learning Applications CLASSIFICATION CLUSTERING PREDICTIONREGRESSION OPTIMIZATION © 2018 J.S. Bradbury [Gol16] Sunila Gollapudi. “Practical Machine Learning.” 2016. 10
  11. 11. Machine Learning Methods • Naïve Bayes • Average One-Dependence Estimators (AODE) • Bayesian Belief Network (BBN) • Support Vector Machine (SVM) • Linear Discriminant Analysis (LDA) • Classification & Regression Tree (CART) • Random Forest • K-Means Clustering • Expectation Maximization (EM) • … © 2018 J.S. Bradbury 11
  12. 12. Challenges with ML • Overfitting • When your model (target function) is tailored too much to past (training) data and doesn’t generalise to future data points • You have trained your model too well! • Underfitting • When your model is not trained well enough for modeling the past (training) data and does not generalise to future data points either • You haven’t trained your model well enough! © 2018 J.S. Bradbury 12
  13. 13. Machine Learning REPRESENTATION LEARNING DEEP LEARNING © 2018 J.S. Bradbury 13
  14. 14. “ Representation Learning a set of methods that allows a machine to be fed with raw data and to automatically discover the representations needed for detection or classification. [LBH15] [LBH15] Yann LeCun, Yoshua Bengio, Geoffrey Hinton (2015). Deep learning. Nature, 521(7553), 436–444. © 2018 J.S. Bradbury 14
  15. 15. “ Deep Learning …are representation-learning methods with multiple levels of representation, obtained by composing simple but non- linear modules that each transform the representation at one level (starting with the raw input) into a representation at a higher, slightly more abstract level. [LBH15] [LBH15] Yann LeCun, Yoshua Bengio, Geoffrey Hinton (2015). Deep learning. Nature, 521(7553), 436–444. © 2018 J.S. Bradbury 15
  16. 16. Deep Learning 16© 2018 J.S. Bradbury Input Layer Output Layer Hidden Layers • Not constrained by traditional machine learning’s limitations with respect to processing raw data (which requires expertise and domain knowledge)
  17. 17. Implementations of Deep Learning • Multiple levels/layers of representation learning can be implemented in various ways including: • Deep neural networks • Deep convolution neural networks (ConvNets) • Recurrent neural networks (RNNs) • Deep believe networks • A good framework to use for deep learning is TensorFlow – https://www.tensorflow.org/ © 2018 J.S. Bradbury 17
  18. 18. AI & SE – Understanding the Relationship © 2018 J.S. Bradbury Artificial Intelligence Software Engineering 18
  19. 19. AI & SE – Understanding the Relationship © 2018 J.S. Bradbury Artificial Intelligence Software Engineering AI + SE 19
  20. 20. AI & SE – Understanding the Relationship © 2018 J.S. Bradbury Artificial Intelligence Software Engineering AI + SE How can AI be applied to SE? 20
  21. 21. How can AI be applied to SE? • Automation of software development activities including the creation of software artifacts (e.g., software test generation) • Recommendation systems to assist software developers improve their performance (e.g., recommended code for review) • The software development problems that can be addressed with AI are those that can be reframed in terms of optimization, classification, prediction… 21© 2018 J.S. Bradbury
  22. 22. How can AI be applied to SE? • There are already several vibrant research communities conducting work in this area: • International Symposiums on Search-based Software Engineering (SSBSE), 2009-2018 • International Workshops on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE), 2012-2016, 2018 22© 2018 J.S. Bradbury
  23. 23. EXAMPLE #1 Automatic Bug Repair © 2018 J.S. Bradbury 23
  24. 24. What do we mean by concurrency bugs? • There are many different kinds of concurrency bugs • We focus on two of the most common kinds – data races and deadlocks © 2018 J.S. Bradbury 24
  25. 25. What do we mean by bug repair? • We view bug repair as a source code modification that fixes a concurrency bug while minimizing the effect on performance © 2018 J.S. Bradbury 25
  26. 26. Automatic Repair of Concurrency Bugs • Several SBSE approaches have been proposed to fix bugs in single threaded programs [LDFW12, Arc11] • genetic programming is used to evolve patches, while testing evaluates fitness • These techniques cannot be applied directly to fix concurrency bugs due to the nondeterministic nature of thread scheduling • We adapt this work to handle concurrency bugs by modifying the fitness function and it’s evaluation [LDFW12] C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer, “A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each,” in Proc. of ICSE 2012, Jun. 2012. [Arc11] A. Arcuri, “Evolutionary repair of faulty software,” in Applied Soft. Computing, vol. 11, 2011, pp. 3494–3514. © 2018 J.S. Bradbury 26
  27. 27. Automatic Repair of Concurrency (ARC) PHASE 1: • Repairing Deadlocks and Data Races PHASE 2: • Optimizing the Performance of Repaired Source Code © 2018 J.S. Bradbury Buggy Java program Java program that exhibits bug-free behavior Java program that exhibits bug-free behavior and is performance optimized PHASE 1 PHASE 2 27
  28. 28. ARC PHASE 1 Repairing Deadlocks and Data Races 28© 2018 J.S. Bradbury INPUT 1. Java program with concurrency bugs 2. Set of JUnit tests The test suite is the oracle (hence, the approach is only as good as the tests!)
  29. 29. ARC PHASE 1 Repairing Deadlocks and Data Races 1. Initialize/update population • Create the population for the genetic algorithm (GA) • The first generation is a set of copies of the original buggy program • Subsequent generations will be updated based on the GA (described in future steps) © 2018 J.S. Bradbury 29
  30. 30. ARC PHASE 1 Repairing Deadlocks and Data Races 2. Generate mutants • Use mutation operators to generate mutants for all members of the population • The generated mutants are optimized using the static analysis tool Chord [NA07] • Allows mutation operators to target specific shared classes, methods and variables when generating mutants [NA07] Naik, M., Aiken, A.: Conditional must not aliasing for static race detection. In: Proc. of ACM SIGPLAN-SIGACT Symp. on Principles of Programming Languages (POPL 2007), pp. 327–338, Jan. 2007. © 2018 J.S. Bradbury 30
  31. 31. Mutation Operator1 Description Acronym Add a synchronized block around a statement ASAT Add the synchronized keyword to the method header ASIM Add a synchronized block around a method ASM Change the order of two synchronized blocks order CSO Expand synchronized region after EXSA Expand synchronized region before EXSB Remove synchronized statement around a synchronized statement RSAS Remove synchronization around a variable RSAV Remove synchronized keyword in method header RSIM Remove synchronization block around method RSM Shrink synchronization block after SHSA Shrink synchronization block before SHSB 1All mutation operators are written in the TXL source transformation language – http://www.txl.ca. © 2018 J.S. Bradbury 31
  32. 32. Mutation Operator1 Description Acronym Add a synchronized block around a statement ASAT Add the synchronized keyword to the method header ASIM Add a synchronized block around a method ASM Change the order of two synchronized blocks order CSO Expand synchronized region after EXSA Expand synchronized region before EXSB Remove synchronized statement around a synchronized statement RSAS Remove synchronization around a variable RSAV Remove synchronized keyword in method header RSIM Remove synchronization block around method RSM Shrink synchronization block after SHSA Shrink synchronization block before SHSB 1All mutation operators are written in the TXL source transformation language – http://www.txl.ca. © 2018 J.S. Bradbury 32
  33. 33. Mutation Operator1 Description Acronym Add a synchronized block around a statement ASAT Add the synchronized keyword to the method header ASIM Add a synchronized block around a method ASM Change the order of two synchronized blocks order CSO Expand synchronized region after EXSA Expand synchronized region before EXSB Remove synchronized statement around a synchronized statement RSAS Remove synchronization around a variable RSAV Remove synchronized keyword in method header RSIM Remove synchronization block around method RSM Shrink synchronization block after SHSA Shrink synchronization block before SHSB 1All mutation operators are written in the TXL source transformation language – http://www.txl.ca. © 2018 J.S. Bradbury 33 Program P: obj.write(var1); synchronized(lock) { myHash.remove(var1); } Program P’: synchronized(lock) { obj.write(var1); myHash.remove(var1); } EXSB
  34. 34. Mutation Operator1 Description Acronym Add a synchronized block around a statement ASAT Add the synchronized keyword to the method header ASIM Add a synchronized block around a method ASM Change the order of two synchronized blocks order CSO Expand synchronized region after EXSA Expand synchronized region before EXSB Remove synchronized statement around a synchronized statement RSAS Remove synchronization around a variable RSAV Remove synchronized keyword in method header RSIM Remove synchronization block around method RSM Shrink synchronization block after SHSA Shrink synchronization block before SHSB 1All mutation operators are written in the TXL source transformation language – http://www.txl.ca. © 2018 J.S. Bradbury 34
  35. 35. ARC PHASE 1 Repairing Deadlocks and Data Races 3. Apply mutation to an individual in population • During execution the GA selects a type of mutation (i.e., a mutation operator) – random on first generation • From the set of mutations created a random instance is used 35© 2018 J.S. Bradbury
  36. 36. ARC PHASE 1 Repairing Deadlocks and Data Races 4. Evaluate individuals • Fitness function is used to evaluate mutants selected in previous step • The mutated individual is maintained only if the fitness function is improved 36© 2018 J.S. Bradbury functional fitness(P) = (s x sw) + (t x tw) where: s = # of successful executions sw = success weighting (high) t = # of timeout executions tw = timeout weighting (low)
  37. 37. ARC PHASE 1 Repairing Deadlocks and Data Races 4. Evaluate individuals • In order to evaluate the fitness function for a given individual we need to evaluate the function over many different interleavings/executions • We use IBM's ConTest [EFN+02], which instruments the program with noise, to ensure that many interleavings are evaluated [EFN+02] Edelstein, O., Farchi, E., Nir, Y., Ratsaby, G., Ur, S.: Multithreaded Java program test generation. IBM Systems Journal 41(1), 111–125 (2002) © 2018 J.S. Bradbury 37
  38. 38. ARC PHASE 1 Repairing Deadlocks and Data Races 4. Check terminating condition • An individual that produces 100% successful executions is a potential fix • However, we perform an additional step to increase confidence that the individual is in fact correct • We evaluate the individual with ConTest again using a safety multiplier (e.g., 20) to increase the number of interleavings explored – a fix is only accepted if we achieved 100% success for this additional evaluation 38© 2018 J.S. Bradbury
  39. 39. ARC PHASE 1 Repairing Deadlocks and Data Races 4. Check terminating condition • Our approach will not always find a successful solution that repairs all of the concurrency bugs in a program • If after a user-defined number of generations a solution is not reached our algorithm will terminate 39© 2018 J.S. Bradbury
  40. 40. ARC PHASE 1 Repairing Deadlocks and Data Races 5. Replace weakest individuals (Optional) • To encourage individuals to explore more fruitful areas of state space we can replace individuals • We can restart with original population • We can replace (e.g., 10%) of underperforming individuals with random high-performance individuals or with original program 40© 2018 J.S. Bradbury
  41. 41. ARC PHASE 1 Repairing Deadlocks and Data Races 6. Calculate operator weighting • We leverage historic information from previous generations to weight the operators and increase the likelihood that useful operators are selected first/more frequently • Strategy 1: weight based on % of dead locks/data races uncovered • Strategy 2: Weight based on a mutation operator’s fitness function success 41© 2018 J.S. Bradbury
  42. 42. ARC PHASE 1 Repairing Deadlocks and Data Races 6. Calculate operator weighting • Strategy 1: weight based on % of dead locks/data races uncovered • For example, some operators are geared towards fixing deadlocks, others data races and some both. • We increase the likelihood that specific operators are selected based on the number of deadlock and data races in our historic evaluations 42© 2018 J.S. Bradbury
  43. 43. ARC PHASE 1 Repairing Deadlocks and Data Races 6. Calculate operator weighting • Strategy 2: Weight based on a mutation operator’s fitness function success • For example, operators that have historically increased the fitness are weighted proportional to their success 43© 2018 J.S. Bradbury
  44. 44. ARC PHASE 2 Optimizing Repaired Source Code • ARC may introduce unnecessary synchronization during Phase 1. • If Phase 1 is successful, an optional second phase attempts to improve the running time of the program- under-repair by shrinking and removing unnecessary synchronization blocks • A new non-functional fitness function and a subset of the mutation operators (e.g., RSAS, SHSA) are used © 2018 J.S. Bradbury 44
  45. 45. Evaluation – Setup • We selected a set of 8 programs from the IBM Concurrency Benchmark [EU04] that have deadlock or data race bugs • 6 programs that exhibit bugs ARC was designed to fix • 2 programs that ARC was not designed to fix (sanity check) • Each program was analyzed using 5 executions of ARC [EU04] Eytani, Y., Ur, S.: Compiling a benchmark of documented multi-threaded bugs. In Proc. of Work. on Parallel and Distributed Sys.: Testing, Analysis, and Debugging (PADTAD 2004), 2004. © 2018 J.S. Bradbury 45
  46. 46. Evaluation – Results1 Program Bug Type Bug Repaired? # Generations to Repair Bug (Avg.) Time Required to Repair Bug (Avg.) account Data Race ✔ 5.0 08m 08s accounts Data Race ✔ 1.0 44m 00s bubblesort2 Data Race ✔ 2.2 1h 40m 20s deadlock Deadlock ✔ 1.0 02m 12s lottery Data Race ✔ 2.4 38m 00s pingpong Data Race ✔ 1.0 12m 32s airline Data Race ✖ - - buffer Data Race ✖ - - 1Our evaluation was conducted on a Linux PC with a 2.33 GHz processor, 4 gigabytes of RAM running Linux. © 2018 J.S. Bradbury 46
  47. 47. Challenges & Future Work • Flexibility – ARC is currently only capable of fixing deadlocks and data races • We place to explore other mutation operators that will increase the kinds of bugs that can be fixed • Readability [FLW12] – automatic repair always has the potential to decrease the readability and maintainability of the source code • We have not studied the readability of the fixes produced by ARC [FLW12] Zachary P. Fry, Bryan Landau, and Westley Weimer. “A Human Study of Patch Maintainability.” In Proc. of the International Symposium on Software Testing and Analysis (ISSTA), 177–187, 2012. © 2018 J.S. Bradbury 47
  48. 48. EXAMPLE #2 Predicting Future Code Changes With Historic Commit Data © 2018 J.S. Bradbury 48
  49. 49. Software Projects • Software projects are often developed in teams or groups of people contributing through a version control system • Overtime software projects evolve • introduce new functionality, remove functionality, fix bugs, optimize code, etc. • Popular git-based version control platforms include GitHub or BitBucket • Host both open and closed source software © 2018 J.S. Bradbury 49
  50. 50. “ What is a Commit? A commit, or "revision", is an individual change to a file (or set of files) … that allows you to keep record of what changes were made when and by who. Commits usually contain a commit message which is a brief description of what changes were made.1 1 https://help.github.com/articles/github-glossary/ © 2018 J.S. Bradbury 50
  51. 51. Research Questions • Can historic commit data be used to predict future changes in software projects? • Using Machine Learning (ML) for prediction • What is the impact the following factors on the performance of the prediction? • Sampling range • Feature set • Balanced sampling © 2018 J.S. Bradbury 51
  52. 52. Research Questions ● If the answer to the research questions is YES! (we can predict future code changes) the benefits include: ● Providing developers with insight into future development development ● Providing project managers a new tool for resource allocation © 2018 J.S. Bradbury 52
  53. 53. Overview of Prediction Approach © 2018 J.S. Bradbury 53 V1.0 V1.01 ... V2.0 Vn GitHub Machine Learning Vn+1 Candidate Features SVM RF Predict ... SVM = Support Vector Machine RF = Random Forest
  54. 54. Experiment Setup • Collected the data from 23 projects including: • ACRA/acra • google/blockly-android • apache/storm • Projects were all: Java, open source, hosted on GitHub, have longer development history (+1 year) • Data collected was all extracted from GitHub repositories and stored locally for experiments. © 2018 J.S. Bradbury 54
  55. 55. Experiment Setup • The features used to predict future code changes will occur in a given unit of code (method, class) include: • Committer • Method Signature • Filename • Overall change frequency • Short-term change frequency • Method Length • Changed in previous 5 commits? (true/false) 55© 2018 J.S. Bradbury
  56. 56. Experiment Setup • Utilized two different machine learning algorithms: • Support Vector Machine (SVM) • Supervised learning • Applications in classification and regression • Can be used for multi-class tasks • Random Forest (RF) • Supervised learning that work by constructing decision trees • Applications in classification and regression • Ensemble learning method 56© 2018 J.S. Bradbury
  57. 57. Experiment Setup • Conducted 6 separate experiments • One experiment for each factor using each machine learning technique © 2018 J.S. Bradbury 57 Sampling range Feature set Balanced sampling Support Vector Machine Random ForestX
  58. 58. Experiment Setup • Conducted 6 separate experiments • One experiment for each factor using each machine learning technique © 2018 J.S. Bradbury 58 Sampling range Feature set Balanced sampling Support Vector Machine Random ForestX
  59. 59. Experiment Setup • Measured performance in terms of: • Accuracy • Precision • Recall © 2018 J.S. Bradbury 59
  60. 60. Experiment 1: Impact of Sampling Range • Does sampling range impact predictive capabilities of the approach? • For each project, 8 values for the range size are tested © 2018 J.S. Bradbury 60
  61. 61. Experiment 1: SVM Results for acra • Performed well with some of the values • Performance did have variations © 2018 J.S. Bradbury 61
  62. 62. Experiment 1: SVM Results for acra • Performed well with some of the values • Performance did have variations © 2018 J.S. Bradbury 62
  63. 63. Experiment 1: Discussion • Performance variations • Large variation for recall • Smaller for precision and accuracy • Smaller projects performed better • Factor is impactful © 2018 J.S. Bradbury 63
  64. 64. Discussion • Predicting changes within a project is possible • Both machine learning algorithms are capable of providing predictions • The Sample Window Range and Feature Set proved most influential • Project dependent factors require further investigation © 2018 J.S. Bradbury 64
  65. 65. AI & SE – Understanding the Relationship © 2018 J.S. Bradbury Artificial Intelligence Software Engineering AI + SE How can AI be applied to SE? 65
  66. 66. AI & SE – Understanding the Relationship © 2018 J.S. Bradbury Artificial Intelligence Software Engineering AI + SE How can SE be applied to AI? 66
  67. 67. Criticisms of Deep Learning • “Deep learning and AI in general ignore too much of the brain’s biology in favor of brute-force computing.” [MIT-DL] • “Google’s attitude is: lots of data makes up for everything” – viewpoint of Jeff Hawkins, founder of Palm Computing, on Google’s approach to deep learning [MIT-DL] • Concerns about bias and comprehension of deep learning algorithms also increase as the algorithms get more complex and the data gets bigger [MIT-DL] https://www.technologyreview.com/s/513696/deep-learning/ © 2018 J.S. Bradbury 67
  68. 68. Criticisms of Deep Learning • “Deep learning and AI in general ignore too much of the brain’s biology in favor of brute-force computing.” [MIT-DL] • “Google’s attitude is: lots of data makes up for everything” – viewpoint of Jeff Hawkins, founder of Palm Computing, on Google’s approach to deep learning [MIT-DL] • Concerns about bias and comprehension of deep learning algorithms also increase as the algorithms get more complex and the data gets bigger [MIT-DL] https://www.technologyreview.com/s/513696/deep-learning/ © 2018 J.S. Bradbury 68 Can Software Engineering help with this?
  69. 69. Can SE help solve AI’s black box? © 2018 J.S. Bradbury 69
  70. 70. 70© 2018 J.S. Bradbury
  71. 71. Funding provided by: http://www.sqrlab.ca Faculty of Science (Computer Science) University of Ontario Institute of Technology Oshawa, Ontario, Canada Automating Software Development Using AI Jeremy Bradbury, PhD jeremy.bradbury@uoit.ca

×