Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (18)
Similar to Enhancing Developer Productivity with Code Forensics
Similar to Enhancing Developer Productivity with Code Forensics (20)
More from TechWell
More from TechWell (20)
Recently uploaded
Recently uploaded (20)
Enhancing Developer Productivity with Code Forensics
- 1. BW10 Session 6/5/2013 3:45 PM "Enhancing Developer Productivity with Code Forensics" Presented by: Anthony Voellm Google, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
- 2. Anthony Voellm Google, Inc. At Google Anthony Voellm is focused on delivering performance, reliability, and security to the Google Compute Engine, Google App Engine, Google Cloud SQL, and Google Cloud BigQuery while also innovating new offerings. His experience ranges from kernel and database engines to image processing and graphics. Anthony is an avid inventor who holds seven technology patents. Prior to joining Google in 2011, Anthony held multiple roles at Microsoft leading the Windows reliability, security, and privacy test teams. Anthony has taught performance testing to more than 2,000 people worldwide and given dozens of informative talks on software fundamentals and the cloud. He writes a technology blog on software fundamentals.
- 3. Enhancing Developer Productivity with Code Forensics: Applications of behavioral analysis and developer assessment to improve productivity Presented at BSE West - Vegas - June 5th 2013 Anthony F. Voellm Google Cloud Security, Performance and Test Manager voellm@google.com / G+ / @p3rfguy The hypothesis: Tomorrow Today One size fits all testing Do everything Do nothing Best guess Static code analysis • • • • The right amount of tests Skills / Knowledge Experience State of mind Behavior • • • •
- 4. Overview • Part 1 - The backdrop • Part 2 - The big question • Part 3 - Measure, Measure, Measure • Part 4 - Where is the science? • Part 5 - The path forward The backdrop
- 5. All developers are are the same - right? Chevy Nova for a car is a great name in english. "No va" however in spanish means "no go." Internationalization Logical Bugs Security bugs We live in a world of bugs Reliability Accessibility bugs Performance bugs ... int x[10]; x[10] = -1; ...
- 6. Where do bugs come from? Humans... The Google Brain - NYTimes article ... in the future machines. How do bugs happen? Fridays… :)
- 7. How do bugs happen? uTest Blog - " If you’re ti d T t Bl "... ’ tired, angry or frustrated for instance (like Patriots fans this morning) then you’re almost guaranteed to make some careless mistakes. ..." mistakes How do bugs happen? College mentee of mine "Cut and paste is wicked."
- 8. How do bugs happen? "How do fixes become bugs" Paper in 2011 - "...the bug-fixing process can also introduce errors... Developers and reviewers for incorrect fixes usually do not have enough knowledge...” How do bugs happen? NIST Study - "Software is error-ridden in part because of its growing i complexity. The size of l it software products is no longer measured in thousands of lines of code, but in millions. Software developers already spend approximately 80 percent of development costs on identifying and correcting defects, and yet few products of any type other than software are shipped with such high levels of errors."
- 9. Speaking of complexity... From - "How do fixes become bugs" paper The cost of bugs • Time o 25%+ of developers time is fixing bugs o A 1 line fix takes 1+ hours of testing is common • Money o ~$60 Billion (9 zero's) to the US economy each year in 2002! • Reputation o 10% Error rate on critical security fixes.
- 10. Reputation … this might be the most important. Years to develop and it only takes minutes to destroy. The big question
- 11. One size fits all With all the evidence that humans are the root cause of bugs *and* we all have different levels and of skill.... Why do we all test the same? From - "How do fixes become bugs" paper Types of testing • All unit tests (15 minutes or less should be the target) o The most basic test with all layers stripped away. • All Integration / System Tests (1 h or lless) hour ) o Uses multiple features together. • All Performance Tests (8 hours or less) o Micro-benchmarks (fio, iperf, ...) o Industry benchmarks (SpecCPU, TPC-C, Hibernate, ...) • All Reliability tests (days) o Longhaul o Leak detection tools • All Security tests (weeks) o Smart Fuzzers o Static code analyzers like Coverity Time Sink
- 12. HURRAY! “…time pressure prevents testers from conducting thorough regression tests before releasing the fix.”* How do we choose what to run? • Today: o Run everything o Run nothing o Selectively run based on complex change to test associations o Best guess based on developer caution
- 13. How do we choose what to run? • Tomorrow: o Run based on developer skill o Run based familiarity with code base o Run based on the complexity of the code o Run based on the type of bug being fixed o Run based on behavioral analysis of the code Measure, Measure, Measure
- 14. What should we measure? Artifacts • What is the frequency of check ins by the developer? • How often has this developer checked in a [severe] bug? [ ] g • Do bugs trail checkings?
- 15. Behavior and skill VS Code ..... Code ..... Code ..... Code ..... Code C d ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Freshman Code ..... Sophomore Code ..... Code ..... Code ..... Code ..... Code ..... Junior Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Senior Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Code ..... Guru Measures • • • • Is the checkin in code the developer is "familiar" with? Knowledge of the code reviewer. Peer ranking on how p p feel about y g people your level of expertise. What is the size of the check in?
- 16. Emotions http://people.brandeis.edu/~sekuler/eegERP.html Measures • Is the the day before a weekend? • Is the time of day of the check in unusual for the developer? • Bug Debt
- 17. Cyclomatic complexity http://en.wikipedia.org/wiki/Cyclomatic_complexity Measures • Does the developer write complex code? • How layered is the code? • Does the developer write units tests? • What percentage of the check in is covered by tests?
- 18. Where is the science? Studies... Empirical investigation of software p g product line quality Researcher: Katerina Goseva-Popstajanova Lane Department of Computer Science and Electrical Engineering West Virginia University est g a U e s ty ***Special thanks to Katerina Goseva-Popstajanova who presented at GTAC2013 and graciously allowed me to use these slides. You can see her full talk here - http://www.youtube.com/watch?v=fiG-SdNcjTE
- 19. West Virginia University Open source product line: basics The following slides are based on the paper “A longitudinal study of post-release faults in an evolving, open-source software product line” by T. Devine, K. Goseva-Popstajanova, S. Krishnan, and R R L t Ki h d R. R. Lutz Submitted to a journal, currently under review West Virginia University Open source product line: basics Eclipse can be treated as a SPL • Currently consists of fourteen different members that share main components and are set apart by variable components • Considered four products: Classic, C/C++, Java and JavaEE • Large size: these four products consist of over 125,000 files and 20 million LoC • Evolving product line: considered seven releases • Goals: assessment and prediction of post-release faults
- 20. West Virginia University Open source product line: basics Total Release Year Classic Pkgs 2.0 2002 2.1 C/C++ Java JavaEE Pkgs Pkgs Pkgs KLoC Pkgs Faulty Pkgs 34 773 34 26 2003 41 1054 41 37 3.0 2004 76 1756 76 70 3.3. Europa 2007 85 62 103 185 3988 185 148 3.4 34 Ganymede 2008 89 62 105 200 4291 200 152 3.5 Galileo 2009 77 61 104 188 3913 188 120 3.6 Helios 2010 77 61 105 206 4262 206 103 West Virginia University Different degrees of reuse Europa Ganymede Galileo Helios
- 21. Metrics West Virginia University Code metrics Change metrics LoC Revisions Average Changeset Statements Refactorings R f t i Age A Percent Branch Statement Bugfixes Weighted Age Method Call Statement Authors Percent Lines with Comments LoC Added Classes and Interfaces Max LoC Added Methods per class Average LoC Added Average Statements per Method LoC D l t d L C Deleted Max Complexity Max LoC Deleted Average Complexity Codechurn Max Block Depth Max Codechurn Average Block Depth Average Codechurn Statements at Block Level n (0, 1, … 9) Max Changeset West Virginia University Assessment: Evolution through releases 1. Does quality, measured by the number of post-release faults for the packages in each release, consistently improve as the SPL matures? Post-release fault density decreases as the product line matures through releases
- 22. West Virginia University Assessment: Post-release fault distribution 2. Do the majority of faults reside in a small subset of packages? For each release, from 66% to 93% of post-release faults were located in 20% of packages, with average around 81% West Virginia University Prediction: What features are good predictors? 7. Are some features better indicators of the number of post-release faults in a package than others? Feature selection via stepwise regression selected from 1 –16 features out of 112 features
- 23. West Virginia University Prediction: What features are good predictors? Of the fifteen features appearing in more than a quarter of models, only four are static code metrics West Virginia University Prediction: What features are good predictors? Change metrics (correlation 0.726 – 0.768) • total and maximum number of bugfixes • total authors • total code churn • total revisions Static code metrics ( S (correlation 0 610 - 0 683) 0.610 0.683) • maximum statements at block level one • maximum and total statements at block level four • maximum method call statements
- 24. The path forward Use the human factor to ship faster... • Create [AI] models that account for... o o o o o • Use the models as part of check in and code health... o o • Experience Knowledge of the code base Code complexity Measures of behavior ... and much more Developers with less risky profiles run less tests Developers with higher risk profile run more tests Let automated systems running in parallel be the safety net.
- 25. Human factors success - Blint! By Erick Fejta Let productivity soar!
- 26. End - Questions? Name: Tony) Contact: Blog: G+: Twitter: Anthony F. Voellm (aka voellm@google.com http://perfguy.blogspot.com http://goo.gl/mPXcX http://goo gl/mPXcX @p3rfguy Appendix
- 27. Abstract: This talk will present data and findings on how behavioral analysis and developer assessment can be applied to improving productivity. Just image an engineering system that could recognize rushed check-ins, "grade" developer knowledge, and use that data to speed up development "Congrats Jane you know this code well ... no check-in test gate for you." The approach has been motivated by looking at today's test systems, tools, and processes and recognizing thes are designed around the premise that all developers are created equal. Studies have shown developer error rates can vary widely and have a number of root causes - mind set of the developer at the time the code was written, experience level, amount of code in a check in complexity of the code, and much more. This talk will in, code more introduce a number of metrics and concepts such as Cyclomatic complexity and Digital Code Forensics, and demonstrate how even modest application of the approach can speed up development. This is the bleeding edge of engineering productivity. The message: Not all developers have the same experience or skill level, and we can use this to improve the speed of development. Speed up the better developers, and slow down the less precise. We dont need a one size fits all policy, however we do need to base the decisions on data.