Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

VL/HCC 2014 - A Longitudinal Study of Programmers' Backtracking

596 views

Published on

  • Be the first to comment

VL/HCC 2014 - A Longitudinal Study of Programmers' Backtracking

  1. 1. A Longitudinal Study of Programmers’ Backtracking YoungSeokYoon (youngseok@cs.cmu.edu) Institute for Software Research Carnegie Mellon University Brad Myers (bam@cs.cmu.edu) Human-Computer Interaction Institute Carnegie Mellon University
  2. 2. Background VL/HCC 2014 2
  3. 3. What is Backtracking? •  Reverting code fragments to an earlier state •  Examples – Reverting a parameter to a previously used value – Removing debugging statements after fixing a bug – Restoring some deleted code – … VL/HCC 2014 3
  4. 4. Previous Studies of Backtracking •  Two qualitative studies of backtracking [Yoon+, CHASE’12] 1.  Preliminary lab study (12 programmers) 2.  Online survey (48 respondents) VL/HCC 2014 4
  5. 5. Previous Studies of Backtracking •  Observation – Programmers face challenges when backtracking •  locating the right code to be backtracked •  restoring some deleted code correctly •  reverting inter-related code fragments together – Programmers backtrack relatively often (75% answered at least “sometimes”) VL/HCC 2014 5
  6. 6. Limitations of the Previous Studies •  Lab study tasks required participants to backtrack •  Survey results may not correctly reflect the reality (e.g., programmers might backtrack unconsciously) •  The analyses were mostly qualitative VL/HCC 2014 6
  7. 7. A Longitudinal Study of Backtracking As a follow-up: VL/HCC 2014 7
  8. 8. Longitudinal Study of Backtracking •  Two main goals – Obtain backtracking statistics in order to quantify the need for backtracking tools – Identify backtracking situations that are not very well supported by existing programming tools VL/HCC 2014 8
  9. 9. Data Collection – Fluorite Logger http://www.cs.cmu.edu/~fluorite/ •  Eclipse logger for fine-grained code editing data [Yoon+, PLATEAU’11] •  Information collected: –  Initial snapshot of each source file –  All edit operations (insert, delete, or replace) –  Timestamps, executed editor commands, etc. •  Distributed to programmers since April 2012 VL/HCC 2014 9 [Image Src:Attribution: Rob Lavinsky, iRocks.com - CC-BY-SA-3.0]
  10. 10. Study Participants Group Description No. of Participants Coding Time (hours)   [min  /  avg  /  max  /  sum]   The first author (myself) 1 294  /  294  /  294  /  294   Graduate students @ CMU 13    3  /    40  /  216  /  520   Research programmers / System scientists @ CMU 5    6  /  118  /  446  /  588   Graduate students @ UPitt 2    6  /    29  /    51  /    57   Total 21 people 1,460 hours VL/HCC 2014 10
  11. 11. Analysis Process •  The data was too big for manual inspection – 1,345,241 coding events in the logs •  Key idea of the automated analysis – Keep the evolution history of individual AST nodes of interest throughout the lifetime of the nodes – Detect backtracking instances within each node VL/HCC 2014 11
  12. 12. Analysis Process Illustrated VL/HCC 2014 12 package example; public class Example { public void printRectangleInfo() { Rectangle rect = getEnclosingRect(); int value = rect.getHeight(); System.out.println("Value:" + value); } public Rectangle getEnclosingRect() { // return some rectangle here // actual code omitted // ... } } [Example Source Code Being Processed] S1 S2 S3 Change history of S1 [v1] Rectangle rect = getEnclosingRect(); Change history of S2 [v1] int value = rect.getHeight(); [v2] int value = rect.getWidth(); [v3] int value = rect.getSize(); [v4] int value = rect.getHeight(); Change history of S3 [v1] System.out.println(value); [v2] System.out.println("Value:" + value); [Memory of the Analyzer] Backtracking Detected!
  13. 13. Backtracking Instance A B C A B A v1 v2 v3 v4 v5 v6 time getHeight(); getWidth(); getSize(); getHeight(); getWidth(); getHeight(); Three Backtracking Instances: •  v1..v4 •  v2..v5 •  v4..v6 NOTE: v1..v6 is NOT a backtracking instance VL/HCC 2014 13
  14. 14. Research Questions 1.  How frequently do programmers backtrack in reality? 2.  How large are the backtrackings? 3.  How exactly do programmers perform backtracking? Are they backtracking manually? 4.  Is there evidence of exploratory programming? 5.  Are there backtrackings performed across multiple editing sessions? 6.  Are there selective backtrackings, which cannot be performed by the undo command? 7.  Do programmers backtrack to the same code repeatedly? VL/HCC 2014 14
  15. 15. 1. Frequency of Backtracking “How frequently do programmers backtrack in reality?” •  A total of 15,095 backtracking instances detected •  10.3 instances/hour on average VL/HCC 2014 15 0 10 20 30 P20 P19 P18 P17 P16 P15 P14 P13 P12 P11 P10 P9 P8 P7 P6 P5 P4 P3 P2 P1 P0 Backtracking Instances per Hour 3.8 (min) 28.4 (max) Average: 10.3/h Rate varied across participants (min=3.8/h, max=28.4/h), but all of them backtracked frequently
  16. 16. 2. Size of Backtracking “How large are the backtrackings?” •  How did we define the size of a backtracking? –  Measured the edit distance (Levenshtein distance) between the original version and the other versions –  Took the maximum value as the size of backtracking instance A B C D E A v1 v2 v3 v4 v5 v6 time farthest version (max edit distance) forward changes backward changes original version VL/HCC 2014 16
  17. 17. 2. Size of Backtracking “How large are the backtrackings?” VL/HCC 2014 17 1304 3752 5269 2026 2259 265 220 0 2000 4000 6000 1 2-9 10 -49 50 -99 100 -499 500 -999 ≥1000 Numberof BacktrackingInstances Backtracking Size (No. of Characters)
  18. 18. 2. Size of Backtracking “How large are the backtrackings?” •  Method / variable names •  String literals •  Number literals VL/HCC 2014 18 1304 3752 5269 2026 2259 265 220 0 2000 4000 6000 1 2-9 10 -49 50 -99 100 -499 500 -999 ≥1000 Numberof BacktrackingInstances Backtracking Size (No. of Characters)
  19. 19. 2. Size of Backtracking “How large are the backtrackings?” •  Simple parameter changes •  Reverting renaming changes on methods or variables VL/HCC 2014 19 1304 3752 5269 2026 2259 265 220 0 2000 4000 6000 1 2-9 10 -49 50 -99 100 -499 500 -999 ≥1000 Numberof BacktrackingInstances Backtracking Size (No. of Characters)
  20. 20. 2. Size of Backtracking “How large are the backtrackings?” •  Single statement changes •  Surrounding existing code (e.g., try-catch) then reverting VL/HCC 2014 20 1304 3752 5269 2026 2259 265 220 0 2000 4000 6000 1 2-9 10 -49 50 -99 100 -499 500 -999 ≥1000 Numberof BacktrackingInstances Backtracking Size (No. of Characters)
  21. 21. 2. Size of Backtracking “How large are the backtrackings?” •  Adding, removing, or modifying multiple statements and then reverting them altogether VL/HCC 2014 21 1304 3752 5269 2026 2259 265 220 0 2000 4000 6000 1 2-9 10 -49 50 -99 100 -499 500 -999 ≥1000 Numberof BacktrackingInstances Backtracking Size (No. of Characters)
  22. 22. 2. Size of Backtracking “How large are the backtrackings?” •  Significant algorithmic changes •  Adding / removing / modifying multiple methods and then reverting VL/HCC 2014 22 1304 3752 5269 2026 2259 265 220 0 2000 4000 6000 1 2-9 10 -49 50 -99 100 -499 500 -999 ≥1000 Numberof BacktrackingInstances Backtracking Size (No. of Characters)
  23. 23. 2. Size of Backtracking “How large are the backtrackings?” VL/HCC 2014 23 1304 3752 5269 2026 2259 265 220 0 2000 4000 6000 1 2-9 10 -49 50 -99 100 -499 500 -999 ≥1000 Numberof BacktrackingInstances Backtracking Size (No. of Characters) Programmers backtrack at varying granularities, from simple name changes to significant algorithmic changes
  24. 24. 3. Backtracking Tactics “How exactly do programmers perform backtracking?” How were the backtrackings performed? Manually 38% Using Existing Tools 49% Others 13% VL/HCC 2014 24
  25. 25. 3. Backtracking Tactics “How exactly do programmers perform backtracking?” How were the backtrackings performed? Manually 38% Using Existing Tools 49% Others 13% •  Undo (37%) •  Paste (6%) •  Redo (3%) •  Content Assist (2%) •  Toggle Comment (1%) VL/HCC 2014 25
  26. 26. 3. Backtracking Tactics “How exactly do programmers perform backtracking?” How were the backtrackings performed? Manually 38% Using Existing Tools 49% Others 13% •  Unidentified (9%) •  Multiple (4%) VL/HCC 2014 26
  27. 27. 3. Backtracking Tactics “How exactly do programmers perform backtracking?” How were the backtrackings performed? Manually 38% Using Existing Tools 49% Others 13% •  Manual Deletion (25%) •  Manual Typing (13%) VL/HCC 2014 27 38% of the backtracking instances were NOT supported by existing tools, indicating programmers need better backtracking tools
  28. 28. 4. Cross-Run Backtracking “Is there evidence of exploratory programming?” •  Make some changes à run the application à revert the code back to the way it was before •  20.4% of all instances were cross- run instances on average. VL/HCC 2014 28 0% 10% 20% 30% 40% 50% P20 P19 P18 P17 P16 P15 P14 P13 P12 P11 P10 P9 P8 P7 P6 P5 P4 P3 P2 P1 P0 Cross-Run Backtracking Percentage Average: 20.4% This provides support that programmers do this kind of exploratory programming.
  29. 29. 5. Cross-Session Backtracking “Are there backtrackings performed across multiple editing sessions?” 96.7% 98.2% 98.8% 99.0% 99.2% 99.3% 96% 97% 98% 99% 100% Same Session ≤1 ≤2 ≤3 ≤4 ≤5 CumulativePercentageofAllBIs Editing Session Distance VL/HCC 2014 29 A backtracking tool would work for 97% of the cases with only the history within the same editing session.
  30. 30. 6. Selective Backtracking “Are there backtrackings that could not have done by regular undo?” •  Selective backtracking? –  There are edits in the middle of a backtracking that change other parts of the same file, that are not backtracked together VL/HCC 2014 30 0% 5% 10% 15% 20% P20 P19 P18 P17 P16 P15 P14 P13 P12 P11 P10 P9 P8 P7 P6 P5 P4 P3 P2 P1 P0 Selective Backtracking Percentage Average: 9.5%
  31. 31. 6. Selective Backtracking “Are there backtrackings that could not have done by regular undo?” •  Selective backtracking? –  There are edits in the middle of a backtracking that change other parts of the same file, that are not backtracked together VL/HCC 2014 31 0% 5% 10% 15% 20% P20 P19 P18 P17 P16 P15 P14 P13 P12 P11 P10 P9 P8 P7 P6 P5 P4 P3 P2 P1 P0 Selective Backtracking Percentage Average: 9.5% On average, 9.5% of all backtracking instances were selective, supporting that programmers need better selective backtracking tools
  32. 32. 7. Repeat Count “Do programmers backtrack to the same code repeatedly?” 85.0% 11.1% 2.7% 0.7% 0.6% 0% 20% 40% 60% 80% 100% 1 2 3 4 ≥5 PercentageofBacktrackedNodes Repeat Count VL/HCC 2014 32 Most (85%) of the time, programmers backtrack once and then never gets back to the same state after diverging from it
  33. 33. Wrapping Up VL/HCC 2014 33
  34. 34. Limitations of the Analysis •  Only exact and successful backtracking instances were detected •  Only for Java / Eclipse •  Could not determine the semantic relationships among the backtracking instances VL/HCC 2014 34
  35. 35. Main Takeaways •  Programmers backtrack quite frequently (10.3/hr) •  38% of the backtrackings are done purely manually •  9.5% of the backtrackings are selective, meaning that they are not supported by conventional undo •  Programmers would benefit from better backtracking tools! VL/HCC 2014 35
  36. 36. Azurite – Selective Undo Tool http://www.cs.cmu.edu/~azurite/ •  A selective undo plug-in for Eclipse IDE –  can handle the 9.5% of selective backtrackings •  Presented atVL/HCC –  Initial User Interfaces of the Tool: Yoon, Myers, & Koo,“Visualization of Fine-Grained Code Change History”, Full Paper atVL/HCC’13 –  Tool Demonstration (yesterday): Yoon & Myers,“A Demonstration of Azurite: Backtracking Tool for Programmers”, Showpiece atVL/HCC’14 VL/HCC 2014 36 [Image Src:Attribution: cobalt, flickr.com - CC-BY-SA-2.0 ]
  37. 37. ThankYou! •  FLUORITE: A logging plug-in for Eclipse (Full of Low-level User Operations Recorded In The Editor) available at: http://www.cs.cmu.edu/~fluorite/ •  AZURITE: A selective undo plug-in for Eclipse (Adding Zest to Undoing and Restoring Improves Textual Exploration) available at: http://www.cs.cmu.edu/~azurite/ •  Thanks for funding from: VL/HCC 2014 37

×