Thank you. Today, I’m going to talk about Capturing and Analyzing Low-Level Events from the Code Editor.In this talk, I will introduce our new logging tool called “Fluorite,” but let’s go over the motivations of this work first.
Also, when we’re trying to evaluate existing tools, we ask many different questions such as…
The resulting Fluorite log file looks like this.I do not expect you to read the details, but you can see that each event is logged as an XML element.
For technical reasons, a document change event always precedes the causing command. That’s why the document change event appears as a thrid element instead of fourth.
Please let us know if you have any ideas on other analyses that might be useful in general.We could implement those in our analyzer.
Finally, Fluorite logs enable us to automatically detect some code editing patterns that are compused of sequences of events.We brainstormed what types of patterns could be detected, and here are some of the examples.
Backup slide for Replay question.. the cursor position.. it would be tricky to replay it in Eclipse
PLATEAU 2011 - Capturing and Analyzing Low-Level Events from the Code Editor
Capturing and Analyzing Low-Level Events from the Code Editor YoungSeok Yoon (firstname.lastname@example.org) Institute for Software Research (ISR) Brad A. Myers (email@example.com) Human-Computer Interaction Institute (HCII) School of Computer Science Carnegie Mellon University
Motivation• When trying to understand what problems the developers have, researchers investigate… how frequently when / how do do those the developers use problems occur? what kinds of certain features? problems do developers have?in what context in what sequence do the does a certain developers complete event occur? certain tasks? PLATEAU 2011 2
Motivation• When trying to evaluate existing tools… how how much do the developers time do they frequently do actually use those spend using developers tools? the tools? use the tools? do the what types of exactly how developers use errors do they are used the tools as they make? in detail? expected? PLATEAU 2011 3
How do we answer these questions?• Essentially, we are collecting the usage data from the developers• The usage data can be collected by: ▪ asking the developers ▪ observing the developers ▪ mining software repositories ▪ logging the developers’ behavior• However, previous tools make it difficult or impossible to get / analyze fine-grained code editing history PLATEAU 2011 4
Asking the developers• Interviews, surveys, focus groups• Limitations ▪ Responses may not be reliable ▪ Developers perform many operations quite automatically Maybe they might not remember the specific occasions PLATEAU 2011 5
Observing the developers• Contextual inquiries, lab studies• Think aloud + videotape• Limitations ▪ Requires manual inspection of the videotape Can be time-consuming and error-prone• [Coman2008], [Ko2003], [Ko2005], ... PLATEAU 2011 6
Mining software repositories• Has been used for studying software evolution• Limitations ▪ Cannot know exactly what happened between two consecutive revisions we may miss some important user behavior ▪ Need to infer from the check-in comments and other clues• [Aversano2007], [Bettenburg2009], [Kim2005], [Murphy-Hill2009], … PLATEAU 2011 7
Eclipse Usage Data Collector (UDC)• Logging tool for aggregate data• Collects usage information from all the Eclipse users who consented to upload their data to UDC• Limitations ▪ Does not capture the sequences of events ▪ Misses some important commands executed • e.g., UDC ignores backspace, moving cursor with arrow keys, … ▪ Command-specific parameters are not captured PLATEAU 2011 8
Other Logging Tools• Mylyn Monitor [Kersten2006] ▪ Textual level changes are not captured ▪ Focuses on more abstract user interaction data on the entire IDE• Syde / Replay [Hattori2011] ▪ Only commands which change the source code are logged ▪ AST-level, not textual level PLATEAU 2011 9
Our new logging tool: FLUORITE Full of Low-level User Operations Recorded In The Editor PLATEAU 2011 10
Our new logging tool: FLUORITE• FLUORITE is a publicly available Eclipse plug-in that captures low-level code editing events and produces XML log files• This is a tool for the researchers• Can be used to overcome the previously listed limitations• For each event, FLUORITE logs: ▪ timestamp ▪ event type / command ID ▪ event-specific parameters PLATEAU 2011 11
Three types of events• FLUORITE logs three different types of events: ▪ Commands: all the user events in the code editor • (e.g., typing new text, moving cursor, copying, …) ▪ Document Changes • logged whenever the active file is modified • contain the actual inserted/deleted code, resulting code length, … • makes it possible to reproduce snapshots of the files at any point ▪ Annotations • logged when the developer manually adds annotations to help the researcher PLATEAU 2011 12
FLUORITE log files• The log files are written in XML format ▪ Anyone can build their own automatic analyzer!• Is FLUORITE practical enough to use? ▪ Already has been useful for a couple of studies • Our own exploratory study with 12 developers • Dörner’s evaluation study of Euklas system ▪ Size of the logs • Average log size: 236.8KB / hr = 9.25MB / week • Could be reduced to 1MB / week if the logs were compressed ▪ Performance • There was no measurable performance loss during our study PLATEAU 2011 22
Built-in analyses• We also provide a log analyzer which has several built-in analyses• Our study focused on when and how the developers backtrack while editing code ▪ The analyses were built for this purpose PLATEAU 2011 23
Example Analysis:Command distribution report PLATEAU 2011 24
Example Analysis:Command distribution report PLATEAU 2011 25
Example Analysis:Command distribution report PLATEAU 2011 26
Example Analysis:Command distribution report PLATEAU 2011 27
Example Analysis:Keystroke distribution report PLATEAU 2011 28
Example Analysis:Keystroke distribution report PLATEAU 2011 29
Example Analysis:Keystroke distribution report PLATEAU 2011 30
Example Analysis:Keystroke distribution report PLATEAU 2011 31
Example Analysis:Code size growth graph PLATEAU 2011 32
Example Analysis:Code editing pattern detection• FLUORITE logs enable us to detect code editing patterns composed of sequences of events• Examples (not all of them are implemented) ▪ Typo correction ▪ Parameter tuning ▪ Commenting out / uncommenting ▪ Cutting/copying and pasting within a project ▪ Manual refactoring (e.g., rename variable)• Preliminary implementations Pattern Counts Rate Precision Typo correction 274 / 288 13.6 / hr 95.14% Parameter tuning 52 / 98 2.6 / hr 53.06% PLATEAU 2011 34
Conclusion• FLUORITE web page: http://www.cs.cmu.edu/~fluorite/• FLUORITE is publicly available, open-sourced tool for Eclipse which can be used when conducting studies• FLUORITE turned out to be useful for our own study, and we hope that it will help you too! PLATEAU 2011 35
Questions?• FLUORITE web page: http://www.cs.cmu.edu/~fluorite/• Acknowledgements ▪ National Science Foundation (NSF) CCF-0811610 ▪ Korea Foundation for Advanced Studies (KFAS) PLATEAU 2011 36
References• [Aversano2007] Aversano, L., Cerulo, L. and Di Penta, M. 2007. How Clones are Maintained: An Empirical Study. In Proc. 11th European Conf. on Soft. Maint. and Reengineering (CSMR’07). 81-90.• [Bettenburg2009] Bettenburg, N., Weyi, S., Ibrahim, W., Adams, B., Ying, Z. and Hassan, A. E. 2009. An Empirical Study on Inconsistent Changes to Code Clones at Release Level. In Proc. 16th Working Conf. on Reverse Eng. (WCRE’09). 85-94.• [Coman2008] Coman, I. D. and Sillitti, A. 2008. Automated Identification of Tasks in Development Sessions. In Proc. 16th IEEE Int’l Conf. on Program Comprehension (ICPC’08). 212-217.• [Kersten2006] Kersten, M. and Murphy, G. C. 2006. Using task context to improve programmer productivity. In Proc. 14th ACM SIGSOFT Int’l Symp. on Foundations of Soft. Eng. (FSE’06). 1-11.• [Kim2005] Kim, M., Sazawal, V., Notkin, D. and Murphy, G. 2005. An empirical study of code clone genealogies. In Proc. 10th Euro. Soft. Eng. Conf. & 13th ACM SIGSOFT Int’l Symp. on Foundations of Soft. Eng. (ESEC/FSE’05). 187-196.• [Ko2003] Ko, A. J. and Myers, B. A. 2003. Development and evaluation of a model of programming errors. In Proc. IEEE Symp. on Human Centric Computing Languages and Environments (HCC’03). 7-14.• [Ko2005] Ko, A. J., Aung, H. H. and Myers, B. A. 2005. Design requirements for more flexible structured editors from a study of programmers’ text editing. In Proc. Extended Abstracts of CHI2005. 1557-1560.• [Hattori2011] Hattori, L., D’Ambros, M., Lanza, M. and Lungu, M. 2011. Software Evolution Comprehension: Replay to the Rescue. In Proc. 19th IEEE Int’l Conf. on Program Comprehension (ICPC’11). 161-170.• [Murphy-Hill2009] Murphy-Hill, E., Parnin, C. and Black, A. P. 2009. How we refactor, and how we know it. In Proc. 31st Int’l Conf. on Soft. Eng. (ICSE’09). 287-297. PLATEAU 2011 37