Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
IBM Watson Life ©2014 IBM Corporation
1University of California - Irvine, 2IBM Watson Group - Watson Life, 3IBM Research
J...
IBM Watson Life :: ©2014 IBM Corporation
Spoiler Alert
• Novelty:
• Identify new opportunities provided by cloud based IDE...
IBM Watson Life :: ©2014 IBM Corporation 3
IBM Watson Life :: ©2014 IBM Corporation 4
IBM Watson Life :: ©2014 IBM Corporation
Experimental Setup - Flow
• Lab Experiment with 8 IBM Employees
• Intake and Exit...
IBM Watson Life :: ©2014 IBM Corporation
Experimental Setup - Man In the Middle Attack
6
Participant
Computer
Execution
Co...
IBM Watson Life :: ©2014 IBM Corporation
Code Growth Extraction
• Captured TCP Session Traces Allow Replay of Code Growth
...
IBM Watson Life :: ©2014 IBM Corporation
Developer Traces
8
0 100 300 500
050100200
Subject 4 (Worst)
time (seconds)
codes...
IBM Watson Life :: ©2014 IBM Corporation
Co-Integration Networks
• Cointegration is an Indicator of Shape Similarity of Tw...
IBM Watson Life :: ©2014 IBM Corporation
Co-Integration Networks
• The “best” developers (subject 6 and 8) share few intra...
IBM Watson Life :: ©2014 IBM Corporation
Individual Programming Styles
• Question: Are code growth traces are more similar...
IBM Watson Life :: ©2014 IBM Corporation
Major Impacts
• Shift from desktop to cloud opens up incredible opportunities for...
Upcoming SlideShare
Loading in …5
×

ICSE 2014 NIER Track - New Opportunities for Extracting Insight from Cloud Based IDEs

677 views

Published on

It used to be that getting fine grained data about development practices was a complicated process that required convincing developers to install a plugin in their IDE. The shift to cloud based IDEs opens up numerous possibilities finer grained understanding of development practices. This research reports on a small preliminary study from IBM where we tracked eight developers as they interacted with a cloud based IDE and yields interesting insights about how much information we can actually learn from these environments.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

ICSE 2014 NIER Track - New Opportunities for Extracting Insight from Cloud Based IDEs

  1. 1. IBM Watson Life ©2014 IBM Corporation 1University of California - Irvine, 2IBM Watson Group - Watson Life, 3IBM Research June 4, 2014 Yi Wang1, Patrick Wagstrom2,3, Evelyn Duesterwald3, David Redmiles1 New Opportunities for Extracting Insight from Cloud Based IDEs @pridkett http://wagstrom.net/ patrick@wagstrom.net
  2. 2. IBM Watson Life :: ©2014 IBM Corporation Spoiler Alert • Novelty: • Identify new opportunities provided by cloud based IDEs • Demonstrate how to realize these opportunities with a case study. • New data analysis technique applied to case study. • Emerging: • Insights extracted from data collected in our case study • Promising future directions for cloud based IDEs • Impact: • Researchers - shift in focus to distributed environments • Practitioners - the future is awesome • IDE Developers - rich possibilities for tool enhancement based on findings 2
  3. 3. IBM Watson Life :: ©2014 IBM Corporation 3
  4. 4. IBM Watson Life :: ©2014 IBM Corporation 4
  5. 5. IBM Watson Life :: ©2014 IBM Corporation Experimental Setup - Flow • Lab Experiment with 8 IBM Employees • Intake and Exit Surveys to Understand Development Skill and Experience • Eight Programming Tasks of Increasing Difficulty • Similar to Interview Questions • Written in JazzHub Orion Code Editor • Instrumented to Record (Audio, Screen, and Network) of All Sessions • Effort and Difficulty Self-Evaluated After Each Task • Solutions Were Graded and Subjects Ranked after Experiment 5
  6. 6. IBM Watson Life :: ©2014 IBM Corporation Experimental Setup - Man In the Middle Attack 6 Participant Computer Execution Container MITMProxy JazzHub Google StackOverflow Data Logger The Internet
  7. 7. IBM Watson Life :: ©2014 IBM Corporation Code Growth Extraction • Captured TCP Session Traces Allow Replay of Code Growth • Collected 41 Total <subject, task> Traces 7
  8. 8. IBM Watson Life :: ©2014 IBM Corporation Developer Traces 8 0 100 300 500 050100200 Subject 4 (Worst) time (seconds) codesize(bytes) 0 20 40 60 80 0100200300400 Subject 6 (Best) time (seconds) codesize(bytes) Insight 1: Differing levels of expertise yield dramatically different code growth patterns
  9. 9. IBM Watson Life :: ©2014 IBM Corporation Co-Integration Networks • Cointegration is an Indicator of Shape Similarity of Two Time Series • We Calculated Pairwise Cointegration for all 41 Traces • Cointegration Network was Created Using
 Cases when p-value < 0.05 9 S1.1 S1.2 S1.3 S1.4 S2.1 S2.2 S2.3 S2.4 S2.5S3.1 S3.2 S3.3 S3.4S3.5 S4.1 S4.2 S4.3S5.1 S5.2 S5.3 S5.4S6.1 S6.2 S6.3 S6.4 S6.5 S6.6 S6.7 S6.8 S7.1 S7.2 S7.3 S7.4 S7.5 S8.1 S8.2 S8.3 S8.4 S8.5 S8.6 S8.7
  10. 10. IBM Watson Life :: ©2014 IBM Corporation Co-Integration Networks • The “best” developers (subject 6 and 8) share few intra-subject edges (5.71%), while the other 6 subjects share 112 edges (34.36%). • The “best” developers traces are mostly located in the peripheral part of the network of Fruchterman-Reingold layout. 10 S1.1 S1.2 S1.3 S1.4 S2.1 S2.2 S2.3 S2.4 S2.5S3.1 S3.2 S3.3 S3.4S3.5 S4.1 S4.2 S4.3S5.1 S5.2 S5.3 S5.4S6.1 S6.2 S6.3 S6.4 S6.5 S6.6 S6.7 S6.8 S7.1 S7.2 S7.3 S7.4 S7.5 S8.1 S8.2 S8.3 S8.4 S8.5 S8.6 S8.7 Insight 2:The “best” developers may have more strategies in programming, making the traces are more diverse.This may be used as an potential indicator to infer developers’ skill levels.
  11. 11. IBM Watson Life :: ©2014 IBM Corporation Individual Programming Styles • Question: Are code growth traces are more similar for a collection of users doing same task, or for an individual user doing a collection of task? • Compute edge density in three categories: • I: (subject-subject)—0.3085 • II: (task-task)—0.2479 • III: (task-subject)—0.2611. • Test the differences using Kruskal-Wallis test: P(I, II) < 0.01, P(I, III) = 0.03. 11 Insight 3: Individual differences rather than Task differences contribute more to the differences of code growth traces. Code growth traces may be some kind of “finger-print” of developers.
  12. 12. IBM Watson Life :: ©2014 IBM Corporation Major Impacts • Shift from desktop to cloud opens up incredible opportunities for studying large numbers of developers in depth • Centralized instrumentation allows data from one developer to benefit other developers in near real-time • In depth knowledge of IDE user behavior from cloud IDEs will enable next leap in productivity for software development professionals 12 @pridkett http://wagstrom.net/ patrick@wagstrom.net

×