Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing

44 views

Published on

As software engineering researchers, we are also zealous tool smiths. Building a research prototype is often a daunting task, let alone building a industry-grade family of tools supporting multiple platforms to ensure the generalizability of our results. In this paper, we give advice to academic and industrial tool smiths on how to design and build an easy-to-maintain architecture capable of supporting multiple integrated development environments (IDEs). Our experiences stem from WatchDog, a multi-IDE infrastructure that assesses developer testing activities in vivo and that over 2,000 registered developers use. To these software engineering practitioners, WatchDog, provides real-time and aggregated feedback in the form of individual testing reports.

Project Website: http://www.testroots.org
Demonstration Video: https://youtu.be/zXIihnmx3UE

Published in: Software
  • Be the first to comment

  • Be the first to like this

How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing

  1. 1. How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing
  2. 2. How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing Moritz Beller @Inventitech
  3. 3. How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing Moritz Beller @Inventitech Igor Levaja, Annibale Panichella, Andy Zaidman
  4. 4. How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing Moritz Beller @Inventitech Igor Levaja, Annibale Panichella, Andy Zaidman Georgios Gousios
  5. 5. Caveat: Not Science Science Image: https://static.pexels.com/photos/34845/pexels-photo.jpg
  6. 6. Caveat: Not Science Science Image: https://static.pexels.com/photos/34845/pexels-photo.jpg
  7. 7. Caveat: Not Science Science Discussion Item! Image: https://static.pexels.com/photos/34845/pexels-photo.jpg
  8. 8. Image: https://twitter.com/anonymousprofs TODO: Add background with Sun Academic Priming (or Blindness?)
  9. 9. Image: https://twitter.com/anonymousprofs TODO: Add background with Sun Academic Priming (or Blindness?)
  10. 10. Image: https://twitter.com/anonymousprofs TODO: Add background with Sun Academic Priming (or Blindness?)
  11. 11. Anonymous professor at TU Munich If you don't use Eclipse, you will fail the course. Image: https://twitter.com/anonymousprofs TODO: Add background with Sun Academic Priming (or Blindness?)
  12. 12. Anonymous professor at TU Munich If you don't use Eclipse, you will fail the course. Image: https://twitter.com/anonymousprofs TODO: Add background with Sun Academic Priming (or Blindness?)
  13. 13. Anonymous professor at TU Munich If you don't use Eclipse, you will fail the course. Image: https://twitter.com/anonymousprofs TODO: Add background with Sun Academic Priming (or Blindness?)
  14. 14. Anonymous professor at TU Munich If you don't use Eclipse, you will fail the course. Image: https://twitter.com/anonymousprofs Prototype TODO: Add background with Sun Academic Priming (or Blindness?)
  15. 15. WatchDog In One Slide
  16. 16. WatchDog In One Slide
  17. 17. WatchDog In One Slide How Much Do Developers Test? Do They Know This?
  18. 18. WatchDog In One Slide Do Developers Follow Test-Driven Development? How Much Do Developers Test? Do They Know This?
  19. 19. WatchDog In One Slide How Do Developers Test? Do Developers Follow Test-Driven Development? How Much Do Developers Test? Do They Know This?
  20. 20. Image: http://www.xda-developers.com/windows-developer-fee-gone/
  21. 21. Traveling Salesman (Problem)
  22. 22. Contacted Companies
  23. 23. Contacted Companies
  24. 24. Contacted Companies
  25. 25. Contacted Companies
  26. 26. Contacted Companies
  27. 27. Contacted Companies
  28. 28. Evolution of My Sales Pitch
  29. 29. Evolution of My Sales Pitch Brooks said that testing takes 50% of project effort. But does it really? Do you know how much you are testing?
  30. 30. Evolution of My Sales Pitch Brooks said that testing takes 50% of project effort. But does it really? Do you know how much you are testing? You can win an iPad!
  31. 31. Evolution of My Sales Pitch Brooks said that testing takes 50% of project effort. But does it really? Do you know how much you are testing? You can win an iPad! Help science!
  32. 32. Evolution of My Sales Pitch Brooks said that testing takes 50% of project effort. But does it really? Do you know how much you are testing? You can win an iPad! Help science! Help me!
  33. 33. Bootstrapping Research With Companies
  34. 34. Bootstrapping Research With Companies ● Not a good idea to come with an existing problem (and a solution).
  35. 35. Bootstrapping Research With Companies ● Not a good idea to come with an existing problem (and a solution). ● We were not fixing one of the companies' immediate problems. You need a company that wants to know this in their own interest.
  36. 36. Bootstrapping Research With Companies ● Not a good idea to come with an existing problem (and a solution). ● We were not fixing one of the companies' immediate problems. You need a company that wants to know this in their own interest. ● Privacy concerns can stop everything, at any stage (German “Betriebsrat”).
  37. 37. Bootstrapping Research With Companies ● Not a good idea to come with an existing problem (and a solution). ● We were not fixing one of the companies' immediate problems. You need a company that wants to know this in their own interest. ● Privacy concerns can stop everything, at any stage (German “Betriebsrat”). Very hard to do general research that everybody agrees on has value but does not fix any one's company specific problem. How can we do such research?
  38. 38. Bootstrapping Research With Companies ● Not a good idea to come with an existing problem (and a solution). ● We were not fixing one of the companies' immediate problems. You need a company that wants to know this in their own interest. ● Privacy concerns can stop everything, at any stage (German “Betriebsrat”). Very hard to do general research that everybody agrees on has value but does not fix any one's company specific problem. How can we do such research? Like so many research advances in SE, it is hard to quantify its value even for super-cool things (e.g., Code Recommenders).
  39. 39. That Didn't Work. What About OSS?
  40. 40. That Didn't Work. What About OSS? ● I was super-desperate.
  41. 41. That Didn't Work. What About OSS? ● I was super-desperate. ● Approached developers individually, but at scale.
  42. 42. That Didn't Work. What About OSS? ● I was super-desperate. ● Approached developers individually, but at scale. ● I approached people …
  43. 43. That Didn't Work. What About OSS? ● I was super-desperate. ● Approached developers individually, but at scale. ● I approached people … ● 16,000 of them.
  44. 44. Reactions to 16,000 Emails
  45. 45. Reactions to 16,000 Emails “Spammer!”
  46. 46. Reactions to 16,000 Emails “Spammer!” Even though not spam in the law sense (IANAL!), developers will perceive your mail as such.
  47. 47. Reactions to 16,000 Emails “Spammer!” “I'm using IntelliJ IDEA, so I won't be able to participate.” Even though not spam in the law sense (IANAL!), developers will perceive your mail as such.
  48. 48. Reactions to 16,000 Emails “Spammer!” “I'm using IntelliJ IDEA, so I won't be able to participate.” “Your research sounds verry [sic!] interesting to me. My Problem is that i am using Intellij for my project.” Even though not spam in the law sense (IANAL!), developers will perceive your mail as such.
  49. 49. Reactions to 16,000 Emails “Spammer!” “I'm using IntelliJ IDEA, so I won't be able to participate.” “Your research sounds verry [sic!] interesting to me. My Problem is that i am using Intellij for my project.” Even though not spam in the law sense (IANAL!), developers will perceive your mail as such. Some academics might be blind on what is going on “out there.” How can we change this?
  50. 50. The Academic Pitfall
  51. 51. The Academic Pitfall Write papers, not code! Good supervisor's advice.
  52. 52. The Academic Pitfall Write papers, not code! Good supervisor's advice. How many of our findings in the SE community are evaluated in 1 company, OSS project, … ?
  53. 53. How To Catch 'Em All
  54. 54. How To Catch 'Em All
  55. 55. How To Catch 'Em All You can support an array of environments with little more than 1x the effort.
  56. 56. WatchDog Architecture
  57. 57. WatchDog Architecture
  58. 58. WatchDog Architecture
  59. 59. WatchDog Architecture
  60. 60. WatchDog Architecture
  61. 61. WatchDog Architecture
  62. 62. WatchDog Architecture
  63. 63. WatchDog Architecture By relying on existing low-interference solutions, we can reduce LOC and effort to maintain.
  64. 64. Create Value For Your Users!
  65. 65. Create Value For Your Users!
  66. 66. Create Value For Your Users!
  67. 67. OK, So What Else Did We Do?
  68. 68. OK, So What Else Did We Do? ● Providing a high-profile project website.
  69. 69. OK, So What Else Did We Do? ● Providing a high-profile project website. ● Raffling off prizes.
  70. 70. OK, So What Else Did We Do? ● Providing a high-profile project website. ● Raffling off prizes. ● Writing articles in magazines and blogs relevant to Java and Eclipse developers (Eclipse Magazin, Jaxenter, EclipsePlanet, Heise News).
  71. 71. OK, So What Else Did We Do? ● Providing a high-profile project website. ● Raffling off prizes. ● Writing articles in magazines and blogs relevant to Java and Eclipse developers (Eclipse Magazin, Jaxenter, EclipsePlanet, Heise News). ● Giving talks and presentations at developer conferences (Dutch Testing Day, EclipseCon).
  72. 72. OK, So What Else Did We Do? ● Providing a high-profile project website. ● Raffling off prizes. ● Writing articles in magazines and blogs relevant to Java and Eclipse developers (Eclipse Magazin, Jaxenter, EclipsePlanet, Heise News). ● Giving talks and presentations at developer conferences (Dutch Testing Day, EclipseCon). ● Participating in a YouTube Java Developer series.
  73. 73. OK, So What Else Did We Do? ● Providing a high-profile project website. ● Raffling off prizes. ● Writing articles in magazines and blogs relevant to Java and Eclipse developers (Eclipse Magazin, Jaxenter, EclipsePlanet, Heise News). ● Giving talks and presentations at developer conferences (Dutch Testing Day, EclipseCon). ● Participating in a YouTube Java Developer series. ● Penetrating social media (Reddit, Hackernews, Twitter, Facebook).
  74. 74. OK, So What Else Did We Do? ● Providing a high-profile project website. ● Raffling off prizes. ● Writing articles in magazines and blogs relevant to Java and Eclipse developers (Eclipse Magazin, Jaxenter, EclipsePlanet, Heise News). ● Giving talks and presentations at developer conferences (Dutch Testing Day, EclipseCon). ● Participating in a YouTube Java Developer series. ● Penetrating social media (Reddit, Hackernews, Twitter, Facebook). ● Putting our plugin in a well-established marketplace.
  75. 75. Is There a Single “Silver Bullet”? To catch 'em all?
  76. 76. Is There a Single “Silver Bullet”? To catch 'em all? I think, yes.
  77. 77. Is There a Single “Silver Bullet”? To catch 'em all? I think, yes. But YMMV.
  78. 78. Has It Worked?
  79. 79. Has It Worked? 3,000
  80. 80. Has It Worked? 3,000
  81. 81. Moritz Beller @Inventitech How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing
  82. 82. Moritz Beller @Inventitech How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing Very hard to do general research that everybody agrees on has value but does not fix any one's company specific problem. How can we do such research?
  83. 83. Moritz Beller @Inventitech How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing Very hard to do general research that everybody agrees on has value but does not fix any one's company specific problem. How can we do such research? How can we contact developers individually without spamming them?
  84. 84. Moritz Beller @Inventitech How to Catch ’Em All: WatchDog, a Family of IDE Plug-Ins to Assess Testing How many of our findings in the SE community are evaluated in 1 company, OSS project, … Very hard to do general research that everybody agrees on has value but does not fix any one's company specific problem. How can we do such research? How can we contact developers individually without spamming them?

×